alt text

Before we start, you can find all the code related to this article in this repository.

In this article, I will be taking you through the process of building a real-time dashboard using Go, ReactJS, and gRPC. I’m writing this article with the assumption that you have a basic understanding of the technologies mentioned above. But I will make sure to explain and provide documentation if you are interested in learning more about a certain topic. So what is gRPC ? gRPC is a RPC ( Remote procedure call ) framework that was built and open-sourced by Google. In simple terms, an RPC means invoking a procedure in a remote server. One cool thing about this is that the developer doesn’t need to explicitly code the details of the network interaction. It’s all handled by the framework instead. As you will see in this article, invoking an RPC feels like you are just calling a function in another server. And also this can be done even if the server and the client are written in different languages.

We will be building a Go server that uses gRPC to communicate with our client, which is a javascript app. gPRC allows you to communicate in multiple methods.

Unary RPC Link to heading

Client sends a request and gets a response back from the server ( this is similar to REST )

Server-side streaming Link to heading

Client sends a request and gets a stream of messages from the server

lient-side streaming Link to heading

The opposite of server-side streaming. Client sends a stream of messages to the server

Bidirectional streaming Link to heading

Server and Client send streams of messages to each other. How you process these messages are based on how your application is structured. But these two streams are independent of one another so you have more flexibility when it comes to deciding on how to handle these streams.

For this project, our goal is to build a dashboard that updates in real-time. Let’s assume we are building this dashboard to display some sensor data. And our server will be collecting all the sensor data for us. Therefore in this case our client should invoke a request to the server ( saying I’m ready to accept data ) and then the server should keep sending data to the client whenever the sensor value changes. So it looks like Server-side streaming is the most suitable in this situation.

Setting up the server Link to heading

  • protobuf or protocol buffers is a mechanism for serializing structured data invented by Google. It tends to be smaller and faster than other popular serialization standards such as JSON and XML. But unlike JSON and XML protocol buffers can be used for defining services interfaces ( not just message interchange formats )
  • gRPC uses Protocol Buffers as it’s Interface Definition Language. Therefore we can define our service interfaces and data structures in a proto file and then use the protoc compiler to generate our Go code for us. This is also a great benefit of using gRPC because we can generate code for many languages just by using one proto file.

sensor.proto Link to heading

syntax = "proto3";
package sensors;
option go_package="sensorpb";

message SensorRequest {
}

message SensorResponse {
    int64 value = 1;
}

service Sensor{
    rpc TempSensor(SensorRequest) returns (stream SensorResponse) {};
    rpc HumiditySensor(SensorRequest) returns (stream SensorResponse) {}; 
}

In this proto file we have defined two messages. One for SensorRequest and another for SensorResponse. The reason why theSensorRequest has no fields is that we are not expecting to pass anything into our RPC functions. In the SensorResponse message, we have a int64 field called value. This field will carry our sensor data. In the Sensor service, we have defined two RPC functions. One for getting the temperature and one for humidity. Both of these return a stream of SensorResponse messages.

Let’s save that file in a directory called proto. And then let’s create another directory called server at the same level as the protos directory. And create another directory called sensorpb inside protos directory. This is where we will store our generated Go code. So the directory structure should look like this.

-- protos 
   - sensor.proto
-- server
   -- sensorpb

First, let’s install some dependencies. We will need the protoc compiler and the protocol compiler plugin for Go. Install the protoc plugin: https://github.com/protocolbuffers/protobuf Install the Go plugin :

export GO111MODULE=on # Enable module mode
go get github.com/golang/protobuf/protoc-gen-go@v1.3A

After installing the dependencies mentioned above, run this command within the protos directory to generate client and server Go code and then store them in theserver/serverpb directory.

protoc sensor.proto — go_out=plugins=grpc:./../server/sensorpb

If you examine the server/sensorpb directory you will see a new file named sensor.pb.go. Now that we have the generated code we are ready to write our server.

Server.go Link to heading

package main

import (
	"grpc_stream_medium/server/sensorpb"
	"log"
	"net"

	"google.golang.org/grpc"
)

type server struct{}

func (*server) TempSensor(req *sensorpb.SensorRequest,
	stream sensorpb.Sensor_TempSensorServer) error {

	return nil
}

func (*server) HumiditySensor(req *sensorpb.SensorRequest,
	stream sensorpb.Sensor_HumiditySensorServer) error {

	return nil
}

func main() {

	lis, err := net.Listen("tcp", "0.0.0.0:8000")

	if err != nil {
		log.Fatalf("Error while listening : %v", err)
	}

	s := grpc.NewServer(
	sensorpb.RegisterSensorServer(s, &server{})

	if err := s.Serve(lis); err != nil {
		log.Fatalf("Error while serving : %v", err)
	}
}

Let’s breakdown this code.

  • In the main function, we will first create a Listener and tell it to run the server on port 8000
  • And then let’s create a *grpc.Server instance by calling grpc.Server()
  • Call the RegisterSensorServer function by importing the code that we generated as a package to register our sensor server. This function takes in a *grpc.Server and a type that satisfies the SensorServer interface ( you can examine the code in the generated sensor.pb.go to the function definition ).

type SensorServer interface {
	TempSensor(*SensorRequest, Sensor_TempSensorServer) error
	HumiditySensor(*SensorRequest, Sensor_HumiditySensorServer) error
}
  • Therefore we will have to define a type and then create two methods on that type called HumiditySensor and TempSensor to satisfy the SensorServer interface .
  • We have defined a type called server and created those two methods in the code block above the main function.
type server struct{}

func (*server) TempSensor(req *sensorpb.SensorRequest,
	stream sensorpb.Sensor_TempSensorServer) error {

	return nil
}

func (*server) HumiditySensor(req *sensorpb.SensorRequest,
	stream sensorpb.Sensor_HumiditySensorServer) error {

	return nil
}
  • Now, let’s run our app.

go run server.go

Since we don’t have real sensors for this example we will have to generate some fake data on our own. So let’s create a package called sensor that generates random data and stores them in a map datastructure.

package sensor

import (
	"log"
	"math/rand"
	"sync"
	"time"
)

type Sensor struct {
	Data map[string]int64
	M    *sync.RWMutex
}

//NewSensor creates a new sensor object
func NewSensor() *Sensor {
	return &Sensor{
		Data: make(map[string]int64),
		M:    &sync.RWMutex{},
	}
}

func (s *Sensor) SetTempSensor() {
	for {
		s.M.Lock()
		s.Data["temp"] = int64(rand.Intn(120))
		s.M.Unlock()
		time.Sleep(5 * time.Second)
	}
}

func (s *Sensor) SetHumiditySensor() {
	for {
		s.M.Lock()
		s.Data["humidity"] = int64(rand.Intn(100))
		s.M.Unlock()
		time.Sleep(2 * time.Second)
	}
}

//StartMonitoring will start fetch data from fake sensors in Goroutines
func (s *Sensor) StartMonitoring() {
	log.Println("Start monitoring...")
	go s.SetHumiditySensor()
	go s.SetTempSensor()
}

//GetTempSensor returns the latest temperature sensor data
func (s *Sensor) GetTempSensor() int64 {
	s.M.RLock()
	defer s.M.RUnlock()
	return  s.Data["temp"]
}

//GetHumiditySensor returns the latest temperature sensor data
func (s *Sensor) GetHumiditySensor() int64 {
	s.M.RLock()
	defer s.M.RUnlock()
	return s.Data["humidity"]
}

Our new directory structure should look like this.

-- protos 
   - sensor.proto
-- server
   -- sensorpb
   -- sensor

Let’s do a quick run through of the sensor package code.

  • setHumiditySensor and setTempSensor functions generate random data and store them in a map. setHumiditySensor sets a new value every 2 seconds, setTempSensor function sets a new value every 5 seconds.
  • StartMonitoring function will run those two functions in the background as Goroutines and keeps generating data until we stop the application.
  • GetTempSensor and GetHumiditySensor functions will return the values for the humidity and temperature sensors.

Now let’s modify the server.go code

package main

import (
	"fmt"
	"grpc_stream_medium/server/sensor"
	"grpc_stream_medium/server/sensorpb"
	"log"
	"net"
	"time"

	"google.golang.org/grpc"
)

type server struct {
	Sensor *sensor.Sensor
}

func (s *server) TempSensor(req *sensorpb.SensorRequest,
	stream sensorpb.Sensor_TempSensorServer) error {
	for {
		time.Sleep(time.Second * 5)

		temp := s.Sensor.GetTempSensor()
		err := stream.Send(&sensorpb.SensorResponse{Value: temp})
		if err != nil {
			log.Println("Error sending metric message ", err)
		}
	}
	return nil
}

func (s *server) HumiditySensor(req *sensorpb.SensorRequest,
	stream sensorpb.Sensor_HumiditySensorServer) error {

	for {
		time.Sleep(time.Second * 2)

		humd := s.Sensor.GetHumiditySensor()

		err := stream.Send(&sensorpb.SensorResponse{Value: humd})
		if err != nil {
			log.Println("Error sending metric message ", err)
		}
	}
	return nil
}

var (
	port int = 8080
)

func main() {

	sns := sensor.NewSensor()

	sns.StartMonitoring()

	addr := fmt.Sprintf("0.0.0.0:%d", port)

	lis, err := net.Listen("tcp", addr)

	if err != nil {
		log.Fatalf("Error while listening : %v", err)
	}

	s := grpc.NewServer()
	sensorpb.RegisterSensorServer(s, &server{Sensor: sns})

	log.Printf("Starting server in port :%d\n", port)

	if err := s.Serve(lis); err != nil {
		log.Fatalf("Error while serving : %v", err)
	}

}
  • First, we have modified our Server type to have an instance of Sensor so we can call methods to get metrics. And we have changed TempSensor and HumiditySensor functions as well. Let’s take a look at one of those functions.
  • In the TempSensor function, we access the Sensor instance and in an infinite loop, we call the GetHumiditySensor function to get the latest value. And we sleep for 2 seconds because we know the value updates every 2 seconds. Now that we have the latest value to send through the stream, we call stream.Send function and then pass in a SensorResponse with our metric assigned to the Value field.
  • In the main function, we create an instance of Sensor by calling the NewSensor function and then we call StartMoniotirng function to start generating out “metrics”. Then we assign an instance of Sensor (sns variable) we created into the Sensor field in the server when we register the SensorServer.

Now that our server is complete let’s talk about creating a JavaScript client that can display our data in real-time.

Creating the js client Link to heading

I’ll use create-react-app to set up our react app.

npx create-react-app js-client
  • Now just like we did previously for Go, we will need to generate the client and server code for Javascript. We can use our sensor.proto file again for this. Let’s create a directory called sensorpb inside the jsclient/src directory to store our generated files. But since our client will be a browser client so we will have to use grpc-web. Before we talk about this more let’s talk about a big problem we have. Most modern browsers don’t support HTTP/2 ( yet ). Since gRPC uses HTTP/2 we need to figure out a way to make our browser client communicate with our gRPC server. grpc-web makes this somewhat possible. grpc-web allows us to use HTTP/1 with a proxy such as Envoy which helps us to translate HTTP/1 to HTTP/2 so that we can still communicate with the gRPC server.
  • Make sure you have the protoc-gen-grpc-web plugin installed → https://github.com/grpc/grpc-web installed
  • And then run this command to generate the code.
protoc sensor.proto \
--js_out=import_style=commonjs,binary:./../js-client/src/sensorpb \
--grpc-web_out=import_style=commonjs,mode=grpcwebtext:./../js-client/src/sensorpb
  • In the js-client/src we can find sensor_grpc_web_pb.js and sensor_pb.js files generated.
  • Let’s clean out the App.js until we are just left with a plain component.
import React from 'react';
import './App.css';

function App() {
  return (
    
  );
}

export default App;

Great! Let’s stop working on the client for a bit and focus on setting up our proxy because we will need it to fetch data from our server.

Now let’s work on setting up the Envoy proxy. Again, the reason for this is because we want our js-client to talk to Envoy first using HTTP/1 so it can translate those requests to our gRPC server by translating to HTTP/2.

Setting up Envoy Link to heading

  • I’ll be using Docker to get an Envoy proxy up and running.
  • We will have to define envoy.yaml with the configurations we need.
admin:
    access_log_path: /tmp/admin_access.log
    address:
      socket_address: { address: 0.0.0.0, port_value: 9901 }
  
  static_resources:
    listeners:
    - name: listener_0
      address:
        socket_address: { address: 0.0.0.0, port_value: 8000 }
      filter_chains:
      - filters:
        - name: envoy.http_connection_manager
          config:
            codec_type: auto
            stat_prefix: ingress_http
            route_config:
              name: local_route
              virtual_hosts:
              - name: local_service
                domains: ["*"]
                routes:
                - match: { prefix: "/" }
                  route:
                    cluster: sensor_service
                    max_grpc_timeout: 0s
                cors:
                  allow_origin:
                  - "*"
                  allow_methods: GET, PUT, DELETE, POST, OPTIONS
                  allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
                  max_age: "1728000"
                  expose_headers: custom-header-1,grpc-status,grpc-message
            http_filters:
            - name: envoy.grpc_web
            - name: envoy.cors
            - name: envoy.router
    clusters:
    - name: sensor_service
      connect_timeout: 0.25s
      type: logical_dns
      http2_protocol_options: {}
      lb_policy: round_robin
      hosts: [{ socket_address: { address: localhost, port_value: 8080 }}]
  • This is a slight modification to the example provided in the grpc-web repository. But I’ll do a quick walkthrough of the configuration file to explain the most important things.
  • Clusters are a group of upstream hosts that accept traffic. In this case, it’s our gRPC server. Therefore we will have to set the hosts field to point to our server address in this case [localhost:8080](<http://localhost:8080>)
  • Listeners can accept downstream connections. In this case, it’s our js-client
  • Filters essentially add extra features. There are many types of filters. For example, Listener filters allow you to manipulate metadata of Layer 4 connections during the initial connection phase. Network filters allow you to manipulate Layer 4 layer connections as well. HTTP filters allow you to manipulate HTTP requests and responses while operating at Layer 7.
  • Routes allow you to match virtual hosts to clusters and create traffic shifting rules. For this example, we are using a static definition of a route but these also can be defined dynamically via the route discovery service ( https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/rds )
  • So in that envoy.yaml file, we are essentially asking Envoy to run a listener on port 8000 that listens to downstream traffic ( this is the port our js-client will be pointing to ). And then direct any traffic that comes to it to the sensor_service which is our gRPC server running on port 8080.

alt text

  • Now let’s create another directory called envoy in the root of the project and then store the envoy.yaml file in there. Now the directory structure will look like this.
-- protos    
  - sensor.proto 
-- server   
  -- sensorpb   
  -- sensor 
-- js-client   
  -- sensorpb 
-- envoy

And then in the same directory let’s create a Dockerfile to also include the envoy.yaml file that we just created.

FROM envoyproxy/envoy:v1.12.2

COPY ./envoy.yaml /etc/envoy/envoy.yaml

CMD /usr/local/bin/envoy -c /etc/envoy/envoy.yaml

Let’s build the Docker image now.

docker build -t grpc-medium-envoy:1.0 .

Now let’s run it.

docker run --network=host grpc-medium-envoy:1.0

Now that we have Envoy up and running let’s get back to work on our js-client

js-client continued … Link to heading

First, let’s install some dependencies.

npm install grpc-web --save
npm install google-protobuf --save

Let’s import SensorRequest and SensorResponse from sensor_pb.js and SensorClient from sensor_grpc_web_pb.js and create a variable for the client.

import { SensorRequest , SensorResponse } from "./sensorpb/sensor_pb"
import { SensorClient} from "./sensorpb/sensor_grpc_web_pb"

var client = new SensorClient('http://localhost:8000')
  • Note how the address that we provided is pointing to port 8000 , not the port where our gRPC server is running. That’s because we want our js-client to talk to the Envoy proxy instead.
  • Let’s try to get our temperature value displayed first. For this section, I will assume that you have a basic understanding of React hooks. You can read about them more here ( https://reactjs.org/docs/hooks-reference.html )
const [temp, setTemp] = useState(-9999);

And then let’s create a function to read from the stream. In that function let’s first create a sensor request. And then let’s create stream variable by calling theclient.tempSensorfunction. And pass the sensorRequest in.

var sensorRequest = new SensorRequest() 
var stream = client.tempSensor(sensorRequest,{})

Finally, let’s call stream.on and then pass in a callback function to handle the response that we get from the stream. And inside the call back function let’s call setTemp function and pass in the value from response.getValue() which is the value that was read from the stream.

const getTemp = () => {
  var sensorRequest = new SensorRequest()
  var stream = client.tempSensor(sensorRequest,{})
  stream.on('data', function(response){
     setTemp(response.getValue())
  });
};

That way we will be updating the state every time a new value is read from the stream. And then let’s call the getTemp() function inside the useEffect hook.

useEffect(()=>{
   getTemp()
},[]);

Note how we pass an empty array for the second argument in the useEffect hook. The reason for this is because when you pass in an empty array as the second argument, it almost works like the componentDidMount() life cycle function. Meaning it will run once when the component renders and mounts for the first time. And then when the stream.on function gets invoked it will keep updating the state every time there’s a new value. This causes the page to render a new value every time the state changes.

import React, { useState, useEffect } from 'react';
import './App.css';

import { SensorRequest  } from "./sensorpb/sensor_pb"
import { SensorClient} from "./sensorpb/sensor_grpc_web_pb"

var client = new SensorClient('http://localhost:8000')
function App() {
  const [temp, setTemp] = useState(-9999);
  
  const getTemp = () => {
    console.log("called")

    var sensorRequest = new SensorRequest()
    var stream = client.tempSensor(sensorRequest,{})

    stream.on('data', function(response){
        setTemp(response.getValue())
    });
  };

  useEffect(()=>{
    getTemp()
  },[]);

  return (
    <div>
      Temperature : {temp} F
    </div>
  );
}

export default App;

Now let’s start the app by running npm start . You might see a compilation error. You can get around this by adding /* eslint-disable */ at the top of sensor_grpc_web_pb.js file and the sensor_pb.js file. If you are interested you can learn about this error from here https://github.com/facebook/create-react-app/issues/7295.

alt text

Let’s try running npm start again.

alt text

Great !! We can see that the temperature value is changing every 5 seconds as we expected. Now let’s do the same for humidity as well.

alt text

As you can see the humidity value changes every 2 seconds while the temperature changes only every 5 seconds just as we expected. The finalized App.js file will look like this.

import React, { useState, useEffect } from 'react';
import './App.css';

import { SensorRequest  } from "./sensorpb/sensor_pb"
import { SensorClient} from "./sensorpb/sensor_grpc_web_pb"

var client = new SensorClient('http://localhost:8000')
function App() {
  const [temp, setTemp] = useState(-9999);
  const [humidity , setHumidity] = useState(-99999)
  

  const getTemp = () => {
    console.log("called")

    var sensorRequest = new SensorRequest()
    var stream = client.tempSensor(sensorRequest,{})

    stream.on('data', function(response){
        setTemp(response.getValue())
    });
  };

  const getHumidity = () => {
    var sensorRequest = new SensorRequest()
    var stream = client.humiditySensor(sensorRequest,{})

    stream.on('data',function(response){
      setHumidity(response.getValue())
    })
  }

  useEffect(()=>{
    getTemp()
  },[]);

  useEffect(()=>{
    getHumidity()
  },[]);

  return (
    <div>
      Temperature : {temp} F
      <br/>
      Humidity : {humidity} %
    </div>
  );
}

export default App;

And that’s it! If you want to make this pretty, with some CSS magic you can end up with something like this.

alt text

The temperature box becomes red when the temperature is greater than 90 or less than 30. The humidity box changes it’s color to red when the humidity is above 80.