Home | Send Feedback

Consume Protocol Buffer messages with Ionic 4

Published: January 14, 2017  •  Updated: December 01, 2018  •  ionic4, spring, java, javascript

In the previous post we created an example that sends Protocol Buffer messages from one Java application to another.

In this post we look at an example that sends Protocol Buffer messages from a Java Spring Boot server to a JavaScript application written in Ionic. To compare Protocol Buffer with JSON the server also provides an endpoint that returns JSON.

For this example we use real data from The United States Geological Survey agency. They provide lists of current earthquakes in different data formats that can be accessed with simple HTTP get requests. The Spring Boot application fetches this data and provides endpoints for the JavaScript application.

The applications is going to use this Protocol Buffer definition file.

syntax = "proto3";

option java_package = "ch.rasc.protobuf";

message Earthquake {
  string id = 1;
  string time = 2;
  double latitude = 3;
  double longitude = 4;
  float depth = 5;
  float mag = 6;
  string place = 7;
  string magType = 8;
}

message Earthquakes {
  repeated Earthquake earthquakes = 1;
}

Earthquake.proto

Every reported earthquake is modelled with an Earthquake message and Earthquakes is a list of zero to n Earthquake messages and represents the root of the server response


Server

We start building the Spring Boot application with the Spring Initializr website. We only need the Web dependency for this example.

Then we add the Protobuf library and the protoc plugin to the pom.xml. The protobuf-java-util library contains support for converting a Protocol Buffer message to JSON. This is very convenient for our demo application so we can use the same generated data class for the JSON and Protobuf endpoint.

    <dependency>
      <groupId>com.google.protobuf</groupId>
      <artifactId>protobuf-java</artifactId>
      <version>3.6.1</version>
    </dependency>
    <dependency>
      <groupId>com.google.protobuf</groupId>
      <artifactId>protobuf-java-util</artifactId>
      <version>3.6.1</version>
    </dependency>    

pom.xml

      <plugin>
        <groupId>com.github.os72</groupId>
        <artifactId>protoc-jar-maven-plugin</artifactId>
        <version>3.6.0.1</version>
        <executions>
          <execution>
            <phase>generate-sources</phase>
            <goals>
              <goal>run</goal>
            </goals>
            <configuration>
              <protocVersion>3.6.1</protocVersion>
              <inputDirectories>
                <include>src/main/protobuf</include>
              </inputDirectories>
            </configuration>
          </execution>
        </executions>
      </plugin>

pom.xml

Next we copy our proto file to src/main/protobuf/ and generate the Java code with

mvn generate-source

Spring 5 supports Protocol Buffers version 2 and 3 out of the box. All we need to do is creating a bean of type ProtobufHttpMessageConverter. You can put this in any @Configuration class.

  @Bean
  public ProtobufHttpMessageConverter protobufHttpMessageConverter() {
    return new ProtobufHttpMessageConverter();
  }

Server.java

Next we create a service class that fetches the CSV file from the usgs.gov site, converts it into objects and stores them in an instance variable. For fetching the data it uses the OkHttp library. With the help of the univocity-parsers library the application parses the CSV file and converts it into objects.

    Request request = new Request.Builder()
        .url("https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.csv")
        .build();

    try (Response response = this.httpClient.newCall(request).execute();
        ResponseBody responseBody = response.body()) {
      if (responseBody != null) {
        String rawData = responseBody.string();

        CsvParserSettings settings = new CsvParserSettings();
        settings.setHeaderExtractionEnabled(true);
        settings.setLineSeparatorDetectionEnabled(true);
        CsvParser parser = new CsvParser(settings);
        List<Record> records = parser.parseAllRecords(new StringReader(rawData));

        Earthquakes.Builder earthquakesBuilder = Earthquakes.newBuilder();
        for (Record record : records) {
          Earthquake.Builder builder = Earthquake.newBuilder()
              .setId(record.getString("id")).setTime(record.getString("time"))
              .setLatitude(record.getDouble("latitude"))
              .setLongitude(record.getDouble("longitude"))
              .setDepth(record.getFloat("depth")).setPlace(record.getString("place"));

          Float mag = record.getFloat("mag");
          if (mag != null) {
            builder.setMag(mag);
          }

          String magType = record.getString("magType");
          if (magType != null) {
            builder.setMagType(magType);
          }

          earthquakesBuilder.addEarthquakes(builder);
        }
        this.earthquakes = earthquakesBuilder.build();
      }
    }

EarthquakeDb.java

Then we create the Controller class that provides a JSON and a Protocol Buffer endpoint. The Spring Boot application contains a scheduled task that reads the data from the USGS site every hour. The /refresh endpoint allows us to force an update, so we don't have to wait an hour.

Because both endpoints handle the same URL (/earthquakes), we need to make them distinguishable. For this example I added the produces attribute to the Protocol Buffer endpoint. Spring takes the Accept HTTP header of incoming requests and compares it with the value of the produces attribute, and if it matches calls that method. Another approach would be to use different path names.

With the help of the JsonFormat class from the protobuf-java-util package we convert the Protobuf object into JSON.

  @GetMapping("/earthquakes")
  public String getEarthquakesJson() throws InvalidProtocolBufferException {
    return JsonFormat.printer().print(this.earthquakeDb.getEarthquakes());
  }

  @GetMapping(path = "/earthquakes", produces = "application/x-protobuf")
  public Earthquakes getEarthquakesProtobuf() {
    return this.earthquakeDb.getEarthquakes();
  }

  @GetMapping(path = "/refresh")
  @ResponseStatus(HttpStatus.NO_CONTENT)
  public void refresh() throws IOException {
    this.earthquakeDb.readEarthquakeData();
  }

EarthquakeController.java

We also add the Protobuf mime type to the compression mime types configuration. Spring Boot compresses all HTTP responses that match with one of these mime types on the fly with gzip before sending them back to the client.

server.compression.enabled=true
server.compression.mime-types=application/json,application/x-protobuf

application.properties


Client

The client is based on the tabs Ionic 4 starter application. For consuming Protobuf messages we add the ProtoBuf.js library to the project.

npm install protobufjs

Next we generate JavaScript and TypeScript code from our proto file. ProtoBuf.js provides a command line tool to create these classes.
For that we add a new task to the package.json file

    "pbts": "pbjs -t static-module ../server/src/main/protobuf/Earthquake.proto -o src/app/protos/earthquake.js && pbts --no-comments src/app/protos/earthquake.js -o src/app/protos/earthquake.d.ts"

package.json

Now we can call the task with npm run pbts and it is creating two files in the src/app/protos/ folder (Make sure that you create the folder first).
The earthquake.js is the implementation and contains methods to serialize and deserialize binary protobuf messages into objects. earthquake.d.ts is a TypeScript definition file. For each Protobuf object it creates an interface and a class (IEarthquake and Earthquake, IEarthquakes and Earthquakes).

Next we create a service which contains methods that call the endpoints of our server.

import {Injectable} from '@angular/core';
import {Observable, throwError} from 'rxjs';
import {catchError, map} from 'rxjs/operators';
import {HttpClient, HttpHeaders} from '@angular/common/http';
import {Earthquakes, IEarthquake} from './protos/earthquake';
import {environment} from '../environments/environment';

@Injectable({
  providedIn: 'root'
})
export class EarthquakeService {

  constructor(private readonly http: HttpClient) {
  }

  refresh(): Observable<void> {
    return this.http.get<void>(`${environment.SERVER_URL}/refresh`);
  }

  fetchJson(): Observable<IEarthquake[]> {
    return this.http.get<any>(`${environment.SERVER_URL}/earthquakes`)
      .pipe(map(res => res.earthquakes), catchError(e => this.handleError(e)));
  }

  fetchProtobuf(): Observable<IEarthquake[]> {
    const headers = new HttpHeaders({'Accept': 'application/x-protobuf'});
    return this.http.get(`${environment.SERVER_URL}/earthquakes`, {headers, responseType: 'arraybuffer'})
      .pipe(map(res => this.parseProtobuf(res)),
        catchError(this.handleError)
      );
  }

  parseProtobuf(response: ArrayBuffer): IEarthquake[] {
    console.time('decodeprotobuf');
    const earthquakes = Earthquakes.decode(new Uint8Array(response));
    console.timeEnd('decodeprotobuf');
    return earthquakes.earthquakes;
  }

  handleError(error): Observable<any> {
    console.error(error);
    return throwError(error || 'Server error');
  }
}

earthquake.service.ts

The refresh() method calls the /refresh endpoint and triggers a refresh of the data stored on the server.
The fetchJson() method requests the data in the JSON format.
The fetchProtobuf() function fetches the data in the binary Protocol Buffers format. We need to specify the Accept header with the value application/x-protobuf that matches the produces attribute in the Spring Boot controller. Because we know that the server returns a binary message and we need that message in the form of an ArrayBuffer we add a parameter to the GET call {responseType: 'arraybuffer'}. When the response comes back from the server, the method maps it to objects with the decode method from the generated Earthquakes class.

Because in our app we use two tabs that show the same content, we create a component to reuse some code. This component represents one row in the item list and shows the detail for one earthquake.

<div class="row">
  <h1>
    {{earthquake.mag | number:'1.1-1'}}
  </h1>
  <div class="place">
    <h2>{{earthquake.place}}</h2>
    <ion-note>{{earthquake.time | date:'MMM d HH:mm'}}</ion-note>
  </div>
  <div class="depth">{{earthquake.depth | number}} km</div>
</div>

detail.component.html

import {Component, Input} from '@angular/core';
import {IEarthquake} from '../protos/earthquake';

@Component({
  selector: 'app-detail',
  templateUrl: './detail.component.html',
  styleUrls: ['./detail.component.scss']
})
export class DetailComponent {

  @Input()
  earthquake: IEarthquake;

}

detail.component.ts

Then we create two pages. One shows the earthquakes read from the JSON endpoint and the other tab shows the same data but read from the Protocol Buffer message endpoint.

<ion-header>
  <ion-toolbar>
    <ion-title>JSON</ion-title>
  </ion-toolbar>
</ion-header>

<ion-content>
  <ion-refresher slot="fixed" (ionRefresh)="doRefresh($event)">
    <ion-refresher-content></ion-refresher-content>
  </ion-refresher>

  <ion-list>
    <ion-item *ngFor="let earthquake of earthquakes">
      <ion-label>
        <app-detail [earthquake]="earthquake"></app-detail>
      </ion-label>
    </ion-item>
  </ion-list>
</ion-content>

json.page.html

import {Component, OnInit} from '@angular/core';
import {EarthquakeService} from '../earthquake.service';
import {IEarthquake} from '../protos/earthquake';

@Component({
  selector: 'app-json',
  templateUrl: './json.page.html',
  styleUrls: ['./json.page.scss']
})
export class JsonPage implements OnInit {

  earthquakes: IEarthquake[];

  constructor(private readonly earthquakeService: EarthquakeService) {
  }

  doRefresh(event) {
    this.earthquakeService.refresh().subscribe(() => {
      this.earthquakeService
        .fetchJson()
        .subscribe(data => {
          this.earthquakes = data;
          event.target.complete();
        });
    });
  }

  ngOnInit() {
    this.earthquakeService.fetchJson().subscribe(data => this.earthquakes = data);
  }

}

json.page.ts

The template and code for the Protobuf page looks very similar. Notable difference is the call to this.earthquakeService.fetchProtobuf() instead of fetchJson().

To test the application you have to start the server first. Either start the class Application class in your IDE or start the server with the maven spring boot plugin from the shell or command prompt with

//Linux and macOS
./mvnw spring-boot:run

// Windows
.\mvnw.cmd spring-boot:run

Then start the client with ionic serve. The browser should automatically open with the URL http://localhost:8100. If everything was successful, you should see a list of earthquakes that happened during the last 24 hours.

You find the complete project on GitHub.