Home | Send Feedback

Adding speech recognition to an Ionic 4 App

Published: December 18, 2017  •  Updated: December 04, 2018  •  ionic4, javascript, java, spring

In this blog post I show you three different ways how you can incorporate speech recognition into a web app, in this case an Ionic app.

As example, I created a simple movie search database with an Ionic front and a Java / Spring Boot back end. The user can search for movie titles by speaking the name into the microphone and let a speech recognition library transcribe it into text. The web app then sends a search request to the Spring Boot application where it searches for matching movies stored in a MongoDB database.


Test data

Before I started with the app, I needed some data about movies to insert into the database. The go to address for movie information is the Internet Movie Database and they provide a set of raw data files for free.
You find all the information about the data files here: https://www.imdb.com/interfaces/
Notice that you can use the data files, you download there, only for personal and non-commercial use.

The files are hosted on Amazon S3. For downloading and importing the data files I wrote a Java application. The program downloads the files from S3 with the Java S3 library, parses the files with the Univocity library (the data in the files are stored as tab separated values) and then inserts the data into a MongoDB database.

You find the complete code of the importer on GitHub: src/main/java/ch/rasc/speechsearch/ImportImdbData.java To download the files you need an Amazon AWS account and credentials for accessing S3.


Client

The client is written with the Ionic framework and based on the blank starter template. The app displays the movies as a list of cards. At the bottom you find three buttons to start the speech search with the three mechanisms I show you in this blog post.

App


Server

The server is written in Java with Spring and Spring Boot. The search controller is a RestController that handles the search requests from the client

  @GetMapping("/search")
  public List<Movie> search(@RequestParam("term") List<String> searchTerms) {

    Set<Movie> results = new HashSet<>();
    MongoCollection<Document> moviesCollection = this.mongoDatabase
        .getCollection("movies");

    MongoCollection<Document> actorCollection = this.mongoDatabase
        .getCollection("actors");

    List<Bson> orQueries = new ArrayList<>();
    for (String term : searchTerms) {
      orQueries.add(Filters.regex("primaryTitle", term + ".*", "i"));
    }

    try (MongoCursor<Document> cursor = moviesCollection.find(Filters.or(orQueries))
        .limit(20).iterator()) {
      while (cursor.hasNext()) {
        Document doc = cursor.next();
        Movie movie = new Movie();
        movie.id = doc.getString("_id");
        movie.title = doc.getString("primaryTitle");
        movie.adult = doc.getBoolean("adultMovie", false);
        movie.genres = doc.getString("genres");
        movie.runtimeMinutes = doc.getInteger("runtimeMinutes", 0);
        movie.actors = getActors(actorCollection,
            (List<String>) doc.get("actors"));
        results.add(movie);
      }
    }

    return results.stream().collect(Collectors.toList());
  }

SearchController.java

The search method takes a list of search terms and starts a regular expression search over the movie title in the movies collection. The application limits the results to 20 entries and uses a cursor to iterate over the returned entries. It then converts the matching documents to POJOs (Movie) and returns them in a list to the client.

1. Cordova speech recognition plugin

When you create a Cordova application with Ionic and the browser does not support the functionality you need you look for a native plugin. Fortunately for us there is a plugin that supports speech recognition:
https://github.com/pbakondy/cordova-plugin-speechrecognition

ionic cordova plugin add cordova-plugin-speechrecognition

The method that handles the search looks a bit complicated because the API uses callbacks instead of Promises.

  searchCordova() {
    window['plugins'].speechRecognition.hasPermission(permission => {

      if (!permission) {
        window['plugins'].speechRecognition.requestPermission(_ => {
          window['plugins'].speechRecognition.startListening(terms => {
            if (terms && terms.length > 0) {
              this.movieSearch([terms[0]]);
            } else {
              this.movieSearch(terms);
            }
          });
        });
      } else {
        window['plugins'].speechRecognition.startListening(terms => {
          if (terms && terms.length > 0) {
            this.movieSearch([terms[0]]);
          } else {
            this.movieSearch(terms);
          }
        });
      }
    });
  }

home.page.ts

The method first checks if the app has permission to access the microphone of the device. If not, it requests the permission. The device present a dialog to the user that asks for permission.

ask1 ask2

Then the method calls the startListening method of the plugin. This method starts listening, the user can now speak his search request into the microphone and the speech plugin automatically transcribes the spoken words into text and returns that back to the application.
On Android this functionality needs Internet access because the plugin sends the recorded speech sample to Google where a service transcribes it into text. According to the documentation the plugin works similar on an iOS device. It records the speech sample and sends it to an Apple server.

After the plugin returned an array of transcribed text strings, the searchCordova methods calls movieSearch.

  async movieSearch(searchTerms: string[]) {
    if (searchTerms && searchTerms.length > 0) {

      const loading = await this.loadingCtrl.create({
        message: 'Please wait...'
      });
      loading.present();

      this.matches = searchTerms;
      let queryParams = '';
      searchTerms.forEach(term => {
        queryParams += `term=${term}&`;
      });
      const response = await fetch(`${environment.serverUrl}/search?${queryParams}`);
      this.movies = await response.json();
      loading.dismiss();
      this.changeDetectorRef.detectChanges();
    } else {
      this.movies = [];
    }
  }

home.page.ts

This method is responsible for sending the search terms to the Spring Boot application and handles the response.

Because this example depends on a Cordova plugin it will not work in a desktop browser and you need to run the app on a device or in an emulator to test it.

2. Web Speech API

Next I looked for a way that works without any native plugins. Good news is there is a specification that provides this functionality, the Web Speech API, bad news is that at the moment (December 2018) only Google implemented this into their Chrome browser (https://caniuse.com/#search=speech).
According to information I found on the Internet the feature is "under consideration" at Microsoft and Mozilla.

The API is easy to use. It's callback based and the app has to implement a few callback handlers. The searchWebSpeech method first checks if the speech recognition object is available in the window object. Then it instantiates the webkitSpeechRecognition object that handles the speech recognition.

  searchWebSpeech() {
    if (!('webkitSpeechRecognition' in window)) {
      return;
    }

    const recognition = new webkitSpeechRecognition();
    recognition.continuous = false;

    recognition.onstart = () => {
      this.isWebSpeechRecording = true;
      this.changeDetectorRef.detectChanges();
    };

    recognition.onerror = event => console.log('error', event);
    recognition.onend = () => {
      this.isWebSpeechRecording = false;
      this.changeDetectorRef.detectChanges();
    };

    recognition.onresult = event => {
      const terms = [];
      if (event.results) {
        for (const result of event.results) {
          for (const ra of result) {
            terms.push(ra.transcript);
          }
        }
      }

      this.movieSearch(terms);
    };

    recognition.start();
  }

home.page.ts

Because I also wanted to disable the button when the recording is running, the onstart and onend handler set and reset a flag.

The API automatically recognises when the user stops speaking, it then sends the recorded speech sample to Google where it gets transcribed to text. This is what the onresult handler receives as parameter. In this handler the code collects all the transcriptions into one array and calls the movieSearch method that sends the request to the server.

I tested this on Chrome on a Windows Desktop and it works very well. I don't know if the API ever gets implemented in other browsers, but it would a nice addition to the Web platform tool belt.

3. Recording with WebRTC and sending it to the Google Cloud Speech API

The second approach works well but is limited to Chrome and I was more interested in a solution that works in most modern browsers without any additional plugin.

With WebRTC it's not that complicated to record an audio stream in almost all modern browsers. There is a very good library available that smooths out the different WebRTC implementations and is able to record audio: RecordRTC

The example I present here runs on Edge, Firefox and Chrome on a Windows computer. I haven't tested Safari but version 11 has a WebRTC implementation so I guess the example should work on Apple's browser too.

After the speech is recorded the app transfers it to our server and from there to the Google Cloud Speech-To-Text API. This is a service that transcribes spoken words into text. The first 60 minutes of recordings are free then you have to pay $0.006 for each 15 second snippet you send to the service.

Unlike the other two example the application has to handle starting and stopping the recording manually. For that the method uses a boolean instance variable and set it to true when the recording is running. The user then has to click on the Stop button when he is finished speaking.

  async searchGoogleCloudSpeech() {
    if (this.isRecording) {
      if (this.recorder) {
        this.recorder.stopRecording(async _ => {
          const recordedBlob = this.recorder.getBlob();

          const headers = new Headers();
          headers.append('Content-Type', 'application/octet-stream');

          const requestParams = {
            headers,
            method: 'POST',
            body: recordedBlob
          };
          const response = await fetch(`${environment.serverUrl}/uploadSpeech`, requestParams);
          const searchTerms = await response.json();
          this.movieSearch(searchTerms);
        });
      }
      this.isRecording = false;
    } else {
      this.isRecording = true;
      const stream = await navigator.mediaDevices.getUserMedia({video: false, audio: true});
      const options = {
        mimeType: 'audio/wav',
        recorderType: RecordRTC.StereoAudioRecorder
      };
      this.recorder = RecordRTC(stream, options);
      this.recorder.startRecording();
    }
  }

home.page.ts

When the user starts the recording the method accesses the audio stream with getUserMedia and calls the RecordRTC object with the stream as source. In this example I set the audio format to wav and uses the stereo recorder. This works fine on all three browser I tested.
When the recording stops the method receives a blob from RecordRTC that contains the recorded audio in wav format. Then it uploads the binary data to Spring boot (/uploadSpeech) and waits for the transcription to return. After that is calls the movieSearch method.

On the server I use the google-cloud-speech Java library to connect the application with the Google Cloud. The project needs this dependency in the pom.xml

    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-speech</artifactId>
      <version>0.72.0-beta</version>
    </dependency>

pom.xml

Before you can access a service in the Google Cloud, you need a credentials file. To get that you need to login into your Google Account and open the Google API Console.
There you either create a new project or select an existing one and add the Google Cloud Speech API to the project.
Then open the credentials menu and create a new service account. You can then download a JSON file that you can add to the project. Don't commit this file into your git repository, it contains sensitive information that allows anybody that has the key to access the API.
In this application I externalize the path to this credential file with a configuration property class (AppConfig)

The SearchController needs to create a SpeechClient instance that allows him to send requests to the Google Cloud Speech API. The code first reads the credential file and then creates an instance of this class.

  public SearchController(AppConfig appConfig) throws IOException {
    this.mongoClient = new MongoClient("localhost");
    this.mongoDatabase = this.mongoClient.getDatabase("imdb");

    ServiceAccountCredentials credentials = ServiceAccountCredentials.fromStream(
        Files.newInputStream(Paths.get(appConfig.getCredentialsPath())));
    SpeechSettings settings = SpeechSettings.newBuilder()
        .setCredentialsProvider(FixedCredentialsProvider.create(credentials))
        .build();
    this.speech = SpeechClient.create(settings);
  }

SearchController.java

Last piece of the puzzle is the handler for the /uploadSpeech endpoint. This method receives the bytes of the recorded speech sample in wav format and stores them in a file.

  @PostMapping("/uploadSpeech")
  public List<String> uploadSpeech(@RequestBody byte[] payloadFromWeb)
      throws Exception {

    String id = UUID.randomUUID().toString();
    Path inFile = Paths.get("./in" + id + ".wav");
    Path outFile = Paths.get("./out" + id + ".flac");

    Files.write(inFile, payloadFromWeb);

    FFmpeg ffmpeg = new FFmpeg("./ffmpeg.exe");
    FFmpegBuilder builder = new FFmpegBuilder().setInput(inFile.toString())
        .overrideOutputFiles(true).addOutput(outFile.toString())
        .setAudioSampleRate(44_100).setAudioChannels(1)
        .setAudioSampleFormat(FFmpeg.AUDIO_FORMAT_S16).setAudioCodec("flac")
        .done();

    FFmpegExecutor executor = new FFmpegExecutor(ffmpeg);
    executor.createJob(builder).run();

    byte[] payload = Files.readAllBytes(outFile);

    ByteString audioBytes = ByteString.copyFrom(payload);

    RecognitionConfig config = RecognitionConfig.newBuilder()
        .setEncoding(AudioEncoding.FLAC).setLanguageCode("en-US").build();
    RecognitionAudio audio = RecognitionAudio.newBuilder().setContent(audioBytes)
        .build();

    RecognizeResponse response = this.speech.recognize(config, audio);
    List<SpeechRecognitionResult> results = response.getResultsList();

    List<String> searchTerms = new ArrayList<>();
    for (SpeechRecognitionResult result : results) {
      SpeechRecognitionAlternative alternative = result.getAlternativesList()
          .get(0);
      searchTerms.add(alternative.getTranscript());
    }

    Files.deleteIfExists(inFile);
    Files.deleteIfExists(outFile);

    return searchTerms;
  }

SearchController.java

The problem I had here, was that the Cloud Speech API cannot handle the wav file it gets from the web app. One problem is the unsupported format and the other is the recording is in stereo but the Speech API requires mono recordings. Unfortunately I haven't found a pure Java library that is able to convert sound files. But there is a way to do that with a native application and still support multiple operating systems.

ffmpeg is a program for handling multimedia files. One task it provides is converting audio files into other formats. On the download page you find builds for many operating system.

For Windows, I downloaded the ffmpeg.exe file. To call the exe from the Java code I found a Java wrapper library that simplifies setting parameters and calling the exe.

    <dependency>
      <groupId>net.bramp.ffmpeg</groupId>
      <artifactId>ffmpeg</artifactId>
      <version>0.6.2</version>
    </dependency>

pom.xml

In the code you see how I use this library to specify the configuration parameter that convert the wav file into a mono flac file. Flac is one of the supported audio formats of the Cloud Speech API. Calling the API itself is very easy when you have the recording in a supported format. All it needs is a call to the recognize method with the binary data of the recording and a few configuration parameters like the description of the format.

RecognizeResponse response = this.speech.recognize(config, audio);

The method returns the text transcription, when the service was able to understand some words in the recording. The uploadSpeech method then sends back these strings to the Ionic app.


This concludes our journey into speech recognition land. If you are developing a Cordova app, the speech recognition plugin is the easiest way to support this use case.

The third solution is more complicated because it also depends on a server part, but supports most modern browsers and does not depend on a plugin.

The Web Speech API looks promising, but as long there is only one browser that supports it, not very useful.