Speaking with Cordova – Text To Speech and Voice Recognition using the Ionic Framework

January 8, 2016By 14 Comments

The following is a guest post by Simon Reimler

Using Siri, Google Now or Cortana to talk with your smartphone is pretty normal these days. But have you ever considered creating your own Voice Dialog? And would you think this is possible within a hybrid app using Cordova and the Ionic Framework? Let me prove it to you.

In this article I will show you how to build text to speech voice output and speech recognition input using Cordova plugins with the Ionic Framework.

Setting up the App

We start with a blank Ionic app and install our two Cordova plugins, so go ahead and run:

ionic start cordova-speech blank
cd cordova-speech
cordova plugin add cordova-plugin-tts
cordova plugin add https://github.com/macdonst/SpeechRecognitionPlugin

This creates a new folder cordova-speech with a clean Ionic Framework app and adds two Cordova plugins we need for our example:

The first plugin is a cordova wrapper for the native text to speech output functions. This plugin uses the AVSpeechSynthesizer on iOS and the android.speech.tts.TextToSpeech package on Android, so it’s really a simple mapper of the best native functions for giving out text. This plugin also seems to work for Windows phone, but I had no chance to test this part. If one of you has a Windows Phone why not give it a try and leave a comment with your findings below?

The second plugin is a cordova wrapper for speech recognition. On Android this plugin uses the native android.speech.SpeechRecognizer package, on iOS it’s a bit harder to achieve this task so the author used the iSpeech SDK which you can test for free for now. If you plan to use this SDK in production additional costs may arise.

Besides those two plugins we don’t need any more tools or fancy components inside our app!

Crafting our Speech View

The view of our app will be quite simple. First of all we need an input field for some text which our app can speak later on.

Below this field we need two buttons, the first is to call our text to speech action so the text we have inserted get’s read to our user. The second button triggers our speech recognition dialog and listens for whatever we say.

The recorded text (or what the app thinks we have said) will be printed below our other components, just to see how precise the recognition is. This can vary quite a bit, depending on surrounding sounds, how clear you speak, how difficult the words and sentences are and so on. You can play around with the results later and leave your opinions about this plugin below!

For now go ahead and open your www/index.html and replace the current body with:

<body ng-app="starter">
    <ion-pane ng-controller="AppCtrl">
      <ion-header-bar class="bar-stable">
        <h1 class="title">Cordova Text-to-Speech</h1>
      </ion-header-bar>
      <ion-content class="padding">
        <div class="list list-inset">
          <label class="item item-input">
            <i class="icon ion-speakerphone placeholder-icon"></i>
            <input type="text" placeholder="Let me speak..." ng-model="data.speechText">
          </label>
        </div>
        <button class="button button-full button-positive" ng-click="speakText()">
          Speak!
        </button>

        <button class="button button-full button-positive"ng-click="record()">
          Record
        </button>
        <div class="card">
          <div class="item item-text-wrap">
            {{recognizedText}}
          </div>
        </div>
      </ion-content>
    </ion-pane>
</body>

There is really no magic in this view so let’s head to the real bread and butter of this article!

Using the Cordova Speech Plugins

We have already installed all the needed plugins and don’t have to import any headers or anything else besides. Therefore, open your www/js/app.js and replace everything inside with:

angular.module('starter', ['ionic'])

.controller('AppCtrl', function($scope) {
  $scope.data = {
    speechText: ''
  };
  $scope.recognizedText = '';

  $scope.speakText = function() {
    TTS.speak({
           text: $scope.data.speechText,
           locale: 'en-GB',
           rate: 1.5
       }, function () {
           // Do Something after success
       }, function (reason) {
           // Handle the error case
       });
  };

  $scope.record = function() {
    var recognition = new SpeechRecognition();
    recognition.onresult = function(event) {
        if (event.results.length > 0) {
            $scope.recognizedText = event.results[0][0].transcript;
            $scope.$apply()
        }
    };
    recognition.start();
  };
});

As you can see there is not that much code needed for quite cool features! Let’s take a closer look at our controller implementation.

We create a data object which will is connected to our input field. We need to make it an object so the value is updated correctly using ng-model.

If we want our inserted text to be spoken, the speakText() function get’s called. We use the Cordova plugin by calling TTS.speak() and also pass an object with some properties. In this example we pass the text (which you should always do) and also a language and a rate which defines how fast the text is spoken. You can create quite funny outputs playing around with these values, trust me I had my fun creating this article!
Finally we can also specify a success and error callback, which I left empty for now because handling those errors is very custom for every app.

Our voice recognition starts on calling record(). Here we create a new instance of a SpeechRecognition object. After creating it, we assign a function to the onresult property of the object. This function will be called once the recognizer thinks he is done listening to you talking.

We check if we actually have any result, and if that’s the case we apply the text our recognizer understood to our scope variable which appears below our other view elements.

After setting the result block we only have to start our recognition and the app starts listening, that’s all!

Conclusion

In this step-by-step tutorial I explained how to use two Cordova plugins for text to speech and voice recognition inside an Ionic app. You might think that these two features are hard to implement, but given the great wrappers they are super cheap and easy to include in your hybrid app.

If you want to learn more about building great mobile Apps make sure to check out my Blog Devdactic and follow me on Twitter @schlimmson!

Cheers,
Simon

Filed in: CordovaionicPhoneGap Tags:

About the Author ()

Teamlead Frontend & Mobile at arvato Bertelsmann | Skilled with Swift, Ionic Framework, AngularJS, Java, Python and Javascript | Blogging and Teaching Mobile Development, Productivity and Personal Development

Comments (14)

Trackback URL | Comments RSS Feed

  1. Thank you very much for the tutorial, I will try it right away. I was looking for it help me a lot to develop my project.

  2. Vikas says:

    Nice experiment..

  3. Jesse Monroy says:

    Holly,
    You may want to add this to your blog. I have recently compiled a version history for Phonegap and Cordova. I noted a similar blog from Simon Macdonald. The story was posted on the ‘Phonegap Network’

    “Anna Has Been Pulled From the Play Store – February 6, 2014”
    Currently, the abstract is on http://phonegap.com/blog/page29/

    Anna Has Been Pulled From the Play Store
    http://simonmacdonald.blogspot.com/2014/02/anna-has-been-pulled-from-play-store.html
    “””
    Back in September of 2012 I posted up my sample app called Anna which
    highlighted the power of doing speech recognition on mobile devices
    using PhoneGap. Today I’ve had to pull the app from the Play Store
    because of this infringement notice from Wolfram Alpha.
    “””
    The source code is still available online.

    Jesse

  4. Jcesarmobile says:

    iSpeech is free to use on production too (but their logo will be displayed). That’s why we used it for the plugin, but we are open to alternatives.

  5. Genie says:

    I can create a brand new cordova project just like you outlined and get it to run. But, when I try to add the code to an existing project that I want to voice enable, nothing happens.

  6. Genie says:

    I found a little bug and don’t know how to fix it. I have the code snippet above now working in my app. Well, sort of. I can run the app and it will speak text that I enter correctly. But, when I record, the very first time through it does not capture and report any text. Only on the second time hitting record will it process correctly. It does this in my app and in the app outlined above when I create it.

    Ex. Start app. Hit record – it beeps, records, and beeps complete. No text is written to screen. Hit record again immediately. It beeps – records, and beeps complete. Then this text will show up on screen and continue to function properly. It is only the first time recording where it is not working.

    Any advice?

    Thanks!

    • Simon Reimler says:

      Sounds pretty weird! Have you tried to debug/log what is coming back in the completion blog when you record for the first time? Would be interesting to see if maybe no text comes back, than this might be some problem of the plugin.

  7. Dave Brumbaugh says:

    Hmm. Well, I’m a newb to ionic and cordova, but have had some success using other plug-ins. However, can’t seem to get TTS or SpeechRecognition to be recognized as variables in the JS (Error: Can’t find variable: TTS and Error: Can’t find variable: SpeechRecognition) when serving up with phonegap. Any hints? Using Windows and VS for development but haven’t had any issues with other plug-ins. Thanks. Dave.

  8. shibani says:

    Hi

    I dont get any output .If i give text there is no speech output.Nor,if i record there is text.How can i Debug.
    Please Help!!.

    Thanks
    Shibani S

  9. Tom R says:

    This is great,

    But the recognition only works once for me when i open the app the first time, and never again.
    Maybe a plugin problem? Any Advice?

    Not getting anything out of the debugger. The recognition.start() fires a second time and i hear the sounds (Start & stop) however the onresult event only fires the first time.

    • Tom R says:

      The issue i was having was related to this… had to modify the plugin to get around it.
      Seems like the recognizer sometimes just kicks out an error instantly that just needs to be ignored.

Leave a Reply