Google Assistant app (Actions on Google) with Microsoft Azure Functions in C# (Serverless service by Microsoft)
Download as PPTX, PDF8 likes3,413 views
The document details the development of a Google Assistant app using Microsoft Azure Functions by a mobile app developer based in Tokyo, who is also a manga artist. It explains how Google Assistant processes commands through various technologies like NLP and machine learning, and discusses the creation of a voice changer app that gained significant interest on Twitter. The author highlights the ease of connecting Azure Functions with other services, showcasing its capabilities in integrating voice interfaces with smart assistants.
Google Assistant app (Actions on Google) with Microsoft Azure Functions in C# (Serverless service by Microsoft)
- 1. Developed Google Home app (Actions on Google) using Microsoft Azure Functions
- 2. Everyone calls me “Chomado”. Based in Tokyo, Japan. Love anime/manga Before joining Microsoft, I’d been a mobile app developer using Xamarin + Microsoft Azure I’m not only an engineer but also a manga artist:
- 3. My manga work: “Run! Coding Academy”
- 5. I love my Grandma She has presbyopia (aged eyes) and difficult to read. So I bought a smart speaker for her. I think VUI (Voice UI) is good for old-aged people.
- 6. What is “Smart Speaker”? Wireless speaker with AI assistant!
- 7. Virtual Assistant (logo) Virtual Assistant Owned by Devices Languages/Markets Microsoft Cortana Harman Kardon INVOKE English for US, Canada, Australia, India Amazon Alexa Amazon Echo English for US, UK, Ireland, Canada, India, Australia, New Zealand; German (Germany, Austria); Japanese (Japan); Google Assistant Google Home English (U.S., U.K., Canada, Australia), French (France, Canada), German, Japanese Siri Apple HomePod English for US, UK, Australia Clova Clova Friends, Clova WAVE Japanese and Korean for Japan and South Korea markets
- 8. Virtual Assistant (logo) Virtual Assistant Owned by Devices Languages/Markets Microsoft Cortana Harman Kardon INVOKE English for US, Canada, Australia, India Amazon Alexa Amazon Echo English for US, UK, Ireland, Canada, India, Australia, New Zealand; German (Germany, Austria); Japanese (Japan); Google Assistant Google Home English (U.S., U.K., Canada, Australia), French (France, Canada), German, Japanese Siri Apple HomePod English for US, UK, Australia Clova Clova Friends, Clova WAVE Japanese and Korean for Japan and South Korea markets (ToT) Google Home
- 9. How Microsoft Cortana skill works?
- 10. How's the weather like in Tokyo today? voice How's the weather like in Tokyo today? text Weather Service When you say a command which Google Assistant knows How Google Assistant works Speech to Text, NLP, Knowledge Graph, ML Ranking, User Profile, Text to Speech Devices Google Assistant Weather Service
- 11. Connect to HelloAzureFunctio ns service voice HelloAzureFunctions I don’t know such service. I don’t know what to do When you say a command which Google Assistant does NOT know Speech to Text, NLP, Knowledge Graph, ML Ranking, User Profile, Text to Speech Devices Google Assistant Connect to HelloAzureFunction s service text How Google Assistant works
- 12. When you say a command which Google Assistant does NOT know Speech to Text, NLP, Knowledge Graph, ML Ranking, User Profile, Text to Speech Devices Google Assistant How Google Assistant works So, to make Google Assistant do your own command, you need to create an Google Assistant app called “Actions on Google” (same as Cortana/Alexa skill development motivation)
- 13. You need to implement this function (Web API) called by Webhook Speech to Text, NLP, Knowledge Graph, ML Ranking, User Profile, Text to Speech Devices Google Assistant Create Actions on Google to run your own command on your Google Assistant
- 14. Hi, I’m Madoka （voice） Hi, I’m Madoka （in cool voice） I made a voice changer app (action) working like this!
- 15. I post its demo video to Twitter two weeks ago (16 March) I made this app using Microsoft Azure Functions (C#) and it works like this … 47.5 K views in 2 weeks! It means many Google developers got interested in Microsoft technology. https://twitter.com/chomado/status/974534165419892736
- 16. Actions on Google “I’m Madoka” She says “I’m Madoka”. Invoke Webhook Microsoft Azure Dialogflow (Natural language user interface like MS LUIS)
- 17. Actions on Google “I’m Madoka” She says “I’m Madoka”. Invoke Webhook Azure Functions (where my code running) “I’m Madoka” (json data) Microsoft Azure Dialogflow (Natural language user interface like MS LUIS)
- 18. Actions on Google “I’m Madoka” She says “I’m Madoka”. Invoke Webhook Azure Functions (where my code running) “I’m Madoka” (json data) VoiceText Web API (text to speech service by HOYA, Inc) “I’m Madoka” (string) Microsoft Azure Dialogflow (Natural language user interface like MS LUIS)
- 19. Actions on Google “I’m Madoka” She says “I’m Madoka”. Invoke Webhook Azure Functions (where my code running) “I’m Madoka” (json data) VoiceText Web API (text to speech service by HOYA, Inc) “I’m Madoka” (string) Azure Storage (Blob) Microsoft Azure Dialogflow (Natural language user interface like MS LUIS)
- 20. Actions on Google “I’m Madoka” She says “I’m Madoka”. Invoke Webhook Azure Functions (where my code running) “I’m Madoka” (json data) VoiceText Web API (text to speech service by HOYA, Inc) “I’m Madoka” (string) Azure Storage (Blob) mp3 data’s URL + Request to play the mp3 file (written in SSML) Microsoft Azure Dialogflow (Natural language user interface like MS LUIS)
- 21. Actions on Google “I’m Madoka” She says “I’m Madoka”. Invoke Webhook I’m Madoka （in cool voice） Azure Functions (where my code running) “I’m Madoka” (json data) VoiceText Web API (text to speech service by HOYA, Inc) “I’m Madoka” (string) Azure Storage (Blob) mp3 data’s URL + Request to play the mp3 file (written in SSML) Microsoft Azure Dialogflow (Natural language user interface like MS LUIS)
- 22. Trigger: HttpTrigger Output: CloudBlockBlob (I chose this because it can directly access the stored mp3 file’s URL) Azure Function (v1) which is called by Google’s Webhook Google Assistant users’ input text (“I’m Madoka!”) is stored in this JSON property named “ResolvedQuery”, so we use it.
- 23. In the Function In speech execution, it’s better to use SSML format. (Speech Synthesis Markup Language: an XML-based markup language for speech synthesis applications) Not only Google Assistant but also our Cortana and Amazon Alexa support it. You can access the URL which stores the mp3 file in Azure Blob Storage As you know, Azure Functions allows us to connect & post data to Azure Storage sooooo easily! Azure Functions is awesome. Everyone knows it.
- 24. It got 58 stars and forked 10 times