Extending the google_assistant
2 likes351 views
This document provides an overview and agenda for building a natural language understanding model and rental application using the Google Assistant. It discusses building an NLU model with Dialogflow to understand user intents and entities, fulfilling requests with Actions on Google using Cloud Functions, and ways to interact with users through speech, text, cards and permissions. The goal is to build a hands-on rental shisha application that can be accessed through the Google Assistant.
Extending the google_assistant
- 2. 2 • How Am I? • Google Assistant • What is it? • How Does it work? • Build a Natural Language Understanding(NLU) Model • What is a NLU model? • Overview of DialogFlow.ai (former Api.ai) • Fulfillment with Actions-On-Google • Backend with Cloud Functions for Firebase • Development Environment with Ngrok • Ask the user • Know more about the user • Reach your users • Hands-on • Let's Build a rental Shisha app! Agenda
- 3. How Am I ? • Ndjido Ardo BAR : Lead Data Scientist @ Davidson Consulting UAE • Background : Research In Mathematics • Now: Working in @RTA as a Consultant • Past: • Working in @DU Telecom as a Consultant • Worked @AXA (Paris) • Worked @BearingPoint Hypercube • Co-Founder of a StartUp (MLouma) • Worked @Pasteur Institute: Involved in BioStatistical Research @ndjido 3
- 5. The Google Assistant The new way to talk to Google! What is it?
- 6. The Google Assistant How Does it Work? (1/2) Source: Google Actions-On-Google
- 7. The Google Assistant How Does it Work? (2/2) Source: Google
- 8. Building a Natural Language Understanding Model 4
- 9. Building a Natural Language Understanding Model What is a NLU model? TextVoice I want to rent a mint Shisha for 6pm NLU model Intent: Rent Shisha Entities: - flavor: Mint - Time: 6pm
- 10. Building a Natural Language Understanding Model DialogFlow Overview Intent Classification - Match & Categorize user utterances to an intent Entity Extraction - Identify parameters for the backend logic.
- 11. Building a Natural Language Understanding Model DialogFlow Overview: Intent Creation Intent Name User saysSystem Entities Custom Entities Slots or parameterRequired Parameter
- 12. Building a Natural Language Understanding Model DialogFlow Overview: Custom Entity Creation Entity Name Entity values
- 13. Building a Natural Language Understanding Model DialogFlow Overview: Action, Contexts and Events Output Context Input Context Lifespan Action Name Built-in & Custom Event for triggering intent
- 14. Building a Natural Language Understanding Model DialogFlow Overview: Fulfillment Webhook URL with SSL termination
- 15. Building a Natural Language Understanding Model DialogFlow Overview: Activate Fulfillment for an Intent Activated Webhook Static Text Response
- 16. Building a Natural Language Understanding Model DialogFlow Overview: NLU Response in JSON format Tests
- 18. Fulfillment with Actions-On-Google Actions-On-Google Overview Source: Google
- 19. Fulfillment with Actions-On-Google Deployment with Cloud Functions for Firebase Going Serveless with Cloud Functions Pub/Sub Firebase Cloud Storage Source: Google We use Cloud Function Local Simulators in Dev Environment
- 20. Fulfillment with Actions-On-Google Development Environment with Ngrok Source: ngrok.com Ngrok creates a secured tunnel to your localhost. Firewall + SSL Termination + tunnel to Internet
- 21. Fulfillment with Actions-On-Google Ask the user: On different Surface Capabilities
- 22. Fulfillment with Actions-On-Google Ask the user: Speech Synthesis Markup language Make the conversation sound more natural with SSML https://developers.google.com/actions/reference/ssml
- 23. Fulfillment with Actions-On-Google Ask the user: Summarize audio with text Use tell() to send a message and terminate the conversation
- 24. Fulfillment with Actions-On-Google Ask the user: Use Suggestion chips
- 25. Fulfillment with Actions-On-Google Ask the user: Guide the user with lists <=30 items
- 26. Fulfillment with Actions-On-Google Ask the user: Use Cards to show more information
- 27. Fulfillment with Actions-On-Google Ask the user: Use Carousels to show more information <=10 items
- 28. Fulfillment with Actions-On-Google Ask the user: Ask for more information with PERMISSIONS https://developers.google.com/actions/reference/rest/Shared.Types/Permission
- 29. Fulfillment with Actions-On-Google Account Linking with OAUTH Automatically Sign Up Users with Streamlined Identity Flows https://developers.google.com/actions/identity/oauth2-assertion-flow
- 30. Fulfillment with Actions-On-Google Transactions Source: Google Leverage millions of Payment Card available in Google
- 31. Reach your Users with Dialog Discovery 4
- 32. Reach your Users Explicit Triggering => Include App Name Single Point Of Entry
- 33. Reach your Users Implicit Triggering => Trigger with "user sayings" Multiple Points Of Entry "Users' sayings" in the added intents can trigger your assistant
- 34. Hands-On 4