Conversational user interfaces (by Jochem Grietens)
2 likes350 views
This document discusses conversational user interfaces and voice interfaces. It provides an overview of conversational interfaces, how they work using techniques like natural language processing, and how platforms like Amazon Alexa can be used to build voice-enabled applications. Examples of using voice for accessibility, data collection, and augmenting humans are also presented.
Conversational user interfaces (by Jochem Grietens)
- 1. Conversational user interfaces. 1 CONFIDENTIAL Template Innovation Day 2017CONFIDENTIAL Conversational user interfaces Ir. Jochem Grietens Systems engineer & AI enthusiast. jochem.Grietens@verhaert.com TRACK 1 - TRANSFORMATIONAL TECHNOLOGIES
- 2. Conversational user interfaces. 2 CONFIDENTIAL Yours truly Ir. Jochem Grietens System engineer @ Verhaert Product development engineer. NOT a software engineer Big enthusiast of software as a tool for innovation. Education: Aerospace engineering @ Kuleuven
- 3. Conversational user interfaces. 3 CONFIDENTIAL What are we talking about You must be new to these galaxy class starships
- 4. Conversational user interfaces. 4 CONFIDENTIAL Conversational user interfaces defined. Conversational user interfaces • Rule-based • AI (Artificial Intelligence) based We will be focusing on spoken language, voice control rather than chatbots or text based VUI.
- 5. Conversational user interfaces. 5 CONFIDENTIAL The future of voice interfaces
- 6. Conversational user interfaces. 6 CONFIDENTIAL Cars are more than carriages without a horse. Smartphones have become more than telephones without a cord. Voice interfaces will be more than computer without keyboards & displays. Voice interface, a transformational technology ?
- 7. Conversational user interfaces. 7 CONFIDENTIAL Rethink Your Product and Business ? Strategy Business Technology User centricity TRANSFORMATION FRAMEWORK Design thinking Digital & data sciences Business model innovation / servitization 3D printing Advanced materials IoT & sensors Robotics & autonomous platforms Early user validation (User) co-creation Behavioral change Roadmap Accelerated by open innovation and technology transfer
- 8. Conversational user interfaces. 8 CONFIDENTIAL "I think there is a world market for maybe five computers." -- Thomas Watson, chairman of IBM, 1943. "Nuclear-powered vacuum cleaners will probably be a reality within ten years.“ -- Alex Lewyt, president of Lewyt vacuum company, 1955 About predicting the future…
- 9. Conversational user interfaces. 9 CONFIDENTIAL Why Voice ?
- 10. Conversational user interfaces. 10 CONFIDENTIAL Natural mode of communication Speech is the most natural mode of communication. = Low effort. “For all the times you are yelling at your devices... Just image them actually listening... “
- 11. Conversational user interfaces. 11 CONFIDENTIAL Natural mode of communication Speech is the most natural mode of communication. = Low effort. “For all the times you are yelling at your devices... Just image them actually listening... “
- 12. Conversational user interfaces. 12 CONFIDENTIAL Processors obey Moore's law, can interfaces follow ? Bandwidth of communication Quad Core mobile: 6,4 Giga operations/s Extreme multicore computer @ 100 Hz Enourmous ammount operations/cycle. = 1016 operation/s 3-4 symbols/sec2-3 symbols/sec 20 symbols/sec UNINTERUPTED
- 13. Conversational user interfaces. 13 CONFIDENTIAL Accessibility Interface with devices where previously impossible. • When hands not free. • Control machines in manufacturing • Hand labor • More complex applications in cars. • No room for keyboard/screen or other interface. • Clean rooms
- 14. Conversational user interfaces. 14 CONFIDENTIAL Factory Talks - analytics for devices Rockwell automation. • Shelby = data scientist in a box • A production line Chabot -> gives production line managers status updates and diagnostics platform. • Already had a text based conversational interface or chatbot. • A voice interface was created based on Microsofts Cortana.
- 15. Conversational user interfaces. 15 CONFIDENTIAL Data capture • Gather information for advertisement • Capture content from customers directly. Speech to text NLP extracts: dishes, amounts, … {Course : fried chicken} {sides : fries , salad} {Course : Ceaser salad} {exclusions : tomatoes } {sides : bread } Local network Queuing system & preprocessing Speech alert, dishes ready Next meal to cook Confirmation
- 16. Conversational user interfaces. 16 CONFIDENTIAL Lowering the threshold Lower the threshold to data and computing
- 17. Conversational user interfaces. 17 CONFIDENTIAL Human replacement Call centers/ telephone services & support. More interactive Information Reception systems
- 18. Conversational user interfaces. 18 CONFIDENTIAL Augmenting humans Invisible interface • Smart home, iot • Linking many devices with simple commands and one user interface. • UI lighter and out-of-the-way. Extend the power of computing to people unable, for one reason or another, to use screens and keyboards. Your personal translator VISUALS NODIG Google pixel earbuds
- 19. Conversational user interfaces. 19 CONFIDENTIAL Are people ready for this interface. 85 % of consumers say: “Voice interface experiences are great when they work but irritating when they don’t work properly .” -Creative Strategies. This raises the question? How do we create a good user interface.
- 20. Conversational user interfaces. 20 CONFIDENTIAL Voice centered design
- 21. Conversational user interfaces. 21 CONFIDENTIAL How to engage users 1. The skill makes a task faster and easier with voice. 2. The skill has an intuitive and memorable name. 3. The skill sets clear expectations on what it can do. 4. The skill minimizes friction. 5. The skill surprises and delights customers. 6. The skill delivers fresh content each time. 7. The skill is consistently reliable. 8. Sufficient guiding information is available to know the options. 9. The provided information is adapted to the specific user and changing in time.
- 22. Conversational user interfaces. 22 CONFIDENTIAL So what’s under the hood ?
- 23. Conversational user interfaces. 23 CONFIDENTIAL Speech capture •Hardware •Microphone •Basic signal processing and conditioning. Speech Recognition •Speech to text. Natural language processing •Recognize intents. •Recognize similar sentences •Extracting words and classify. Voice interaction Model •Model that links user utterances to actions. ( Actions in the software). Software program •Code that is linked to voice interaction model. •Is called by voice interaction model and executed Actions •Speech to text. •Electronics being controlled.(I.e. Lights turned on) • Audio player started. •External software call ( i.e. Google calander entry) Components
- 24. Conversational user interfaces. 24 CONFIDENTIAL Components of speech recognition Utterance : “Set light brightness to 9” 1. Audio collection 1. Pre-filtering of audio and relaying to hardware or cloud for processing. 2. Calling the attention of the assistant. = wake word like “alexa”,”sirri”,”ok google” 2. Speech to text. 1. Cutting the incoming waveform into pieces & encoding the waveform. 2. Acoustic model : Speech Waveform -> phones ( basic sound units ) = acoustic model. (HMM) 3. Probability of a sequence of words Language model. 3. Recognizing the sentence 1. Recognize the text as a certain INTENT of the user. 2. We want this recognition to be flexible. -> Some sort of voice interaction model. “Please, can the brightness be turned up to 9” 4. Start some action based on this input 5. Formulate a response.
- 25. Conversational user interfaces. 25 CONFIDENTIAL Natural Language Processing (NLP)
- 26. Conversational user interfaces. 26 CONFIDENTIAL Intent and entities
- 27. Conversational user interfaces. 27 CONFIDENTIAL Speech recognition Speech Waveform generation Grapheme-to- phenome conversion Text She has twenty dollars In her pocket
- 28. Conversational user interfaces. 28 CONFIDENTIAL Text to speech Pattern matching with Acoustic modelling & Language modelling. Acoustic modelling Acoustic Modelling is the process of taking a waveform of speech and analysing it using statistical models. Hidden Markov Modeling, is used for breaking speech into component parts called phones.
- 29. Conversational user interfaces. 29 CONFIDENTIAL Deep Learning In Speech Recognition
- 30. Conversational user interfaces. 30 CONFIDENTIAL Deep Learning In Speech Recognition Conclusions on how we got here: • Theory and algorithm design in the 80s and 90s • Orders of magnitude more data available • Orders of magnitude more computational capacity • A few algorithmic inventions enabled deep networks • The rise of distributed SGD training
- 31. Conversational user interfaces. 31 CONFIDENTIAL Conclusion • Spoken conversational UI requires sophisticated infrastructure. Service applications, Complex ML models running on servers, data handling infrastructure… • A lot of data and expertise needed to train these complex models to perform the needed task. • Given the complexity of these systems and the difficulty of building this ML infrastructure, businesses would not be able to implement this without using existing tech. This all takes us away from our job to build engaging and powerful products and services. • Existing software puts the power of AI in the hands of programmers (even engineers)
- 32. Conversational user interfaces. 32 CONFIDENTIAL Alexa , Google home, … Device & software Platform Alexa Skills portal Have you started development on the skill for you product company ? Using existing frameworks 25,000 skills
- 33. Conversational user interfaces. 33 CONFIDENTIAL Let’s get specific “Any sufficiently advanced technology is indistinguishable from magic.” - Arthur C. Clarke
- 34. Conversational user interfaces. 34 CONFIDENTIAL AMAZON ALEXA framework
- 35. Conversational user interfaces. 35 CONFIDENTIAL 1. Define a Voice User Interface. 2. Provide a finite number of sample utterances 3. ASK automatically builds and deploys machine learning models. Alexa Skills Kit
- 36. Conversational user interfaces. 36 CONFIDENTIAL ASK 1. Define a Voice User Interface 2. Provide a finite number of sample utterances 3. ASK automatically builds and deploys machine learning models. 1 2 3
- 37. Conversational user interfaces. 37 CONFIDENTIAL Alexa Skills Kit for developers (ASK) Cloud endpoint: Amazon lambda Request: IntentRequest requestId: EdwRequestId.a6c89145-279e- 4e46-86bd-7833d1b90ddd intent: ChangeNumberIntent slot: Number value= 5
- 38. Conversational user interfaces. 38 CONFIDENTIAL Let’s start building
- 39. Conversational user interfaces. 39 CONFIDENTIAL Goal: setting the level of a scoreboard simply by saying something like: “Set the display to five.” With Alexa this process is normally started by uttering something like : Components of that sentence: Wake word : Activates the alexa to listen. Launch command : Launches the application that we aim on using. Invocation name : The name I gave the application Utterance : The command you want give to the application. Input Sentence Alexa, ask Number Control to set display to five.
- 40. Conversational user interfaces. 40 CONFIDENTIAL Dataflow overview Skill: Number Control Utterance: “display number five”
- 41. Conversational user interfaces. 41 CONFIDENTIAL Audio front end Alexa voice service • Enables developers to integrate Alexa directly into their products • Bringing the convenience of voice control to any connected device. Amazon Alexa 7-Mic Far- Field Dev Kit XMOS VocalFusion 4-Mic Dev Kit for Amazon AVS Conexant 4-Mic Development Kit for Amazon AVS Audio front end • Can be replaced with custom hardware. • Custom hardware must run ALEXA VOICE SERVICE. • Preprocessing and filtering of audio must be managed by developer. Number of Mics, Mic Arrangement, Audio Algorithms
- 42. Conversational user interfaces. 42 CONFIDENTIAL Data Flow - ASK • Speech to text • Voice interaction model builds voice recognition on sample utterances and links to intents. • Utterance detection • Slot extraction. • Event triggering based on utterances.
- 43. Conversational user interfaces. 43 CONFIDENTIAL Data Flow - ASK
- 44. Conversational user interfaces. 44 CONFIDENTIAL Data Flow – Cloud Backend • Intent handler • Handles incoming events from the Alexa platform • Receives intents and extracted values
- 45. Conversational user interfaces. 45 CONFIDENTIAL Data Flow – Cloud Backend Request: IntentRequest requestId: EdwRequestId.a6c89145-279e-4e46-86bd-7833d1b90ddd intent: ChangeNumberIntent slot: Number value= 5 Response: OutputSpeech: ssml: <speak> ok </speak>, shouldEndSession : true
- 46. Conversational user interfaces. 46 CONFIDENTIAL Data Flow – IOT platform • AWS IoT is a managed cloud platform • Manages devices and data interaction. • Easy and secure interaction with cloud applications and other devices. • Scaleableith AWS IoT
- 47. Conversational user interfaces. 47 CONFIDENTIAL Data Flow – Endpoints • Any aws IoT enabled device. • Raspi is a good choice for prototyping. • Enalbles I/O control.• External API’s can be triggered. • Links to other applications. • ….
- 48. Conversational user interfaces. 48 CONFIDENTIAL Data Flow – Improvements ? • Embed AVS in custom hardware • Enable Local Actions
- 49. Conversational user interfaces. 49 CONFIDENTIAL Innovation Day is an initiative of Masters in Innovation, the umbrella brand of the Verhaert Group which aims to connect, train and accelerate professional innovators. Kruibeke Belgium Hogenakkerhoekstraat 21 B-9150 Kruibeke T +32 3 250 19 00 E info@verhaert.com www.verhaert.com Nivelles Belgium Noordwijk Netherlands Av. Robert Schuman 102 B-1400 Nivelles T +32 67 47 57 10 E info@lambda-x.com www.lambda-x.com Kapteynstraat 1 2201 BB Noordwijk T +31 71 760 05 50 E info@verhaert.com connect.verhaert.com INDUSTRY TECHXFER MEDICAL AEROSPACE TECHXFER FMCGCONNECT TECHXFER FMCGCONNECT MEDICAL Aveiro Portugal Av. Dr. Lourenço Peixinho 96D 4o 3800-159 Aveiro T +351 234 604 088 E info@load-interactive.com www.load-interactive.com CONNECT Gentbrugge Belgium Bruiloftstraat 55-57 B-9050 Gentbrugge T +32 9 330 27 90 E info@moebiusdesign.be www.moebiusdesign.com ON SITE CONSULTANCY