Serverless and Chatbots: A Match Made in the Cloud

Chatbots and
Serverless: A match
made in the Cloud
Gillian Armstrong
@virtualgill
QConSF

InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
serverless-chatbots

Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

Gillian Armstrong
@virtualgill
Technologist
Cognitive Technologies
Liberty IT
Gillian Armstrong @virtualgill

2016
2020
7%
SOURCE: US Census, Gartner,
Edison Research, Voicebot.ai
75%
Smart Speaker US Household
Ownership Forecast

2016
2020
7%
SOURCE: US Census, Gartner,
Edison Research, Voicebot.ai
75%
Smart Speaker US Household
Ownership Forecast
The world is
changing
fast

So we need to
be able to
learn fastThe world is
changing
fast

Change
Fast
Learn
Fast
use what we learn to
so we can keep
on learning

Change
Fast
Learn
Fast
Both Chatbots and Serverless let us
and

What is a
Chatbot???
What do you
mean byServerless???
???
???
???

A Serverless solution is one that costs
you nothing to run if nobody is using it
(excluding data storage cost)
For the purposes of this presentation...
- Paul Johnston

For the purposes of this presentation...
A Chatbot is something you can
interact with conversationally
using natural language
- me

CHANNEL/DEVICE
NLU NLP
INTENTS ENTITIES
I n t e r a c t i o n M o d e l
Fulfillment
MachineLearning
ASR TTS
CUX/VUX
A very quick
overview of a
Chatbot using
Conversational AI

CHANNEL/DEVICE
NLU NLP
INTENTS
Fulfillment
MachineLearning
ASR TTS
CUX/VUX
Utterance
(“what the
user said”)
Automatic
Speech
Recognition
(Speech to Text)
Text to Speech
Mapping of Utterance to Intent
(“what the user meant”)
Extraction of Entities
(variables like time, place, number, etc)
Oh hey, there’s still UI design
Insert cool
serverless
architecture here
Conversational / Voice User Experience
(“how we sound and the words we say”)
Conversational User Interface
ENTITIES
Intent maps to Fulfillment
(“what we need to do”)

CHANNEL/DEVICE
NLU NLP
INTENTS
Fulfillment
MachineLearning
ASR TTS
CUX/VUX
Design it so
a Human can
understand it
Train your Bot
to understand
Humans
Write code to make
it do something!
1
2
3
What we need to do…
ENTITIES

CHANNEL/DEVICE
NLU NLP
INTENTS
Fulfillment
MachineLearning
ASR TTS
CUX/VUX
Design it so
a Human can
understand it
Train your Bot
to understand
Humans
Write code to make
it do something!
1
2
3
What we need to do…
Often starting
point and
focus
Hardcoded
and then
panic-written
after that
well-received
demo
Forgotten
ENTITIES

Design it so a
Human can
understand it
1

In a chatbot,
Conversation is
the Interface

Source: http://hosanna1.com/
Not a CSS
and HTML
problem!
Design
Matters!

Up to now we’ve
kept designing
based on what
came before

We’ve carried the
concept of menus
and commands
into our graphical
user interfaces

We base our web
forms on the
paper forms that
came before

Model your design on a
human, not a website
People are the original
conversational interface

When you start with
thinking about trying out
a design you may
discover that a chatbot
isn’t even the right thing
to build…

But assuming it is…

Be Clear
Be Helpful Be Nice
• Proactively offer help
• Get people back on track
• Repetition is annoying
• Repetition is annoying
• Take the blame – in a conversation the
human is never wrong
• Chatbots aren't smarter than a human
• Be understanding if people don’t want
to talk to you – offer alternatives.
1 minute
Conversational
Design
overview
• Keep your personality consistent
• Keep answers short and to the point
• Set expectations – what can you do
• Reflect back key information and
confirm before critical actions.

The Web is about Telling
the user what to do
Conversational UI is
about Listening to what
the user wants to do

Train your Bot
to understand
Humans
2

Amazon Lex Console
What the Web Tutorials told me
building a chatbot looked like…

BOOK MEETING
IntentUtterance
SAY HELLO
GET HELP
Fulfillment

IntentUtterance
As simple
providing a set
of sample
utterances!
Book Meeting at <TIME> on <DATE>
Set up Appointment
Create event on <DATE>
Set up Meeting
Book me a meeting at <TIME>
…
Hello
Hi there
Hey
Good Morning
…
Help
What can you do
Help me
I need help
…
BOOKMEETINGSAYHELLOGETHELP

Amazon Lex Console
SET OF
INTENTS
Represents the
functionality
the bot has
SAMPLE
UTTERANCES
Examples of what a
user might say to
trigger this Intent.
This is training data -
not a regex match.

Amazon Lex Console
SLOTS (ENTITIES)
are defined showing
type (place, name,
time, custom, etc.)
If the user does not supply this data
the bot knows to prompt for it.
The sample
utterances show
where to expect the
data to be supplied in
an utterance

FULFILLMENT location
defined.
Intent and Slot matches will
be sent to this lambda or
returned directly to client.
Amazon Lex Console

Amazon Lex Console
Publish to make
the functionality
live to users.
Build to create the
Interaction Model

Amazon Lex Console
Built-in Test
Console allows you
to test directly here
after building, but
before publishing.

Amazon Lex Console
Easy to connect
your bot to popular
channels through
console.

Amazon Lex Console
Chatbot Done…?

Sorry, I’m just a
baby bot. I’m still
learning.

Sorry, I’m just a
learning.
are you?

IntentUtterance
As complex
as providing a
set of sample
utterances!
Book Meeting at <TIME> on <DATE>
Set up Appointment
Create event on <DATE>
Set up Meeting
Book me a meeting at <TIME>
…
Hello
Hi there
Hey
Good Morning
…
Help
What can you do
Help me
I need help
…
BOOKMEETINGSAYHELLOGETHELP
Model your design on a
human
Get your sample utterances
from your users…
you wouldn’t make up other
machine learning training data!

Track metrics and review
conversation to improve your
Bot Interaction Model.
Learn
Fast
Sorry, I’m just a
learning.
Change
Fast

Look for Vocabulary used
Should we add new synonyms
Look for the Utterance Structure
Should we add new training data
Look for Interaction Patterns
Can we improve the conversation flow
Learn
Fast
Change
Fast
Reviewing Conversation Transcripts…

Chatbots aren’t magic
(although I wish they were)

Write code to
make it do
something!
3

Pop a lambda
here – and
you’re done!

CONTEXT
CONVERSATION
SESSION
CHAT
STREAM
ANALYTICS
NLP +
INTENT
MAPPING
FULFILLMENT…
Amazon
Lex
Amazon
VPC
AWS
Direct
Connect
AWS
Lambda
AWS
Lambda
Amazon
DynamoDB
Amazon
DynamoDB
AWS
Lambda
Amazon
CloudWatch
Amazon
ES
IAM
Amazon
Polly
Amazon
S3
AWS
X-Ray
Amazon API
Gateway
Amazon
Athena Amazon
S3
Amazon
API
Gateway
THE DIGITAL ASSISTANT BOT PLATFORM ARCHITECTURE

Number of
conversations
Length of
conversations
Functionality
being used
Number of Users
What people
want us to add
Where we are
getting it right
Where we are
getting it wrong
Examples of our
Conversational Analytics

Observability is critical
This lets you quickly learn
where there might be problems
with your system.

Why Serverless??

Lower Cost
lets us
Change
Fast
Learn
Fast
Off-loading Maintenance
lets us

Instead of custom
building and managing
your own servers, db, etc…

Use standard pieces –
that someone else has
built and manages - and
put them together to get
the system you want

Serverless Services

Lambdas are quick to develop,
isolate small pieces of
functionality and allow you to
scale different parts of the
conversation independently.

Ensure that
everything scales
- or at least know
your limits.
API
Lambda
functions
Amazon
DynamoDB

Serverless means you don’t
pay if no-one talks to your
bot…
…and lets you scale
seamlessly if everyone
wants to talk to your bot!

Cost
Usage
No usage,
no cost
Higher usage,
higher cost

Cost
Usage
Watch the
free tier…
Cost may not
be linear

Monolith
Some notes for
those with
existing systems

Monolith Microservices
async!
event
driven!

Monolith Microservices Function as a Service

REWRITE is not a
four-letter word

A good human conversation is
synchronous
You wait for a response
You expect a response
immediately

A Conversational
Architecture needs to
- Be fast!
- Appear synchronous to the
user (there should be a
response)
This is particularly critical for a voice-based bot

Do not just lift and shift your
microservices into FaaS

invoke invoke invoke
Combining the time to invoke and the time to ‘spin up’
can hit performance and add complexity to your system.
Choose to ‘keep warm’ only as a last resort.
Do not end up with a collection of
distributed microservices

invoke
Shhhhhh….. It’s ok to have more than one
function in your Function as a Service…
Look for where you can pull functionality
into reusable libraries instead

Having said that… don’t just lift and shift
your microservices into FaaS

API API API
APIs designed for webforms tend to be
set up to collect sets of data page by page

API
A Conversation is non-linear – the user can give
partial information, or information that would have
been collected later on your webform.

Both Serverless
and Chatbots
require new ways
of thinking about
your architecture.

CHANGE
FAST
LEARN
FAST
We get there iteratively by being able to
and

CHANGE
FAST
LEARN
FAST
and

CHANGE
FAST
LEARN
FAST
and
but Good Supporting Engineering Practices are
needed to do both with confidence and at scale

CODE HERE
Fulfillment
in Code

CODE HERE
CODE HERE
Bot
Interaction
Model
in Code

CODE HERE
CODE HERE
CODE HERE
Design
in Code

CODE HERE
CODE HERE
CODE HERE
CODE HERE
CODE HERE
CODE HERE
CODE HERE
Infrastructure
in Code

CODE HERE
EVERYTHING
in Code
CODE HERE
CODE HERE
CODE HERE
CODE HERE
CODE HERE
CODE HERE

EVERYTHING
in Code
• Testable
• Traceable
• Repeatable
• Observable

Conversation as Code - Amazon Lex
Fully Validated and Repeatable Bot Build and Deploy to any AWS Account
Everything written in CFN,
with JSON for API calls to
Amazon Lex Model Building
API as part of deploy

Local and Pre-Commit
Hook Tests
• Unit Test
• Lambdas
• Libraries
• Static analysis
• Cloud Formation
• Code
• JSON for API calls

Code push triggers tests, and automatic build
and deploy across multiple environments

Build Tests
• Unit Test
• Lambdas
• Libraries
• Static analysis
• Cloud Formation
• Code
• JSON for API calls

All tests, builds, deploys and
alarms sent to Slack channels to
alert developers immediately

AWS
CloudFormation
Templates
Amazon S3
deployment
bucket
Amazon
Lex
Model
Building
Service
Botbuilder
Lambda
function
Packaged
Fulfillment
Lambdas
Bot JSON files

Deploy Tests
• Smoke Tests
• Component Tests
(Lex) Non-Prod Only

Fully tested new
functionality
available in
minutes

Live
Analytics
Fully tested new
functionality
available in
minutes

Live
Analytics
‘Live’ Tests - APM
• Alarms
• Cloudwatch metrics
and Dashboards
• Centralized Logging
• Tracing

If your chatbot stopped working, would you
know? Don’t just wait for user complaints!
Know what isn’t working and fix it, and keep
adding more of what is!
Monitor, no, seriously, monitor

Conversational Analytics are vital
Work out what metrics you need to track to
know how your chatbot is performing.
Know what isn’t working and fix it, and keep
adding more of what is!

Getting Feedback is vital
You won’t get it right first time – listen to
your users.
Know what isn’t working and fix it, and
keep adding more of what is!

Change
Fast
Learn
Fast
lets us
Being able to

Conversational UI is about
Listening to what the user
wants to do…
CHANGE
FAST
LEARN
FAST
…and that’s the secret to a great chatbot!

Got questions?
Let’s chat!
@virtualgill
KEY
TAKEAW
AYS • Model your Chatbot UI and Architecture
based on observing Humans and real
Conversation.
Break out of old mindsets!
• New tech still needs the good
engineering practices you already know

Appendix – Additional Information on Conversational Design
Starting out:
https://hackernoon.com/new-to-conversational-design-start-here-7f2f3a1b81bb
Error Handling in Chatbot Series:
Part 1 — Voice Recognition Errors: The one where we end up shouting at a computer
https://chatbotsmagazine.com/difficult-conversations-1-voice-recognition-error-handling-74d93056ddce
Part 2 – Conversational Errors: The one where it gets really interesting
https://chatbotsmagazine.com/helping-your-baby-bot-learn-to-chat-like-a-grown-up-bot-99f5170f1c55
Part 3 – Technical Faults: The one we want to pretend will never happen (or ‘Help! My chatbot has fallen over
and can’t get up’)
https://chatbotsmagazine.com/good-ux-when-your-chatbot-is-having-a-very-bad-day-e4f22885d7fb
You can find my other Blogs at https://medium.com/@virtualgill
Come talk to me on Twitter – I hang out there a lot @virtualgill
For my other talks and ways to get in contact with me head to http://virtualgill.io

Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
serverless-chatbots

Serverless and Chatbots: A Match Made in the Cloud

More Related Content

Similar to Serverless and Chatbots: A Match Made in the Cloud (20)

More from C4Media (20)

Recently uploaded (20)

Serverless and Chatbots: A Match Made in the Cloud