Application Development and Data Modeling on Amazon DynamoDB

Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
Application Development and Data Modeling on
Amazon DynamoDB
David Yanacek, Sr. Software Development Engineer,
Amazon DynamoDB

Compute
Networking & CDN
Databases
Analytics
Deployment & Management
Application Services
Storage

High performance databases
throughput elasticity
availability
latency
query flexibility
storage capacity
operational complexity

Scale
Performance
Predictable, consistent
Degraded performance
with scale

= more problems
Data sharding
Data caching
Provisioning
Cluster management Fault management

Undifferentiated
heavy lifting

Amazon DynamoDB

Highlights
Flexible
Durable storage
Predictable performance
Zero admin
Low latency
Large scale
Seamless scaling
Excellent availability

Plan
• What is DynamoDB?
• In-depth application modeling
– Website Session Data (basic data model)
– Tic Tac Toe (conditional writes)
– Image Tagging (indexes and querying)
– Gaming Leaderboard (materialized views)
• High-level use cases
– Application architecture, data models

Website Session Data
basic data model and APIs
Web Sessions

Web Sessions

EC2 Instance
(web server)
Users
(first hit to website)
set-cookie: {
“SessionId”: “af872e”,
“History”: [“index.html”]
}
Web Sessions

Website Session Data: ElastiCache
Elastic Load
Balancing
EC2 Instances
(web server)
Users
ElastiCache
• ElastiCache distributed
cache
• Fast, scales
• Node failures cause
loss of session data
• Adding and Removing
nodes is tricky
Web Sessions

Website Session Data: RDBMS
Elastic Load
Balancing
Users
RDS
• Durable
• Adding and Removing
nodes is tricky
• Database blips impact
site availability
• Traffic can interfere
with other DB
workloads
Web Sessions
EC2 Instances
(web server)

Website Session Data: DynamoDB
Elastic Load
Balancing
Users
• Durable
• Transparently scalable
• Low latency
• High availability
• Read consistency
• Simple to develop
• Simple to operate
DynamoDB
Web Sessions
EC2 Instances
(web server)

DynamoDB Session Handling
Sessions Table
SessionId Last Accessed Shopping Cart
dc32f 2014-06-10T12:... {“Products”: [“Harry Potter DVD”]}
2d231 2014-06-10T12:... {“Products”: [“Amazon Fire Phone”]}
e0da3 2014-06-10T12:... {“Products”: [“Kindle Fire”]}
13f47 2014-06-10T12:...
Web Sessions

GetItem: Getting the current session
Web Sessions

PutItem: Updating the session state
Web Sessions

Using higher-level SDKs
Web Sessions

DynamoDB Session Handlers (even simpler)
• Open-source session plugins in:
– Java (Tomcat)
– Ruby (Rails)
– .NET
– PHP
– Python (Django)
– Node.js (Connect)
– …
Web Sessions

Offline development
• DynamoDB Local: Offline DynamoDB for testing,
development
Web Sessions

Recap: Website Session Data
• DynamoDB’s scalability, performance, availability,
and durability, and consistency make it a great fit
for website session data
• Very easy to use for your website – already
implemented open source
• Other key/value lookup use cases are natural fits
too
Web Sessions

Game State
conditional writes
Social Gaming

Tic Tac Toe
Social Gaming

Tic Tac Toe
{
Id : abecd,
Players : [ Alice, Bob ],
State : STARTED,
Turn : Bob,
Top-Right : O
}
Game Item
Social Gaming

Tic Tac Toe
Amazon
DynamoDB
Alice Bob
Social Gaming

Tic Tac Toe
Amazon
DynamoDB
Alice Bob
Update:
Top-Right : O
Turn : Bob
Social Gaming

Tic Tac Toe
Amazon
DynamoDB
Alice Bob
Update:
Top-Left : X
Turn : Alice
Social Gaming

Gaming the System
Alice Bob (1)
Amazon
DynamoDB
Bob (2) Bob (3)
Social Gaming

Gaming the System
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
State : STARTED,
Turn : Bob,
Top-Right : O
Social Gaming

Gaming the System
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
Update:
Turn : Alice
Top-Left : X
Update:
Turn : Alice
Mid : X
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Low-Right : X
Social Gaming

Gaming the System
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
Update:
Turn : Alice
Top-Left : X
Update:
Turn : Alice
Mid : X
State : STARTED,
Turn : Alice,
Top-Right : O,
Top-Left : X,
Mid: X,
Low-Right: X
Update:
Turn : Alice
Low-Right : X
Social Gaming

Conditional Writes
• Accept a write only if values are as expected
• Otherwise reject the write
• Performance like normal writes
Social Gaming

Tic Tac Toe (Fixed)
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
Social Gaming

Tic Tac Toe (Fixed)
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null
Social Gaming

Tic Tac Toe (Fixed)
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
State : STARTED,
Turn : Alice,
Top-Right : O,
Top-Left : X
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
Social Gaming

Recap: Game State
• UpdateItem can update specific attributes
• PutItem replaces the entire item
• Conditional writes synchronize state transitions
Social Gaming

Image Tagging
querying tables, secondary indexes
Tagging

Image Tagging: Requirements
• Query a user’s images
• Query a user’s images by date
• Tag other users in images
• Query images a user is tagged in
Tagging

Image Tagging
Images Table
Tagging

Image Tagging
Images Table
User Image Date Link
Bob aed4c 2013-10-01 s3://…
Bob cf2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
Tagging

Image Tagging
Images Table
Bob aed4c 2013-10-01 s3://…
Bob cf2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
Hash and Range primary key schema (composite key)
Range: Index is sorted by range key, can do range queries
Hash: Equality queries only
Tagging

Image Tagging
Images Table
Bob aed4c 2013-10-01 s3://…
Bob cf2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
Bob
Query for Bob’s Images
Query
FROM Images
WHERE User=Bob
Tagging

Image Tagging
• Query a user’s images
• Query a user’s images by date
• Tag other users in images
• Query images a user is tagged in
Tagging

Local Secondary Indexes
• Alternate Range Key for your table
• More flexible Query patterns
• Local to the Hash Key
– (more query options within “Bob’s” images)
Tagging

Image Tagging
Images Table
Bob aed4c 2013-10-01 s3://…
Bob cf2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
User Date Image
Bob 2013-09-05 cf2e2
Bob 2013-10-01 aed4c
Bob 2013-10-08 f93bae
Alice 2013-09-12 ca61a
Table ByDate Local Secondary Index
Local Secondary Index on Date
Tagging

Image Tagging
Images Table
Bob aed4c 2013-10-01 s3://…
Bob cf2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
User Date Image
Bob 2013-09-05 cf2e2
Bob 2013-10-01 aed4c
Bob 2013-10-08 f93bae
Alice 2013-09-12 ca61a
Table ByDate Local Secondary Index
Query for Bob’s
two most recent imagesQuery
FROM Image-Index
WHERE User=Bob, DESC, LIMIT 2
Tagging

Image Tagging
Images
Table
ImageTags
Table
Tagging

Image Tagging
ImageTags Table
Image User
aed4c Alice
aed4c Bob
f93bae Bob
Tagging

Image Tagging
ImageTags Table
Image User
aed4c Alice
aed4c Bob
f93bae Bob
Hash and Range Primary Key Schema
Tagging

Image User
aed4c Alice
aed4c Bob
f93bae Bob
Image Tagging
ImageTags Table
Bob
Query
FROM ImageTags
WHERE Image=aed4c
Tagging

Image Tagging
ImageTags Table
Image User
aed4c Alice
aed4c Bob
f93bae Alice
f93bae Bob
Bob
Tag Alice in Image f93bae
Tagging

Global Secondary Indexes
• Alternate Hash and/or Range Key for your table
• Even more flexible Query patterns
Tagging

Image Tagging
ImageTags Table
Image User
aed4c Alice
aed4c Bob
f93bae Alice
f93bae Bob
Table
Tagging

Image Tagging
ImageTags Table
Global Secondary Index on User, Image
User Image
Bob aed4c
Bob f93bae
Alice aed4c
Alice f93bae
ByUser Global Secondary Index
Image User
aed4c Alice
aed4c Bob
f93bae Alice
f93bae Bob
Table
Alternate Hash and Range Keys
Tagging

Image Tagging
ImageTags Table
Alice
Query for images tagged Alice
User Image
Bob aed4c
Bob f93bae
Alice aed4c
Alice f93bae
ByUser Global Secondary Index
Image User
aed4c Alice
aed4c Bob
f93bae Alice
f93bae Bob
Table
Query
FROM ImageTags-Index
WHERE User=Alice
Tagging

Recap: Image Tagging
• Schema design on DynamoDB is all about the
indexes
• Local Secondary Indexes support flexible queries
• Global Secondary Indexes unlock even more
flexible queries
• Many other use cases for indexing metadata on
DynamoDB
Tagging

Social Leaderboard
materialized views
Leaderboards

Friends Leaderboard
• Find friends playing the same game
• See where you stack up against your friends
Leaderboards

Friends Leaderboard
User HighScore
Chuck 20
Alice 1000
Bob 470
Dave 850
Erin 580
Friends Table HighScores Table
User Friend
Bob Alice
Bob Chuck
Alice Bob
Alice Chuck
Alice Erin
Leaderboards

Friends Leaderboard
User HighScore
Chuck 20
Alice 1000
Bob 470
Dave 850
Erin 580
User Friend
Bob Alice
Bob Chuck
Alice Bob
Alice Chuck
Alice Erin
Alice
Leaderboards

Friends Leaderboard
User HighScore
Chuck 20
Alice 1000
Bob 470
Dave 850
Erin 580
User Friend
Bob Alice
Bob Chuck
Alice Bob
Alice Chuck
Alice Erin
1. Query 2. BatchGetItemAlice
Leaderboards

Inefficiencies in friends leaderboard
• Always retrieve all friends
– Some friends don’t play
– Might care about top 10 scores only
• Single-item operations for each friend
– Retrieving 10 items individually is more expensive
than 10 items in a row
Leaderboards

Improved Friends Leaderboard: Reads
User Friend H.S.
Bob Alice 1000
Bob Chuck 20
Alice Bob 470
Alice Chuck 20
Alice Erin 580
User HighScore
Chuck 20
Alice 1000
Bob 470
Dave 850
Erin 580
FriendScores Table HighScores Table
Leaderboards

User Friend H.S.
Bob Alice 1000
Bob Chuck 20
Alice Bob 470
Alice Chuck 20
Alice Erin 580
User HighScore
Chuck 20
Alice 1000
Bob 470
Dave 850
Erin 580
User H.S. Friend
Bob 20 Chuck
Bob 1000 Alice
Alice 20 Chuck
Alice 470 Bob
Alice 580 Erin
Table Local Secondary Index
Leaderboards

User Friend H.S.
Bob Alice 1000
Bob Chuck 20
Alice Bob 470
Alice Chuck 20
Alice Erin 580
User HighScore
Chuck 20
Alice 1000
Bob 470
Dave 850
Erin 580
User H.S. Friend
Bob 20 Chuck
Bob 1000 Alice
Alice 20 Chuck
Alice 470 Bob
Alice 580 Erin
Table Local Secondary Index
Query
FROM FriendScores-Index
WHERE User=Alice, DESC, LIMIT 10Alice
Leaderboards

User HighScore
Chuck 30
Alice 1000
Bob 470
Dave 850
Erin 580
Improved Friends Leaderboard: Writes
User Friend H.S.
Bob Alice 1000
Bob Chuck 30
Alice Bob 470
Alice Chuck 30
Alice Erin 580
Chuck
1. UpdateItem
2. UpdateItem(s)
Leaderboards

Recap: Friends Leaderboard
• BatchGetItem helps retrieve many items
• Query is cheaper than retrieving items
individually
• Trade off: reads can be made faster, cheaper, by
doing more writes
Leaderboards

High-level use cases
application architecture
Architecture

High-level use cases
• Online game coordination
• Real-time ad impression bids
• Indexed event logging
• Social messaging
• Real-time voting
• Real-time analytics
Architecture

Online Game Coordination
Availability Zone #1
DynamoDB
region
EC2
EC2
EC2
CloudFron
t
Amazon S3
(content)
(client app)
(game servers)
(lobby, stats service)
Architecture

DynamoDB
region
EC2
EC2
EC2
1) Login
2) Request matchmaking
3) Get game server IP
CloudFron
t
Amazon S3
Architecture

DynamoDB
region
EC2
EC2
EC2
1) Login
4) Connect to server
5) Pull down assets
6) Other players join
CloudFron
t
Amazon S3
Architecture

DynamoDB
region
EC2
EC2
EC2
1) Login
4) Connect to server
5) Pull down assets
6) Other players join
7) Game ends
8) Update stats
CloudFron
t
Amazon S3
Architecture

Indexed event logging
User DateTime Product
David 2014-06-22T21:01 Fire TV
David 2014-06-22T21:02 Fire Phone
David 2014-06-23T08:32 Fire Game Controller
Alice 2014-06-23… …
… … …
Index webpage clicks by user
Architecture

Real-time ad impression bids
Architecture

Social networks, messaging
Thread MessageId User Message
Fri Night 21:01.1 David Hi!
Fri Night 21:01.2 Steve What’s up?
Fri Night 21:02.1 David Happy Hour?
Participants Table,
Secondary Index
Messages Table
Thread User
Fri Night David
Fri Night Steve
DynamoDB? David
User Thread
David DynamoDB?
David Fri Night
Steve Fri Night
Architecture

Real-time voting
Voters
AggregateVotes
Table
RawVotes
Table
Voting App
Amazon
Redshift
Amazon EMR
Architecture

Real-time analytics
Kinesis-enabled
app
Amazon Kinesis
DynamoDB
Ingestion
In-memory
first-pass
aggregation
Durable
aggregation
Architecture

Real-time analytics
Page Time Views
index.html 2014-06-25T12:00 120
index.html 2014-06-25T12:01 122
index.html 2014-06-25T12:02 125
search.html 2014-06-25T12:00 200
PageViews
Table
Kinesis-enabled
app
UPDATE PageViews
ADD 5 to Views
WHERE Page=index.html
AND Time=2014-06-25T12:00
Architecture

Real-time analytics
Page Time Views
index.html 2014-06-25T12:00 120
index.html 2014-06-25T12:01 122
index.html 2014-06-25T12:02 125
search.html 2014-06-25T12:00 200
PageViews
Table
Kinesis-enabled
app
Query PageViews
WHERE Page=index.html
AND Time >= 2014-06-25T12:00
LIMIT 10
Architecture

Recap
• What is DynamoDB?
• In-depth application modeling
– Website Session Data (basic data model)
– Tic Tac Toe (conditional writes)
– Image Tagging (indexes and querying)
– Gaming Leaderboard (materialized views)
• High-level use cases
– Application architecture, data models

Thanks!
aws.amazon.com/dynamodb/developer-resources

Application Development and Data Modeling on Amazon DynamoDB

More Related Content

Viewers also liked (7)

Similar to Application Development and Data Modeling on Amazon DynamoDB (20)

More from Amazon Web Services Japan (20)

Recently uploaded (20)

Application Development and Data Modeling on Amazon DynamoDB