Transforming Mobile Push Notifications with Big Data

Transforming Mobile
Push Notifications with
Big Data
Dennis Waldron, Data Engineering
Pablo Varela, Systems Engineering

Who is Plumbee?
● 12.8M Installs
● 209K Daily Active Users
● 818K Monthly Active Users
● Social Games Studio
● Mirrorball Slots & Bingo
● Facebook Canvas, iOS

Data Providers
Inhouse data = 99.9% of all data
In Total:
● 98TB (907 days of data)
● All stored in Amazon S3
Daily:
● 78GB compressed
● ~450M events/day
● 4,800 events/second (peak)

Architecture - Overview
Events (JSON)
Daily Batch Processing
Aggregates
Application/Game Servers
End Users (Desktop & Mobile)
Log Aggregators
Amazon S3
Amazon EMR
(Elastic MapReduce)
DataPipeline (Simple Storage Service)
Amazon Redshift
Plumbee Employees
Analytics (SQL Queries)
SQS Analytics Queue
Events (JSON)

Amazon Web Service
● Collect everything!
● RPC events intercepted by
annotated endpoints. (Requests)
● All mutating state changes
recorded:
○ DynamoDB, MySQL, Memcache
(Blobs Updates)
● Custom Telemetry (Other):
○ Client: click tracking, loading time
statistics, GPU data...
○ Server: promotions, transactions,
Facebook user data...
Game Data
MySQL
MemCache
RPC
77%
9%
OTHER 15%
GENERATES
DynamoDB

Game Data - Example RPC Endpoint Annotation
/**
* Example annotation
*/
@SQSRequestLog(requestMessage = SpinRequest.class)
@RequestMapping(“/spin”)
public SpinResponse spin(SpinRequest spinRequest) {
…
}

Example Event - userStats
● All events are recorded in JSON.
● Structure:
○ Headers
○ Categorization Data (metadata)
○ Payload (message)
● Important Headers:
○ timestamp
○ testVariant
○ plumbeeUid

Architecture - Collection
Aggregates
Amazon S3
Amazon EMR
(Elastic MapReduce)
Amazon Redshift
Plumbee Employees
Log Aggregators
Events (JSON)
SQS Analytics Queue
Events (JSON)

Data Collection (I) - PUT
Events (JSON)
SQS Queue
Log Aggregators
Producers Consumers
What is SQS (Simple Queue Service)?
A cloud-based message queue for transmitting
messages between producers and consumers
SQS Provides:
● ACK/FAIL semantics
● Unlimited number of messages
● Scales transparently
● Buffer zone

Data Collection (II) - GET
SQS Queue
What is Apache Flume?
A distributed, reliable, and available service
for efficiently collecting, aggregating, and
moving large amounts of log data
Apache Flume
Consumers
Amazon S3
(Simple Storage Service)
S3 Data:
● Partitioned by: date / type / sub_type
● Compressed with: Snappy
● Aggregated in 512MB chunks

Data Collection (III) - Flume
Flume Agent
Source
(Custom)
Sink
(HDFS)
SQS Queue
Channel
(File Based)
● Pluggable component architecture
● Durability via transactions
● File channel use Elastic Book Store (EBS) volumes (network attached storage)
○ Protects against Hardware failure
● SQS Flume Plugin: https://github.com/plumbee/flume-sqs-source
S3 Bucket
Transactions
A + B + C = Flow
A B C

Architecture - Processing
Events (JSON)
Aggregates
Amazon S3
Amazon EMR
(Elastic MapReduce)
Amazon Redshift
Plumbee Employees
SQS Analytics Queue
Events (JSON)

Extract, Transform, Load
● Daily activity
● Orchestrated by Amazon DataPipeline
● Includes generation of reports
● Configured with JSON
What is DataPipeline?
A cloud-based data workflow service that
helps you process and move data between
different AWS services
RESOURCE COMMAND SCHEDULE

Extract & Transform (I)
What is Elastic Map Reduce?
Cloud-based MapReduce implementation to
process vast amounts of data built on top of
the open-sourced Hadoop framework.
Two phases:
● Map() Procedure -> Filtering & Sorting
● Reduce() -> Summary operation
Penguin
Horse
Cake
Cake
Penguin
Penguin
Penguin
Horse
Horse
Cake
Cake
Horse
Horse
Horse
MAP()
Penguin
Penguin
Penguin
Penguin
REDUCE()
Cake: 2 Horse: 3
RESULT SORTED QUEUES RAW DATA
Penguin:
4

Extract & Transform (II)
What is Hive?
An open-sourced Apache project with provides a
SQL-Like interface to summarize, query and
analysis large datasets by leveraging Hadoop’s
MapReduce infrastructure.
● Not really SQL, HQL -> HiveQL
● No transactions, materialized views,
limited subquery support, ...
SELECT plumbeeuid,
COUNT(*) AS spins
FROM eventlog
-- Partitioned data access
WHERE event_date = '2014-11-18'
AND event_type = 'rpc'
AND event_sub_type = 'rpc-spin'
-- Aggregation
GROUP BY plumbeeuid;
Table: Eventlog
● Mounted on top of raw data
● SerDe provides JSON parsing
● Target data via partition filters

Extract & Transform (III)
● Hive has limitations!
○ Speed, JSON
● Most of our transformations use:
Streaming MapReduce Jobs
What is Streaming?
“A Hadoop utility that allows you to create
and run MapReduce jobs using any
executable script as a mapper or reducer”
for line in sys.stdin:
data = json.loads(line)
print data['plumbeeUid'] + 't' + 1
Emits, Key value Pairs
466264 => 1, 376166 => 1
983131 => 1, 466264 => 1
Hadoop sorts and shuffles the data making sure
matching keys are processed by a single reducer!
results = defaultdict(int)
for line in sys.stdin:
plumbee_uid, count = line.split('t')
results[plumbee_uid] += int(count)
print results
JSON rpc-spin
Data
Result:
{ 466264: 2, 376166: 1, 983131: 1 }
map()
reduce()

Results
Load (I) - Problem
Raw S3 JSON Data Aggregated Data
EMR Transformed data:
● Referred to as aggregates
● Stored in S3
● Accessible via EMR cluster
EMR Transformation
(Hive & Streaming Jobs)
5.4TB
Problem
● We don’t run long-lived EMR clusters.
EMR requires:
● Specialists knowledge
● Is slow, processing and booting “offline”.
Use Amazon Redshift for fast “online” data access

What is Redshift?
A column-oriented database which uses
Massive Parallel Processing (MPP) techniques
to support analytics style SQL based
workloads across large datasets.
Power comes from:
● Query parallelization
● Column-oriented design
Redshift Provides:
● Low latency JDBC and ODBC access
● Fault Tolerance
● Automated Backups
Load (II) - Redshift
Redshift (x3 nodes): 0.33s
EMR (x20 nodes): 135.46s

Load (II) - Column-Oriented Databases
Row-oriented Database - MySQL
ID First Name Last Name Country
1 Penguin Situation GB
2 Cheese Labs US
3 Horse Barracks GB
Column-oriented Database - Redshift
ID First Name Last Name Country
1 Penguin Situation GB
2 Cheese Labs US
3 Horse Barracks GB
● East to add/modify records
● Could read irrelevant data.
● Great for fast lookups (OLTP)
● Only read in relevant data
● Adding rows requires multiple
updates to column data.
● Great for aggregation queries
(OLAP)

Architecture - Revisit
Aggregates
Amazon S3
Amazon EMR
(Elastic MapReduce)
Amazon Redshift
Plumbee Employees
Log Aggregators
Events (JSON)
SQS Analytics Queue
Events (JSON)

Mirrorball Slots: Kingdom of Riches

Mirrorball Slots: Challenges
● recurring timed event
● collect symbols from non-winning
spins
● get free coins if enough symbols are
collected

Some players ask for notifications

Data Collection
Players
Amazon Redshift

Architecture - Overview
Amazon Redshift
Amazon S3
Trigger Publisher Segmentation Workers
Batch Processors Amazon SNS
Players
Targeting
Mobile Push

User targeting
Run SQL queries directly against Redshift
SQL Query
Amazon Redshift User Segment

User targeting: Query example
-- Target all mobile users
SELECT plumbee_uid, arn
FROM mobile_user

User targeting: Query example (II)
-- Target lapsed users (1 week lapse)
SELECT plumbee_uid, arn
FROM mobile_user
WHERE last_play_time < (now - 7 days)

Demo (I)
Mobile MBS Notifications

Architecture - Mobile Push
Amazon Redshift
Amazon S3
Players
Targeting
Mobile Push

Amazon Simple
Notification Service

What is SNS?
“Amazon Simple Notification Service (Amazon
SNS) is a fast, flexible, fully managed push
messaging service”

Amazon SNS: Device Registration
Players Game Servers SQS Analytics Queue Amazon Redshift
Amazon SNS
register device
event
register

Amazon SNS: ARN Retrieval
private String getArnForDeviceEndpoint(String platformApplicationArn, String deviceToken) {
CreatePlatformEndpointRequest request =
new CreatePlatformEndpointRequest()
.withPlatformApplicationArn(platformApplicationArn)
.withToken(deviceToken);
CreatePlatformEndpointResult result = snsClient.createPlatformEndpoint(request);
return result.getEndpointArn();
}

Amazon SNS: Analytics Event
private String registerEndpointForApplicationAndPlatform( final long plumbeeUid,
String platformARN, String platformToken) {
final String deviceEndpointARN = getArnForDeviceEndpoint( platformARN , platformToken );
sqsLogger.queueMessage( new HashMap<String, Object>() {{
put( "notification", "register");
put( "plumbeeUid", plumbeeUid );
put( "provider", platformName );
put( "endpoint", deviceEndpointARN );
}}, null);
return deviceEndpointARN;
}

Amazon SNS: Mobile Push
private void publishMessage(UserData userData, String jsonPayload) {
amazonSNS.publish(new PublishRequest()
.withTargetArn( userData.getEndpoint())
.withMessageStructure( "json")
.withMessage( jsonPayload ));
}
Payload example
{"default": "The 5 day Halloween Challenge has started today! Touch to play NOW!"}

Architecture - Orchestration
Amazon Redshift
Amazon S3
Players
Targeting
Mobile Push

What is Amazon SWF?
“Amazon Simple Workflow (Amazon SWF) is a
task coordination and state management
service for cloud applications.”

What Amazon SWF provides
● consistent execution state management
● workflow executions and tasks tracking
● non-duplicated dispatch of tasks
● task routing and queuing
● the AWS Flow Framework

Mobile Push: Scheduling
Trigger Publish Service Amazon
Simple Workflow

Mobile Push: Targeting
query query
target
users
Amazon SWF
Amazon EC2
Worker
(Segmentation)
Amazon
Redshift
Amazon
S3

Mobile Push: Processing
batch 1-N publish push
Workers
(Processing)
Amazon SWF Read data + push End User

Mobile Push: Reporting
send send
Amazon SWF
Amazon EC2
Worker
(Reporting)
Amazon
SES

Transforming Mobile Push Notifications with Big Data

More Related Content

What's hot (19)

Viewers also liked (11)

Similar to Transforming Mobile Push Notifications with Big Data (20)

Recently uploaded (20)

Transforming Mobile Push Notifications with Big Data