AWS Summit Singapore 2019 | Big Data Analytics Architectural Patterns and Best Practices

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Big Data & Analytics Architectural
Patterns and Best Practices
Renee Lo
Head of Big Data, Analytics and AI
Amazon Web Services
Natalia Kozyura
Head of Innovation Center
FWD Group

Agenda
• Big data challenges
• Why AWS analytics
• Implementation patterns: serverless, streaming, data warehouse
modernization, Hadoop
• FWD Group

Customers want more value from their data
Growing
exponentially
From new
sources
Increasingly
diverse
Used by
many people
Analyzed by
many applications

Cloud data lakes are the future
Data Lake
Customers want:
To move to a single store; i.e., a data lake in the cloud
To store data securely in standard formats
To grow to any scale, with low costs
To analyze their data in a variety of ways
To democratize data access and analysis

Why choose AWS for data lakes and analytics?
Most
comprehensive
Most
secure
Easiest
to build
Most
cost-effective
Most
customers
& partners

Most comprehensive
Broadest and deepest portfolio, purpose-built for builders
Migration & Streaming Services
Infrastructure Data Catalog
& ETL
Security &
Management
Dashboards Predictive Analytics
Data
Warehousing
Big Data
Processing
Interactive
Query
Operational
Analytics
Real time
Analytics
Serverless
Data processing
Visualization & Machine Learning
Data Movement
Analytics
Data Lake Infrastructure & Management

Data Movement
Analytics
Most comprehensive
Broadest and deepest portfolio, purpose-built for builders
+ 10 more
Redshift
EMR (Spark &
Hadoop)
Athena
Elasticsearch
Service
Kinesis Data
Analytics
AWS Glue
(Spark &
Python)
S3/Glacier AWS GlueLake
Formation
Visualization & Machine Learning
QuickSight SageMake
r
Comprehend Lex Polly Rekognition Translate Transcrib
e
Deep Learning
AMIs
Database Migration Service | Snowball | Snowmobile | Kinesis Data Firehose | Kinesis Data Streams | Managed Streaming for Kafka
Data Lake Infrastructure & Management

Most secure — Certifications
CSA
Cloud Security
Alliance Controls
ISO 9001
Global Quality
Standard
ISO 27001
Security Management
Controls
ISO 27017
Cloud Specific
Controls
ISO 27018
Personal Data
Protection
PCI DSS Level 1
Payment Card
Standards
SOC 1
Audit Controls
Report
SOC 2
Security, Availability, &
Confidentiality Report
SOC 3
General Controls
Report
Global United States
CJIS
Criminal Justice
Information Services
DoD SRG
DoD Data
Processing
FedRAMP
Government Data
Standards
FERPA
Educational
Privacy Act
FIPS
Government Security
Standards
FISMA
Federal Information
Security Management
GxP
Quality Guidelines
and Regulations
ISO FFIEC
Financial Institutions
Regulation
HIPPA
Protected Health
Information
ITAR
International Arms
Regulations
MPAA
Protected Media
Content
NIST
National Institute of
Standards and Technology
SEC Rule 17a-4(f)
Financial Data
Standards
VPAT/Section 508
Accountability
Standards
Asia Pacific
FISC [Japan]
Financial Industry
Information Systems
IRAP [Australia]
Australian Security
Standards
K-ISMS [Korea]
Korean Information
Security
MTCS Tier 3 [Singapore]
Multi-Tier Cloud
Security Standard
My Number Act [Japan]
Personal Information
Protection
Europe
C5 [Germany]
Operational Security
Attestation
Cyber Essentials
Plus [UK]
Cyber Threat
Protection
G-Cloud [UK]
UK Government
Standards
IT-Grundschutz
[Germany]
Baseline Protection
Methodology
X P
G

Most cost effective
Decouple compute and storage, choice of PAYG analytics services
Storage
S3 tiers &
intelligent tiering
From $0.023 per
GB/mo to as low as
$0.004 per GB/mo
Compute
Spot & reserved
instances
Save up to 90% off
on-demand prices
EMR
Autoscaling
57% less than
on-premises
per IDC report
Redshift
less than a tenth
of the cost of
traditional solutions.
Athena &
QuickSight
Serverless pay
only for what is used

Easiest to Build: Serverless analytics
Deliver on-demand analytics on the data lake
S3
Data lake
Glue
(ETL &
Data Catalog)
Athena
QuickSight
Serverless. Zero
infrastructure. Zero
administration
Never pay for
idle resources
$
Availability and
fault tolerance
built in
Automatically scales
resources with
usage
AWS IoT
AI/ML
Devices Web Sensors Social

“One of the big attractions
of Amazon Athena is that
it’s serverless and purely
consumption-based.”
—Matt Chesler, director of DevOps
at Movable Ink
“We only pay when we’re actually querying the
data, and we don’t have to keep a cluster
running all the time. Using Amazon Athena,
we’re able to query seven years’ worth of
data—adding up to hundreds of terabytes—
get results at least 50 percent faster, and
save nearly $15,000 per month.”

Set up a catalog, ETL, and data prep
with AWS Glue
Serverless provisioning, configuration,
and scaling to run your ETL jobs on
Apache Spark
Pay only for the resources used for jobs
Crawl your data sources, identify data
formats and suggest schemas and
transformations
Automates the effort in building,
maintaining and running ETL jobs

Easiest to Build: Streaming
Mobile Apps Web Clickstream Application Logs
Metering Records IoT Sensors Smart Buildings
[Wed Oct 11 14:32:52
2018] [error] [client
127.0.0.1] client
denied by server
configuration:
/export/home/live/ap/ht
docs/test

The diminishing value of data over time

125+ million players
Data provides a constant feedback loop
for game designers
Up-to-the-minute analysis of gamer
satisfaction to drive gamer engagement
Resulting in the most popular
game played in the world
Fortnite

Stream Data in Seconds
Get actionable insights quickly
Streaming
Ingest video
& data as it’s
generated
Process data
on the fly
Real-time
analytics/ML,
alerts, actions

“Amazon Kinesis makes it simple to
scale our solution end to end,
including the capture, processing,
and delivery of actionable
insights. This empowers our
customers to better understand
their user base.”
— Indu Narayan, Director of Data, Yieldmo

Easiest to Build: Data warehouse modernization

Data Warehousing on AWS serves the analytical needs of today
OLTP ERP CRM LOB
Data Warehouse
Business
Intelligence
Data Lake
1001100001001010111001
0101011100101010000101
1111011010
0011110010110010110
0100011000010
Devices Web Sensors Social
Catalog
Machine
Learning
DW
Queries
Big data
processing
Interactive Real-time
The analytical power of data warehouse
The limitless scalability of serverless compute
The distributed processing of big data systems
=
+
+

The Forrester Wave™ is copyrighted by Forrester Research, Inc. Forrester and Forrester
Wave™ are trademarks of Forrester Research, Inc. The Forrester Wave™ is a graphical
representation of Forrester's call on a market and is plotted using a detailed spreadsheet
with exposed scores, weightings, and comments. Forrester does not endorse any vendor,
product, or service depicted in the Forrester Wave. Information is based on best available
resources. Opinions reflect judgment at the time and are subject to change.
“Customer references like AWS’s elastic scale, administration, serverless architecture,
security and compliance, and high availability and disaster recovery capabilities…”
AWS received a score of 5/5 (highest score possible) in the market presence category,
use cases, ability to execute, and road map criteria.
AWS Recognized as a Leader in Data Warehousing and Analytics by Forrester and Gartner
Gartner Magic Quadrant for Data Management
Solutions for Analytics, 2018
Gartner, Magic Quadrant for Data Management Solutions for Analytics, 13 February 2018. This graphic was published by
Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The
Gartner document is available upon request from Amazon. Gartner does not endorse any vendor, product or service
depicted in its research publications, and does not advise technology users to select only those vendors with the highest
ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and
should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this
research, including any warranties of merchantability or fitness for a particular purpose.
Forrester Wave™ Cloud Data Warehouse Q4 2018

CHALLENGE
Needed to analyze data to find
insights, identify opportunities, and
evaluate business performance.
The Oracle DW did not scale, was
difficult to maintain, and costly.
SOLUTION
Deployed a data lake with Amazon S3,
and run analytics with Amazon
Redshift, Amazon Redshift Spectrum,
and Amazon EMR.
Result: They doubled the data stored
(100PB), lowered costs, and was able
to gain insights faster.
50 PB of data
600,000 analytics jobs/day

Equinox Fitness migrated from Teradata to Redshift
Maximilian
(ELT scripts)
Spark
on EMR
Redshift
S3
Clickstream
Cycling logs
Club
management
software
Applications
Social
Redshift
Spectrum
EMR
Athena
Equinox
apps
3rd party
apps
Migrated from Teradata data warehouse
Built a DW with Redshift and data lake with S3
Analytics on data lake with Amazon Athena,
Amazon Redshift Spectrum, and Amazon EMR
Increased user productivity to move faster
Amazon Redshift costs ~20% of its original
Teradata maintenance & support

Migration to Redshift
Spark
on EMR
Redshift
S3
Clickstream
Cycling logs
Club
management
software
Applications
Social
Redshift
Spectrum
EMR
Athena
3rd party
apps

With over 20,000 Rio Tinto CRM
users globally, QuickSight is
providing an interactive solution
to explore thousands of data
points quickly and to ensure
safety in every decision

Easiest to Build: n-
premises Hadoop
Cloud Hadoop/Spark Platforms
Q1 2019
The 11 Providers That Matter Most
and How They Stack Up
by Noel Yuhanna and Mike Gualtieri
February 13, 2019
The Forrester Wave™ is copyrighted by Forrester Research, Inc. Forrester and
Forrester Wave™ are trademarks of Forrester Research, Inc. The Forrester Wave™
is a graphical representation of Forrester's call on a market and is plotted using a
detailed spreadsheet with exposed scores, weightings, and comments. Forrester does
not endorse any vendor, product, or service depicted in the Forrester Wave™.
Information is based on best available resources. Opinions reflect judgment at the
time and are subject to change.

Hybrid Architecture
On-Premise AWS Cloud
HDFS / Storage Tier
Access ControlData Caching & Virtualization
On-prem Hadoop Clusters
Existing Data Science Environment DX
Glue / EMR Clusters
Sagemaker / Data Science Tools
Notebooks
Model
Deployment
Training, Evaluation,
Tuning
EMR Cluster
ML Notebooks
On-prem to AWS Storage sync
AWS Data Sync
Access Control
Connects to
on-prem
Kerberos AP
EMR Notebook
Storage Tier

FINRA’s legacy system did not
scale to handle 75 billion events
per day. They needed to run
complex surveillance queries over
20+ PB of data
FINRA migrated their big data
appliance to a S3 Data Lake
and uses EMR for ingestion
and processing

An intro to FWD
FWD GROUP OUR MARKETS
• Established in Asia in 2013
• FWD is the insurance business of investment
group, Pacific Century Group
• More than 3 million customers and 1.75 million
group members
• More than 4,500 employees
• Over US$28.3 billion in assets
• Focused on creating fresh customer experiences,
with easy-to-understand products, supported by
digital technology
Present in 9 markets across APAC

here to change the way people feel about
insurance
simple. reliable. direct.

Co-Creation with Customers for Term Life Sales Journey
Why FWD Chose AWS Cloud vs. Traditional On-prem approach
Equipment
Resources and
Administration
Contracts Cost
No Up Front Expense
Pay for what you Use
Improve Time to
Market & Agility
Scale Up and
Down
Self-Service
Infrastructure
Traditional
Infrastructure
AWS Cloud

FWD Singapore Data Lake
• Deployed Phase 1 to
production
• Using 5+ AWS services for Data
Lake
• Data Lake is an enabler for
many Innovation projects, one
example is our Chatbot

Co-Creation with Customers for Term Life Sales Journey
AI Chatbot
AI Chatbot running on AWS enabling 24 x 7 x 365
servicing

The FWD Innovation story continues…
“
best-in-class technology
”
Natalia Kozyura
Head of Innovation Centre
FWD Group

More data lakes and analytics than anywhere else
More than 10,000 data lakes on AWS

Related breakouts
Realizing business value with AWS Analytics Services
Yu Hua Lim
Zero to data lake in 40 minutes
Unni Pillai
Digital User Engagement
Zach Barbitta and John Burry

AWS Summit Singapore 2019 | Big Data Analytics Architectural Patterns and Best Practices

More Related Content

Similar to AWS Summit Singapore 2019 | Big Data Analytics Architectural Patterns and Best Practices (20)

More from AWS Summits (20)

AWS Summit Singapore 2019 | Big Data Analytics Architectural Patterns and Best Practices