Builders' Day - Building Data Lakes for Analytics On AWS LC
1 like136 views
This document discusses big data analytics and architectural principles for building big data solutions. It covers collecting and storing data from various sources, processing and analyzing data using services like Amazon Kinesis, Redshift, EMR and Athena, and choosing the right tools based on factors like data structure, access patterns, and latency requirements. Key principles emphasized include building decoupled systems, leveraging managed services, using event-driven architectures, and focusing on cost efficiency.
![© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Which stream/message storage should I use?
Amazon
Kinesis Data
Streams
Amazon
Kinesis Data
Firehose
Apache Kafka (on
Amazon Elastic
Compute Cloud
[Amazon EC2])
Amazon Simple
Queue Service
(Amazon SQS)
(Standard)
Amazon SQS
(FIFO)
AWS managed Yes Yes No Yes Yes
Guaranteed ordering Yes No Yes No Yes
Delivery (deduping) At least once At least once At least/At
most/exactly once
At least once Exactly once
Data retention period 7 days N/A Configurable 14 days 14 days
Availability Three AZ Three AZ Configurable Three AZ Three AZ
Scale/throughput No limit /
~ shards
No limit /
automatic
No limit /
~ nodes
No limits /
automatic
300 TPS /
queue
Multiple consumers Yes No Yes No No
Row/object size 1 MB Destination
row/object size
Configurable 256 KB 256 KB
Cost Low Low Low (+admin) Low-medium Low-medium](https://image.slidesharecdn.com/3buildersdaybuildingdatalakesforanalyticsonawslc-190718154036/85/Builders-Day-Building-Data-Lakes-for-Analytics-On-AWS-LC-12-320.jpg)
Builders' Day - Building Data Lakes for Analytics On AWS LC
- 1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Luis Caro, PhD Solution Architect - AWS
- 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Types of big data analytics Batch/ interactive Stream processing Machine learning
- 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Delivery of big data services Virtualized Managed services Serverless/ clusterless/ containerized
- 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Plethora of tools
- 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Big data challenges Why? How? What tools should I use? Is there a reference architecture?
- 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Architectural principles Build decoupled systems • Data → Store → Process → Store → Analyze → Answers Use the right tool for the job • Data structure, latency, throughput, access patterns Leverage managed and serverless services • Scalable/elastic, available, reliable, secure, no/low admin Use event-journal design patterns • Immutable datasets (data lake), materialized views Be cost-conscious • Big data ≠ big cost Machine learning (ML) enable your applications
- 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Collect Store Process/ Analyze Consume Time to answer (latency) Throughput Cost Simplify big data processing
- 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Collect
- 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Collect Devices Sensors IoT platforms AWS IoT STREAMS IoT EventsData streams Migration Snowball Logging Amazon CloudWatch AWS CloudTrail FILES Datatransport&logging Import/expo rt Files/objects Log files Media files Mobile apps Web apps Data centers AWS Direct Connect RECORDS Applications Transactions Data structures Database records Type of data
- 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Events Files Transactions Collect Devices Sensors IoT platforms AWS IoT STREAMS IoT Data streams Migration Snowball Logging Amazon CloudWatch AWS CloudTrail FILES Datatransport&logging Import/expo rt Log files Media files Mobile apps Web apps Data centers AWS Direct Connect RECORDS Applications Data structures Database records Type of data Store NoSQL In-memory SQL File/object store Stream storage
- 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Collect Devices Sensors IoT platforms AWS IoT STREAMS IoT Migration Snowball Logging Amazon CloudWatch AWS CloudTrail FILES Datatransport&logging Import/expo rt Mobile apps Web apps Data centers AWS Direct Connect RECORDS Applications Store NoSQL In-memory SQL File/object store Amazon Kinesis Firehose Amazon Kinesis Streams Apache Kafka Apache Kafka • High throughput distributed streaming platform Amazon Kinesis Data Streams • Managed stream storage Amazon Kinesis Data Firehose • Managed data delivery Stream storage
- 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Which stream/message storage should I use? Amazon Kinesis Data Streams Amazon Kinesis Data Firehose Apache Kafka (on Amazon Elastic Compute Cloud [Amazon EC2]) Amazon Simple Queue Service (Amazon SQS) (Standard) Amazon SQS (FIFO) AWS managed Yes Yes No Yes Yes Guaranteed ordering Yes No Yes No Yes Delivery (deduping) At least once At least once At least/At most/exactly once At least once Exactly once Data retention period 7 days N/A Configurable 14 days 14 days Availability Three AZ Three AZ Configurable Three AZ Three AZ Scale/throughput No limit / ~ shards No limit / automatic No limit / ~ nodes No limits / automatic 300 TPS / queue Multiple consumers Yes No Yes No No Row/object size 1 MB Destination row/object size Configurable 256 KB 256 KB Cost Low Low Low (+admin) Low-medium Low-medium
- 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Collect Store File Apache Kafka Amazon Kinesis Streams Amazon Kinesis Firehose Hot Stream Mobile apps Web apps Devices Sensors IoT platforms AWS IoT Data centers AWS Direct Connect Migration Snowball Logging Amazon CloudWatch AWS CloudTrail RECORDS FILES STREAMS Datatransport&loggingIoTApplications Import/expo rt NoSQL In-memory SQL Amazon S3 Amazon Simple Storage Service (Amazon S3) • Managed object storage service built to store and retrieve any amount of data File/object storage
- 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Use Amazon S3 as the storage for your data lake • Natively supported by big data frameworks (Spark, Hive, Presto, and others) • Decouple storage and compute • No need to run compute clusters for storage (unlike HDFS) • Can run transient Amazon EMR clusters with Amazon EC2 Spot Instances • Multiple & heterogeneous analysis clusters and services can use the same data • Designed for 99.999999999% durability • No need to pay for data replication within a region • Secure – SSL, client/server-side encryption at rest • Low cost
- 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What about HDFS & data tiering? • Use HDFS/local for working data sets • (for example, iterative read on the same data sets) • Use Amazon S3 Standard for frequently accessed data • Use Amazon S3 Standard–IA for less frequently accessed data • Use Amazon Glacier for archiving cold data Use S3 analytics to optimize tiering strategy S3 analytics
- 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Which data store should I use? Ask yourself some questions What is the data structure? How will the data be accessed? What is the temperature of the data? What will the solution cost?
- 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Which data store should I use? Access patterns What to use? Put/get (key, value) In-memory, NoSQL Simple relationships → 1:N, M:N NoSQL Multi-table joins, transaction, SQL SQL Faceting, search Search Graph traversal GraphDB Data structure What to use? Fixed schema SQL, NoSQL Schema-free (JSON) NoSQL, search Key/value In-memory, NoSQL Graph GraphDB How will the data be accessed?
- 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. In-memory SQL Request rate High Low Cost/GB High Low Latency Low High Data volume Low High Amazon Glacier Structure NoSQL Hot data Warm data Cold data Low High S3 Search Graph What is the temperature of the data?
- 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Process/ analyze
- 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Process/analyze Amazon Redshift & Spectrum Amazon Athena BatchInteractive Amazon ES Interactive & batch analytics Amazon Elasticsearch Service Managed service for ElasticSearch Amazon Redshift & Amazon Redshift Spectrum Managed data warehouse Spectrum enables querying S3 Amazon Athena Serverless interactive query service Amazon EMR Managed Hadoop framework for running Apache Spark, Flink, Presto, Tez, Hive, Pig, HBase, and others Presto Amazon EMR
- 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Stream/real-time analytics Spark Streaming on Amazon EMR Amazon Kinesis Data Analytics • Managed service for running SQL on streaming data Amazon KCL • Amazon Kinesis Client Library AWS Lambda • Run code serverless (without provisioning or managing servers) • Services such as S3 can publish events to AWS Lambda • AWS Lambda can pool event from a Kinesis Process/analyze KCL Apps AWS Lambda Amazon Kinesis Analytics Stream Streaming
- 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Which analytics should I use? Process/analyze Batch Takes minutes to hours Example: Daily/weekly/monthly reports Amazon EMR (MapReduce, Hive, Pig, Spark) Interactive Takes seconds Example: Self-service dashboards Amazon Redshift, Amazon Athena, Amazon EMR (Presto, Spark) Stream Takes milliseconds to seconds Example: Fraud alerts, one-minute metrics Amazon EMR (Spark Streaming), Amazon Kinesis Data Analytics, Amazon KCL, AWS Lambda, and others Predictive Takes milliseconds (real-time) to minutes (batch) Example: Fraud detection, forecasting demand, speech recognition Amazon SageMaker, Amazon Polly, Amazon Rekognition, Amazon Transcribe, Amazon Translate, Amazon EMR (Spark ML), Amazon Deep Learning AMI (MXNet, TensorFlow, Theano, Torch, CNTK, and Caffe) FastSlow Amazon Redshift & Spectrum Amazon Athena BatchInteractive Amazon ES Presto Amazon EMR Predictive AmazonML KCL Apps AWS Lambda Amazon Kinesis Analytics Stream Streaming Fast
- 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Which stream processing technology should I use? Amazon EMR (Spark Streaming) KCL application Amazon Kinesis analytics AWS Lambda Managed service Yes No (EC2 + Auto Scaling) Yes Yes Serverless No No Yes Yes Scale / throughput No limits / ~ nodes No limits / ~ nodes No Limits / automatic No limits / automatic Availability Single AZ Multi-AZ Multi-AZ Multi-AZ Programming languages Java, Python, Scala Java, others through MultiLangDaemon ANSI SQL with extensions Node.js, Java, Python, .Net Core Sliding window functions Build-in App needs to implement Built-in No Reliability KCL and Spark checkpoints Managed by Amazon KCL Managed by Amazon Kinesis Data Analytics Managed by AWS Lambda
- 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Which Analytics Tool Should I Use? Amazon Redshift Amazon Redshift Spectrum Amazon Athena Amazon EMR Presto Spark Hive Use case Optimized for data warehousing Query S3 data from Amazon Redshift Interactive queries over S3 data Interactive query General purpose Batch Scale/throughput ~Nodes ~Nodes Automatic ~ Nodes Managed service Yes Yes Yes, Serverless Yes Storage Local storage Amazon S3 Amazon S3 Amazon S3, HDFS Optimization Columnar storage, data compression, and zone maps AVRO, PARQUET TEXT, SEQ RCFILE, ORC, etc. AVRO, PARQUET TEXT, SEQ RCFILE, ORC, etc. Framework dependent Metadata Amazon Redshift catalog AWS Glue catalog AWS Glue catalog AWS Glue catalog or Hive Meta-store Auth/access controls IAM, users, groups, and access controls IAM, users, groups, and access controls IAM IAM, LDAP, & Kerberos UDF support Yes (Scalar) Yes (Scalar) No Yes Slow
- 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Raw data stored in data lake Preparation N orm ali zed P art i t i oned C om pressed St o rage o p t i m i zed ELT/ETL ELT/ETL: Preparing raw data for consumption Data lake on AWS Raw immutable DataSets Curated DataSets Data catalog
- 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Why ELT?
- 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Which ELT/ETL tool should I use? AWS Glue ETL AWS Data Pipeline Amazon Data Migration Service (Amazon DMS) Amazon EMR Apache NiFi Partner solution Use case Serverless ETL Data workflow service Migrate databases (and to/from data lake) Customize developed Hadoop/Spark ETL Automate the flow of data between systems Rich partner ecosystem for ETL Scale/throughput ~DPUs ~Nodes through EMR cluster EC2 instance type ~Nodes Self managed Self manager or through partner Managed service Clusterless Managed Managed EC2 on your behalf Managed EC2 on your behalf Self managed on Amazon EMR or Marketplace Self manager or through partner Data sources S3, RDBMS, Amazon RedShift, DynamoDB S3, JDBC, DynamoDB, custom RDBMS, data warehouses, Amazon S3* (*limited) Various Hadoop/Spark managed Various through rich processor framework Various Skills needed Wizard for simple mapping, code snippets for advanced ETL Wizard and code snippets Wizard and drag/drop Hadoop/Spark coding NiFi processors and some coding Self manager or through partner
- 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Consume
- 29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Collect Store ConsumeProcess/analyzeETL Amazon QuickSight Analysis&visualizationDatasceince AI Apps Apps SW engineers/ business users Data scientists/ data engineers
- 30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Putting it all together
- 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Collect Store ConsumeProcess/analyze Apache Kafka Amazon Kinesis Streams Amazon Kinesis Firehose Hot SQLNoSQLCacheFileStream Mobile apps Web apps Devices Sensors IoT platforms AWS IoT Data centers AWS Direct Connect Migration Snowball Logging Amazon CloudWatch AWS CloudTrail RECORDS FILES STREAMS Amazon QuickSight Analysis&visualizationDatasceince DataTransport&LoggingIoTApplications Amazon S3 Import/expo rt AI Apps Apps Model Train/ Eval Models Deploy ETL FastSlow Amazon Redshift & Spectrum Amazon Athena BatchInteractive Amazon ES Presto Amazon EMR Predictive AmazonML KCL Apps AWS Lambda Amazon Kinesis Analytics Stream Streaming Fast Amazon DynamoDB Amazon RDS Amazon Aurora Amazon Neptune Amazon ElastiCache Amazon DAX
- 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Design patterns
- 33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Interactive & batch analytics Process Store Batch Interactive Amazon EMR Hive Pig Spark Amazon AI Batch prediction Real-time prediction Amazon S3 Files Amazon Kinesis Data Firehose Amazon Kinesis Data Analytics Amazon Redshift Amazon ES Consume Amazon EMR Presto Spark Amazon Athena
- 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Interactive & batch Amazon S3 Amazon Redshift Amazon EMR Presto Hive Pig Spark Amazon ElastiCache Amazon DynamoDB Amazon RDS Amazon ES AWS Lambda Spark Streaming on Amazon EMR Applications Amazon Kinesis App state or materialized view KCL Amazon AI Real-time Amazon DynamoDB Amazon RDS Change data capture or export Transactions Stream Files Data lake Amazon Kinesis Data Analytics Amazon Athena Amazon Kinesis Data Firehose Amazon ES
- 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What about metadata? • AWS Glue catalog • Hive Metastore compliant • Crawlers - Detect new data, schema, partitions • Search - Metadata discovery • Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum compatible • Hive Metastore (Presto, Spark, Hive, Pig) • Can be hosted on Amazon RDS Data catalog Metastore Amazon RDS AWS Glue catalog AWS Glue
- 36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Summary Build decoupled systems • Data → Store → Process → Store → Analyze → Answers Use the right tool for the job • Data structure, latency, throughput, access patterns Leverage AWS managed and serverless services • Scalable/elastic, available, reliable, secure, no/low admin Use log-centric design patterns • Immutable logs, data lake, materialized views Be cost-conscious • Big data ≠ big cost AI/ML enable your applications
- 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.