Big Data Analytics Infrastructure
Download as PPTX, PDF2 likes707 views
This document discusses the data infrastructure for a big data startup, including tracking data formats, message servers, Kafka, real-time processing with Storm and Spark Streaming, real-time data stores like MySQL and Cassandra, and batch processing with AWS Redshift, HDFS, and Spark/Presto. It also covers considerations around using cloud services like AWS versus running an in-house cluster.
Big Data Analytics Infrastructure
- 1. Data Infrastructure Training Aug 2015 A Big Data Startup Infra Advisor: Min Zhou
- 2. Send Tracking Streaming Batch Track SDK Kafka User’s end Reports Load Format Conversion + COPY Tracking schema: JSON format Avro format Query My SQL Cluster bin log replication AWS Redshift Batch Query Message Server
- 3. Send Tracking Streaming Real-time Track SDK Kafka Storm Streaming User’s end Reports Tracking schema: JSON format Avro format My SQL Cluster bin log replication MySQL Cassandra Message Server
- 4. Send Tracking Streaming Lambda Architecture SDK Kafka Storm Streaming User’s end Reports Load Format Conversion + COPY Tracking schema: JSON format Avro format Query My SQL Cluster bin log replication MySQL Cassandra AWS Redshift Batch Query Message Server
- 5. Tracking data • Protocol – HTTP – TCP – UDP • Format – JSON – Apache Avro – Apache Thrift – Google Protobuf
- 6. Message Server • Need be implemented by yourself • Nginx / Netty • Need Load balancer in high traffic
- 7. Kafka • High throughput • Scalable • Persist • Replayable • Guarantee message order • Need zookeeper
- 8. Real-time processing System • Storm • Samza • Spark Streaming
- 9. Real-time Data store • MySQL • KV store – Cassandra – HBase
- 10. Batch processing System • AWS Redshift – Only available on Amazon cloud – Easy to operate – Easy to use – Extra Format Conversion + COPY cost • HDFS + Spark/Presto – Need extra efforts to operate and user – No Format Conversion + COPY cost
- 11. Cloud vs In house cluster • Operation cost • Hardware cost