IOT meetup presentation
1 like232 views
The document discusses the challenges of handling IoT data and outlines a modern data architecture incorporating lambda and kappa architectures. Key components include distributed logs, stream processing, serving layers, and the use of Cassandra for data storage due to its scalability and availability. Specific examples and data models for time series data management are provided, emphasizing design considerations and query language usage.
IOT meetup presentation
- 1. Handling IOT Data with a Modern Data Architecture Cliff Gilmore - Data Practice Director @ 1904labs
- 2. Capture, Process and Serve (All the Things)
- 3. Challenges of IOT Data 3 Scale Frequency of events Size of Data Number of Devices Number of Users Latency Demands Geo Distribution Processing Batch Analytics Realtime Analytics Aggregations Machine Learning Reporting Applications Realtime Access Report Visualization Production Analytics ML Driven Decisions Microservices
- 4. IOT Data Architecture Leading to Lambda and Kappa
- 5. Architectural Requirements ❏ Must scale out linearly on ingestion, processing, storage, and access. ❏ Need to be able to store huge amounts of data organized for different access patterns. ❏ Must have the ability to process data inflight for real time decision making, alerting and pattern matching ❏ Need to serve the data to the rest of the organization through a common API/Service ❏ The architecture must be agile to accommodate new changes to business logic and processing algorithms
- 6. Stack Components ❏ Distributed Log / Queue ❏ A pub/sub partitioned queue ❏ Kafka is the defacto choice due to it’s wide use in production ❏ Stream Processing ❏ Ability to process events as they arrive ❏ Event at a Time ❏ Samza, Flink, Storm ❏ Micro Batch ❏ Spark Streaming ❏ Batch Processing ❏ Process event history in bulk ❏ Spark, MapReduce on top of HDFS or Wide Column Stores ❏ Serving ❏ Expose data to the rest of the organization and serve application requests ❏ Wide Column Store ❏ Cassandra, HBase, BigTable ❏ Can also be RDBMS for some data sets (Reports, BI Rollups, etc)
- 7. Lambda Architecture Events Distributed Log Batch Layer Speed Layer Serving Layer Raw Data, Pattern Matching and Aggregates Patterns, Rollups, Recommendations
- 8. Kappa Architecture Events Distributed Log Batch Streaming Serving Stream Results Stream V1 Stream V2 Table V1 Table V2 Raw DataRaw
- 9. Cassandra to Serve IOT Data The Art of Time Series
- 10. Why Cassandra? ❏ Proven linear scale up to 1000s of nodes in a single cluster ❏ Geo redundancy to collect data where it is created and replicate across the globe ❏ High capacity to ingest parallel individual writes ❏ Low latency and high throughput reads ❏ Wide-column store data model allows for data to be structured around query patterns ❏ Continuous availability suited to and used for the most mission critical systems ❏ AP platform by definition of the CAP theorem, consistency is tunable to give availability
- 12. Physical Data Model Partition Key Clustering Key Val1 Col1:Val Col2:Val Col3:Val Col4:Val Col5:Val Col6:Val …. Clustering Key Val2 Col1:Val Col6:Val …. Clustering Key Val3 Col6:Val …. Clustering Key Val4 Col1:Val Col2:Val Col3:Val Col4:Val Col5:Val Col6:Val …. Clustering Key Val5 Col1:Val Col2:Vall …. ... ….
- 13. CQL - Cassandra Query Language ❏ Simple to use language that looks like SQL ❏ No joins, group by etc ❏ Example Queries ❏ SELECT * FROM readings WHERE event_time > ? AND event_tiime <= ? WHERE device_id= ?; ❏ INSERT INTO readings (device_id, event_time, temperature) VALUES (?,?,?); I’ve got this!
- 14. TimeSeries Table Example CREATE TABLE readings ( sensor_id text, event_time TimeUUID, temperature decimal, PRIMARY KEY (sensor_id,event_time) );
- 15. TimeSeries Table Example CREATE TABLE readings ( sensor_id text, event_time TimeUUID, temperature decimal, PRIMARY KEY (sensor_id,event_time) ); Time Ordered Sortable UUID
- 16. TimeSeries Table Example CREATE TABLE readings ( sensor_id text, event_time TimeUUID, temperature decimal, PRIMARY KEY (sensor_id,event_time) ); Partition Key
- 17. TimeSeries Table Example CREATE TABLE readings ( sensor_id text, event_time TimeUUID, temperature decimal, PRIMARY KEY (sensor_id,event_time) ); Clustering Key
- 18. Physical Data Model Station #1 12:05.15 15.9 C 12:05.16 15.9 C 12:05.17 16.0 C 12:05.18 16.1 C 12:05.19 16.0 C ... …. Station #2 12:05.15 22.0 C 12:05.20 22.1 C 12:05.25 27.9 C 12:05.30 27.7 C 12:05.35 30.2 C ... ….
- 19. Advanced Data Model Topics ❏ Consider bucketing time in Partition Key if sample rate is high ❏ Primary Key ((device_id,year,week),event_time) ❏ If per event granularity not needed can batch or rollup events ❏ Primary Key (device_id, event_minute) ❏ If we batch events ❏ JSON blob of sensor readings within the minute ❏ Can’t update sensor readings without read-before-write