Aws Kinesis
0 likes370 views
This document provides an overview of AWS Kinesis and its components for streaming data. It discusses Kinesis Streams for ingesting and processing streaming data at scale. Kinesis Streams uses shards to provide throughput capacity. To ingest 10,000 records per second of 512 byte size each would require a Kinesis stream configured with 10 shards. Kinesis Firehose is for delivering streaming data to destinations like S3 or Redshift. Kinesis Analytics allows running SQL queries on streaming data and processing it in real-time.
![AWS KINESIS ANALYTICS
STREAMING SQL
▸ Tumbling Window
[...] GROUP BY
FLOOR((“SOURCE_SQL_STREAM_001”.ROWTIME – TIMESTAMP
‘1970-01-01 00:00:00’) SECOND / 10 TO SECOND)
▸ Sliding Window
SELECT AVG(change) OVER W1 as avg_change
FROM "SOURCE_SQL_STREAM_001"
WINDOW W1 AS (PARTITION BY ticker_symbol RANGE INTERVAL
'10' SECOND PRECEDING)](https://image.slidesharecdn.com/awskinesis-170505071201/85/Aws-Kinesis-26-320.jpg)
Aws Kinesis
- 1. AWS KINESIS SCRATCHING THE SURFACE OF
- 3. QUESTION What (Kafka?) cluster do you need to ingest: - 10,000 record/sec - record size 512Bytes
- 4. KINESIS
- 5. AWS KINESIS WHAT IS KINESIS? ▸ Platform for streaming data on AWS ▸ "sometimes TBs per hour"
- 6. AWS KINESIS MOTIVATION ▸ Highly Scalable ▸ Durable ▸ Elastic ▸ Replay-able reads
- 7. AWS KINESIS KINESIS COMPONENTS Firehose AnalyticsStreams
- 10. AWS KINESIS STREAMS TERMINOLOGY ▸ Streams - ordered sequence of data records ▸ Data record - Sequence Number, Partition Key, Data Blob ▸ 1MB max ▸ Retention period - 24h - 7d ▸ Producers, Consumers ▸ Shards
- 11. AWS KINESIS STREAMS KINESIS STREAMS - SHARDS ▸ Fixed unit of capacity ▸ Read ▸ 5 transaction / sec ▸ 2MB / sec ▸ Write ▸ 1000 records / sec ▸ 1MB / sec
- 12. QUESTION What cluster do you need to ingest - 10,000 records/sec - record size 512Bytes
- 13. AWS KINESIS STREAMS KINESIS STREAMS - SHARD CALCULATION Requirement Kinesis Stream Write Capacity 10,000 records/sec 1,000 Record/sec 512 Bytes/rec (5MB/sec) 1MB/sec
- 14. AWS KINESIS STREAMS KINESIS STREAMS - SHARD CALCULATION ▸ 10 Shards ▸ 10,000 records / sec ▸ 10MB / sec
- 15. DEMO
- 16. AWS KINESIS STREAMS DEMO ▸ Create a stream with 1 shard ▸ Put 10 records / sec ▸ Read in batches every sec { RecordId: 11, KeyPressCount: 22, UserId: 33 }
- 17. KINESIS FIREHOSE
- 19. AWS KINESIS FIREHOSE TERMINOLOGY ▸ Firehose delivery stream ▸ record - 1MB max ▸ data producer ▸ buffer size, buffer interval
- 20. AWS KINESIS FIREHOSE DATA DELIVERY ▸ KINESIS STREAM ▸ S3 ▸ Redshift ▸ Elasticsearch
- 21. DEMO
- 25. AWS KINESIS ANALYTICS TERMINOLOGY ▸ Input ▸ Application Code ▸ In-App-Streams ▸ Pumps ▸ Streaming SQL ▸ Output
- 26. AWS KINESIS ANALYTICS STREAMING SQL ▸ Tumbling Window [...] GROUP BY FLOOR((“SOURCE_SQL_STREAM_001”.ROWTIME – TIMESTAMP ‘1970-01-01 00:00:00’) SECOND / 10 TO SECOND) ▸ Sliding Window SELECT AVG(change) OVER W1 as avg_change FROM "SOURCE_SQL_STREAM_001" WINDOW W1 AS (PARTITION BY ticker_symbol RANGE INTERVAL '10' SECOND PRECEDING)
- 27. AWS KINESIS ANALYTICS STREAMING SQL - TUMBLING WINDOW
- 28. DEMO
- 29. AWS KINESIS ANALYTICS DEMO ▸ Check BDMeetup Application in AWS Console ▸ Producer / Consumer PRODUCER APP BDMEETUP STREAM ANALYTICS APP BDMEETUP-OUTPUT STREAM CONSUMER APP
- 30. THANK YOU