Streams and serverless at DAZN
5 likes2,106 views
The document discusses serverless architectures using AWS Lambda and Amazon Kinesis. It describes how DAZN uses a serverless-first approach with services like API Gateway, SQS, SNS, Step Functions, and Kinesis. It provides examples of using Kinesis as a task queue, transaction log, and for building analytics pipelines. It also covers best practices for streaming data, security, observability, and challenges of building distributed systems with streams.
Streams and serverless at DAZN
- 1. serverless and streams at
- 2. Yan Cui http://theburningmonk.com @theburningmonk Principal Engineer @ Independent Consultant
- 3. “Netflix for sports” offices in London, Leeds, Katowice and Amsterdam
- 4. available in Austria, Switzerland, Germany, Japan, Canada, Italy, US, Spain and Brazil
- 5. available on 30+ platforms
- 7. “Netflix for sports” offices in London, Leeds, Katowice and Amsterdam We’re hiring! Visit engineering.dazn.com to learn more. follow @DAZN_ngnrs for updates about the engineering team.
- 9. we use a LOT
- 10. serverless-first
- 11. there are no silver bullets
- 12. serverless is not right for every use case (yet)
- 16. we use a LOT
- 17. API Gateway SQS SNS Step Functions
- 18. Kinesis
- 19. Kinesis as a task queue
- 20. Kinesis as a task queue as a transaction log
- 21. Kinesis as a task queue as a transaction log part of an analytics pipeline
- 24. stream
- 29. stream shard shard shard doX doX doX PutRecord: { PartitionKey: … Data: … } ordering is preserved within the same partition key
- 30. stream shard shard shard doX doX doX PutRecord: { PartitionKey: … Data: … } ordering is preserved within the same partition key data is retained for up to 7 days, default is 24 hours
- 35. stream doY doX shard doY doX doY doX shard shard every subscriber gets the same data, in the same order
- 36. stream doY doX shard doY doX doY doX shard shard every subscriber gets the same data, in the same order each subscriber process the stream at their own pace
- 37. stream doY doX doY doX doY doX shard shard INGRESS 1MB/s data, 1000 records/s shard
- 38. stream doY doX doY doX doY shard INGRESS 1MB/s data, 1000 records/s shard EGRESS 2MB/s data per subscriber, 5 reads/s doX shard
- 40. stream as a task queue
- 41. vs. vs. Kinesis SQS SNS
- 42. ordering replay events Kinesis SQS SNS mode retry concurrency subscribers
- 43. ordering replay events Kinesis SQS SNS mode retry concurrency subscribers many one-to-one many
- 44. ordering replay events Kinesis SQS SNS by shard none (standard) global (FIFO) none mode retry concurrency subscribers many one-to-one many
- 45. ordering replay events Kinesis SQS SNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry concurrency subscribers many one-to-one many
- 46. ordering replay events Kinesis SQS SNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry batched batched (up to 10) singular concurrency subscribers many one-to-one many
- 47. ordering replay events Kinesis SQS SNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry batched batched (up to 10) singular retried until success retry + DLQ retry + DLQ concurrency subscribers many one-to-one many
- 48. ordering replay events Kinesis SQS SNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry batched batched (up to 10) singular retried until success retry + DLQ retry + DLQ concurrency 1 per shard auto-scaled fan-out!!! subscribers many one-to-one many
- 49. ordering replay events Kinesis SQS SNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry batched batched (up to 10) singular retried until success retry + DLQ retry + DLQ concurrency 1 per shard auto-scaled fan-out!!! subscribers many one-to-one many
- 50. use the concurrency model to amortize spikes in traffic
- 51. leveraging the retry-until-success behaviour
- 53. 1 msg/s for a month, 1KB per msg 1 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per hr $10.836 1 x 60s x 60m x 24hr x 30days @ $0.5 per mil $1.296 1 x 60s x 60m x 24hr x 30days @ $0.4 per mil (send) + 1 x 60s x 60m x 24hr x 30days @ 10 batch @ 20s long polling @ $0.4 per mil (receive) $1.089
- 54. 1K msg/s for a month, 1KB per msg 1k x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per hr $47.088 1k x 60s x 60m x 24hr x 30days @ $0.5 per mil $1296.00 1k x 60s x 60m x 24hr x 30days @ $0.4 per mil (send) + 1k x 60s x 60m x 24hr x 30days @ 10 batch @ 20s long polling @ $0.4 per mil (receive) $1041.98
- 55. stream as a transaction log
- 57. events are an enabler for COMPOSABILITY
- 59. Kinesis
- 60. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B
- 61. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B
- 62. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B AWS Lambda AWS Lambda AWS Lambda
- 63. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B AWS Lambda AWS Lambda AWS Lambda DynamoDBIOT
- 64. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B AWS Lambda AWS Lambda AWS Lambda DynamoDBIOT
- 65. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B AWS Lambda AWS Lambda AWS Lambda DynamoDBIOT AWS Lambda AWS Lambda
- 66. build loosely-coupled system through events
- 67. service A service B service C service D bounded context bounded context
- 68. service A service B service C service D bounded context bounded context
- 69. service A service B service C service D
- 71. service A service B service C service D
- 72. service A service B service C service D
- 73. service A service B service C service Dbackward-compatible?
- 75. bounded context DON’T use events to orchestrate workflows within the same bounded context
- 76. bounded context adds unnecessary complexity to logging, tracing, and end-to-end reporting
- 77. bounded context the workflow doesn’t exist as a standalone concept, but as the sum of a series of loosely connected parts
- 78. Step Functions use Step Functions instead
- 79. Step Functions don’t forget to emit events from the workflow
- 80. Step Functions so others can react to state changes that happened as part of the workflow
- 82. S3 Buckets
- 83. S3 BucketsKinesis Streams Kinesis Firehose
- 84. S3 BucketsKinesis Streams AWS Lambda Kinesis Firehose
- 85. S3 BucketsKinesis Streams AWS Lambda AWS Lambda Kinesis Firehose
- 86. S3 BucketsKinesis Streams AWS Lambda AWS LambdaAWS Lambda Kinesis Firehose
- 87. S3 BucketsKinesis Streams AWS Lambda AWS LambdaAWS Lambda DynamoDB ElasticSearch Kinesis Firehose
- 88. S3 BucketsKinesis Streams AWS Lambda AWS Lambda Athena QuickSight AWS Lambda Kinesis Firehose
- 89. S3 BucketsKinesis Streams AWS Lambda AWS Lambda Athena QuickSight AWS Lambda Kinesis Firehose
- 90. secure sensitive data both at rest and in-transit
- 96. principle of Least Privilege
- 97. keep functions single-purposed & apply per function policies
- 101. stream doY doX doY doX doY shard shard EGRESS 2MB/s data per subscriber, doX shard 5 reads/s
- 102. throughput is maintained as you add more subscribers, but…
- 104. it’s difficult to add more than 15 subscriber functions to a stream
- 106. my microservices
- 108. user_events order_events promo_events my microservices still possible to have too many subscribers for a stream
- 109. user_events order_events promo_events my microservices implicit coupling between publisher and subscriber
- 110. user_events order_events promo_events my microservices lose ordering of related events in different streams
- 111. lose ordering of related events in different streams
- 113. preserve ordering of related events
- 114. extra cost for these consumer apps
- 116. observability
- 119. { error: … message: … correlation-id: … } { message: … correlation-id: … }