Building real time data-driven products
1 like2,823 views
The document discusses building data-driven products, emphasizing the importance of real-time data processing and event collection. It outlines architectural patterns such as the unified log, lambda, and kappa architectures, while highlighting the role of technologies like Kafka for managing data streams. It also addresses the need for schemas in streaming applications and offers considerations for data collection and processing.
![Reactive Streams
• [...] an initiative to provide a standard for asynchronous stream processing with
non-blocking back pressure. [...] aimed at runtime environments (JVM and
JavaScript) as well as network protocols.
• The scope [...] is to find a minimal set of interfaces, methods and protocols that will
describe the necessary operations and entities to achieve this goal.
• “Glue” between libraries.
Reactive Kafka ->
Akka Stream ->
RxJava](https://image.slidesharecdn.com/buildingreal-timedata-drivenproductsv1-161016150154/85/Building-real-time-data-driven-products-48-320.jpg)
Building real time data-driven products
- 1. Building Real-time Data-driven Products Øyvind Løkling & Lars Albertsson Version 1.3 - 2016.10.12
- 2. Øyvind Løkling Staff Software Engineer Schibsted Product & Technology Oslo - Stockholm - London - Barcelona - Krakow http://jobs.schibsted.com/
- 4. .. and more
- 5. Presentation goals ● Spark your interest in building data driven products. ● Give an overview of components and how these relate. ● Suggest technologies and approaches that can be used in practice. ● Event Collection ● The Unified Log ● Stream Processing ● Serving Results ● Schema 5
- 6. Data driven products • Services and applications primarily driven by capturing and making sense of data • Health trackers • Recommendations • Analytics CC BY © https://500px.com/tommrazek
- 7. Data driven products • Hosted services need to • Handle large volumes of data • Cleaning and structuring data • Serve individual users fast CC BY © https://500px.com/tommrazek
- 8. Big Data, Fast Data, Smart Data • Accelerating data volumes and speeds • Internet of Things • AB Testing and Experiments • Personalised products CC BY © https://500px.com/erfgorini
- 9. Big Data, Fast Data, Smart Data A need to make sense of data and act on fast • Faster development cycle • Data driven organisations • Data driven products CC BY © https://500px.com/erfgorini
- 10. Time scales - what parts become obsolete? 10 Credits: Confluent Credits: Netflix
- 11. The Log • Other common names • Commit log • Journal https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
- 12. Jay Kreps - I <3 Logs The state machine replication principle: If two identical, deterministic processes begin in the same state and get the same inputs in the same order, they will produce the same output and end in the same state.
- 13. The Unified Log Simple idea; All of the organizations data, available in one unified log that is; • Unified: the one source of truth • Append only: Data items are immutable • Ordered: addressable by offset unique per partition • Fast and scalable: Able to handle 1000´s msg/sec • Distributed, robust and fault tolerant.
- 14. Kafka • “Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.” But… :-) http://www.confluent.io/kafka-summit-2016-101-ways-to-configure-kafka-badly
- 15. Kafka • “Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.” But… :-) http://www.confluent.io/kafka-summit-2016-101-ways-to-configure-kafka-badly
- 16. © 2016 Apache Software Foundation
- 17. © 2016 Apache Software Foundation
- 18. A streaming data product’s anatomy 18 Pub / sub Unified log Ingress Stream processing Egress DB Service TopicJob Pipeline Service Export Visualisation DB
- 19. Architectural patterns The Unified Log and Lambda, Kappa architectures
- 20. Lambda Architecture Example: Recommendation Engine @ Schibsted
- 26. Kappa Architecture A software architecture pattern… where the canonical data store is an append-only immutable log. From the log, data is streamed through a computational system and fed into auxiliary stores for serving. A Lambda architecture system with batch processing removed. https://www.oreilly.com/ideas/questioning-the-lambda-architecture
- 30. Lambda vs Kappa ● Lambda ○ Leverages existing batch infrastructure ○ Cognitive overhead maintaining two approaches in parallel ● Kappa ○ Is real-time processing is inherently approximate, less powerful, and more lossy than batch processing. True? ○ Simpler model https://www.oreilly.com/ideas/questioning-the-lambda-architecture
- 31. Cold Storage • Source of truth for replay in case of failure • Available for ad-hoc batch querying (Spark) • Wishlist; Fast writes, Reliable, Cheap • Cloud storage - s3 (with Glacier) • Traditional - HDFS, SAN • Consider what read performance do you need for a) error recovery b) bootstrapping new deployments
- 33. Event collection 33 Cluster storage HDFS (NFS, S3, Google CS, C*) Service Reliable, simple, write available Kafka Event Bus with history (Secor, Camus) Immediate handoff to append-only replicated log. Once in the log, events eventually arrive in storage. Unified log Immutable events, append-only, source of truth
- 34. Event collection - guarantees 34 Unified log Service (unimportant) Events are safe from here Replicated Topics Non-critical data: Asynchronous fire-and-forget handoff Critical data: Synchronous, replicated, with acknowledgement Service (important)
- 35. Event collection • Source Types • Firehose api´s • Mobile apps and websites • Internet of things / embedded sensors • Event sourcing from existing systems
- 36. Event collection • Considerations • Can you control the data flow, ask sender to wait? • Can clients be expected to have their logic updated? • Can you afford to loose some data, make tradeoffs and still solve your task?
- 38. Pipeline graph • Parallelised jobs read and write to Kafka topics • Jobs are completely decoupled • Downstream jobs do not impact upstream • Usually an acyclic graph CC BY © https://500px.com/thakurdalipsingh Illustration from Apache Samza Doc - Concepts
- 39. Stream processing components ● Building blocks ○ Aggregate ■ Calculate time windows ■ Aggregate state (database/in memory) ○ Filter ■ Slim down stream ■ Privacy, Security concerns ○ Join ■ Enrich by joining with datasets (geoip) ○ Transform ■ Bring data into same “shape”, schema
- 40. Stream Processing Platforms • Spark Streaming • Ideal if you are already using Spark, same model • Bridges gap between data science / data engineers, batch and stream • Kafka Stream • Library - New, positions itself as a lightweight alternative • Tightly coupled to on Kafka • Others ○ Storm, Flink, Samza, Google Dataflow, AWS Lambda http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/
- 41. Schemas
- 42. Schemas • You always have a schema • Schema on write • Requires upfront schema design before data can be received • Synchronised deployment of whole pipeline • Schema on read • Allows data to be captured -as is-. • Suitable for “Data lake” • Often requires cleaning and transform bring datasets into consistency downstream.
- 43. Schema on read or write? 43 DB DB DB Service Service Export Business intelligenceChange agility important here Production stability important here
- 44. Schemas • Options In streaming applications • Schema bundled with every record • Schema registry + id in record • Schema formats • JSON Schema • AVRO
- 45. Evolving Schemas • Declare schema version even if no guarantee. Captures intention of source. • Be prepared for bad and non-validating data. • Decide on strategy for bringing schema versions in alignment. • Maintain upgrade path through transforms. • What are the needs of the consumer. • Data exploration vs Stable services.
- 46. Results
- 47. Serving Results ● As streams ○ Internal Consumer ○ External Consumer bridges ■ ex. REST post to external ingest endpoint ● As Views ○ Serving index, NoSQL ○ SQL / cubes for BI
- 48. Reactive Streams • [...] an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. [...] aimed at runtime environments (JVM and JavaScript) as well as network protocols. • The scope [...] is to find a minimal set of interfaces, methods and protocols that will describe the necessary operations and entities to achieve this goal. • “Glue” between libraries. Reactive Kafka -> Akka Stream -> RxJava
- 50. Thank you!
- 51. Schemas • You always have a schema • Even if you are “Schemaless” • Build tooling and workflows for handling schema changes