FlinkForward Asia 2019 - Evolving Keystone to an Open Collaborative Real Time ETL Platform
Download as PPTX, PDF0 likes247 views
The document outlines the evolution of Netflix's real-time data infrastructure, focusing on the development of a collaborative ETL platform called Keystone. It emphasizes the importance of improving content production efficiency and enhancing customer experience through data-driven insights and streamlined processes. Key personnel are introduced, highlighting their roles and motivations in utilizing technology to drive better recommendations, analytics, and operational workflows.
![Stream Processing connector ecosystem in Netflix
● Hive
● Iceberg
● Kafka
● Elasticsearch
● ...more coming
// Example in Java
DataStream<YourOutputDataType> dataStream = ...
// attach Iceberg sink to the DAG
getSinkBuilder()
.toIceberg("<sink_name>", dataStream)
.buildAndAddSink();
// Example in Scala
val srcStream = getSourceBuilder
.fromKafka("example-kafka-source")
.buildScala
.map(r => r.getPayload)
getSinkBuilder.toIcebergScala[util.Map[String,Object]](config,
"iceSink", srcStream)
.buildAndAddSink()](https://image.slidesharecdn.com/ffa2019evolvingkeystonetoanopencollaborativereal-timeetlplatform-final-191129060657/85/FlinkForward-Asia-2019-Evolving-Keystone-to-an-Open-Collaborative-Real-Time-ETL-Platform-26-320.jpg)
FlinkForward Asia 2019 - Evolving Keystone to an Open Collaborative Real Time ETL Platform
- 1. Zhenzhong Xu Real-Time Data Infrastructure @ Netflix Evolving Keystone to an Open Collaborative Real-time ETL Platform
- 2. Evolving Keystone to an Open Collaborative Real-time ETL Platform Zhenzhong Xu Real-Time Data Infrastructure
- 3. Making the world smile Netflix is to bring diverse stories around the world to our global members, to entertain everyone and make the world smile!
- 8. and she is not alone ...
- 9. Content Production 101 Bring joy to our members: ● Best in class experience Give the customer the freedom, flexibility and best in class experience to enjoy the entertainment.
- 10. Content Production 101 Bring joy to our members: ● Best in class experience ● Produce more exciting content Build a technology driven Studio to empower storytelling and content production.
- 11. Zhenzhong Xu Publish, Collect, Move & Compute event data in near real time @ Cloud Scale Stream Consumers Router Batch System Fronting Kafka Event Producer Consumer Kafka Control Plane Self Service UI Mantis
- 12. Keystone is ... … a single self-contained PaaS Event Processing Pipeline
- 13. Keystone is ... … a multi-tenants, self-serving tool
- 14. Keystone is ... Stream Processing Service Transport / Messaging Service Producer API Control Plane Consumer API Self Service UI … powered by a collection of building blocks
- 15. Putting together ... Stream Consumers Router Batch System Fronting Kafka Event Producer Consumer Kafka Control Plane Self Service UI mantis
- 16. Elliot is a data scientist works in the Data Science Engineering organization. His main motivation is to bring the magic out of customer data and improve customer experience. Charlie is an application developer who works in Studio organization. His main motivation is leverage technology to improve the content production business. Introducing our super heros ...
- 17. Elliot’s work will result in better data powered analytic engine, drive better recommendation and personalization features for Emily. Charlie’ work will increase content production workflow efficiency, and ultimately help Eleven enjoy higher quality content.
- 18. Data engineers/scientists prefers simplicity and faster turnaround time to be effective at their work. Lots of common data engineering patterns are not generalized. The A/B test we are doing today takes 28 days to complete, we don’t have a way to detect early issues so we can optimize our experiments! Elliot
- 21. A/B testing
- 23. Real time Reporting Real time Alerting Faster training of ML models Resource Efficiency Why Stream Processing?
- 24. Anatomy of a stream processing job
- 26. Stream Processing connector ecosystem in Netflix ● Hive ● Iceberg ● Kafka ● Elasticsearch ● ...more coming // Example in Java DataStream<YourOutputDataType> dataStream = ... // attach Iceberg sink to the DAG getSinkBuilder() .toIceberg("<sink_name>", dataStream) .buildAndAddSink(); // Example in Scala val srcStream = getSourceBuilder .fromKafka("example-kafka-source") .buildScala .map(r => r.getPayload) getSinkBuilder.toIcebergScala[util.Map[String,Object]](config, "iceSink", srcStream) .buildAndAddSink()
- 30. 3030 public class SpaasApplication extends SpaasBaseApplication { @Override public void constructJobDag(SpaasConfig config, StreamExecutionEnvironment env) { // build a kafka source SingleOutputStreamOperator<Record<Map<String, Object>>> sourceStream = getSourceBuilder().fromKafka("kafkasource").build(); // dynamically pick the selected sink during deployment time SinkFunction<Record<Map<String, Object>>> sink = getSinkBuilder().toSelector("dynamicsink") .declareWith("noopsink", new NoopSink<Record<Map<String, Object>>>()) .or("kafkasink", getSinkBuilder().toKafka("kafkasink").build()) .build(); sourceStream.addSink(sink); } } spaas.myTemplateNameSpace.source.names=dynamicsource # "dynamicsource" is a dynamic selector that currently configured to pick kafka source # the selector.selected configure can be override on SPaaS UI at runtime, and take effect after relaunch of the job. spaas.myTemplateNameSpace.source.dynamicsource.type=selector spaas.myTemplateNameSpace.source.dynamicsource.selector.selected=kafka spaas.myTemplateNameSpace.source.dynamicsource.selector.candidates=kafka,hive Dynamic Source / Sink selector
- 33. 3333 @Override public void constructJobDag(SpaasConfig config, StreamExecutionEnvironment env) { ObjectMapper myObjectMapper = new ObjectMapper(); DeserializationSchema<Person> deserializer = new PersonDeserializer(); TypedSerializationSchema<Person, Record<Map<String, Object>>> serializer= new PersonSerializer(); SingleOutputStreamOperator<Record<Person>> sourceFunction = getSourceBuilder() .fromKafka("kafka") .withOutputType(Person.class) .withDeserializer(deserializer) .build(); SinkFunction<Person> sinkFunction = getSinkBuilder() .toNull("null") .withType(Person.class) .withSerializer(serializer) .build(); sourceFunction.map(Record::getPayload).addSink(sinkFunction); } } Record Abstractions
- 34. … and more
- 35. A B C A Map C Filter A Filter B Filter C Extractor Pattern Join Pattern Enrichment Pattern Data Engineering ETL patterns
- 36. Lots of applications are built to support studio production, but performing consistent data synchronization is hard. Data are spread in different space/time, making data search and discovery hard. I can’t believe it takes a team weeks of time to find out how many scripts are written by female writers! Charlie
- 37. LaunchProductionCreativeForecast Program Deals Post Production Financial Reporting
- 39. Heterogeneous Data Synchronization Operation Reporting Entity Search & Discovery Challenges
- 40. A case study: entity search
- 41. New event driven alternative ...
- 43. Challenge 1: Ordering Semantics
- 45. Me, 4 years old My uncle, 2 years old
- 46. When forcing a global generation order... Can Bob and Dave be logically the same generation?
- 48. The cone shape shows the causal/partial ordering from Dave’s frame of reference.
- 49. The light cone representing the past, present, and future ... https://en.wikipedia.org/wiki/Light_cone
- 50. In a distributed system, it is sometimes impossible to say that one of two events occurred first. The relation “happened before” is therefore only a partial ordering of the events in the system.
- 51. Figure referenced from wikipedia: https://en.wikipedia.org/wiki/Vector_clock
- 53. Figure on the right referenced from DataStax: https://docs.datastax.com/en/cassandra/3.0/cassandra/dml/dmlHowDataWritten.html Multi-Master replication
- 56. Challenge 2: Processing Contracts
- 57. Message Contract
- 58. { // Infra layer (Chaski) "magicByte": 0x01, // hex "version": "0x12", // hex "attributes": { "id": "chaski-id-from-keystone", "app": "example-app", "host": "localhost", "timestamp": 1234, // long }, "payload": { // Platform layer (PlatformRecord) "id": 1, "operation_ts_utc_usec": 1234, // long "extended_attributes": { "netflix/data.platform/data.mesh/cassandra": { // Platform Developer-defined Avro record. "schema_id": 1, "payload": { "encryption_key": "some-rsa-pub" } } }, "operation": "UPDATE", "payload": { // User layer. Represents state of the event after an update operation occurred upstream at the source. "encrypted": false, "format": "AVRO", "schema_id": 1, "payload": { "row_id": 1, "partition_id": 1, "first_name": "net", "last_name": "flix" }, "remote_payload": null }, "secondary_payload": null // User layer. Represents state of the event before the operation occurred upstream at the source. This example is null to show that the source doesn't support pre-images. } } Message Contract
- 59. { // Infra layer (Chaski) "magicByte": 0x01, // hex "version": "0x12", // hex "attributes": { "id": "chaski-id-from-keystone", "app": "example-app", "host": "localhost", "timestamp": 1234, // long }, "payload": { // Platform layer (PlatformRecord) "id": 1, "operation_ts_utc_usec": 1234, // long "extended_attributes": { "netflix/data.platform/data.mesh/cassandra": { // Platform Developer-defined Avro record. "schema_id": 1, "payload": { "encryption_key": "some-rsa-pub" } } }, "operation": "UPDATE", "payload": { // User layer. Represents state of the event after an update operation occurred upstream at the source. "encrypted": false, "format": "AVRO", "schema_id": 1, "payload": { "row_id": 1, "partition_id": 1, "first_name": "net", "last_name": "flix" }, "remote_payload": null }, "secondary_payload": null // User layer. Represents state of the event before the operation occurred upstream at the source. This example is null to show that the source doesn't support pre-images. } } Message Contract
- 60. { // Infra layer (Chaski) "magicByte": 0x01, // hex "version": "0x12", // hex "attributes": { "id": "chaski-id-from-keystone", "app": "example-app", "host": "localhost", "timestamp": 1234, // long }, "payload": { // Platform layer (PlatformRecord) "id": 1, "operation_ts_utc_usec": 1234, // long "extended_attributes": { "netflix/data.platform/data.mesh/cassandra": { // Platform Developer-defined Avro record. "schema_id": 1, "payload": { "encryption_key": "some-rsa-pub" } } }, "operation": "UPDATE", "payload": { // User layer. Represents state of the event after an update operation occurred upstream at the source. "encrypted": false, "format": "AVRO", "schema_id": 1, "payload": { "row_id": 1, "partition_id": 1, "first_name": "net", "last_name": "flix" }, "remote_payload": null }, "secondary_payload": null // User layer. Represents state of the event before the operation occurred upstream at the source. This example is null to show that the source doesn't support pre-images. } } Message Contract
- 61. ● Processor Metadata ● Configurations Management ● Processor capabilities ● Operation responsibilities Processor Contract
- 62. A simple use case: notify upon new deal!
- 64. Bring it all together - rethink ETL ...
- 66. Keystone Routers Flink Platform Don’t waste time here! High Value! Our first stab - an Open, Collaborative, Composable, Configurable ETL Platform...
- 69. Unleash creativity via an open collaborative data platform @ Netflix
- 70. THANKS