Distributed Data Quality - Technical Solutions for Organizational Scaling
- 1. Justin Cunningham justinc@yelp.com Distributed Data Quality Technical Solutions for Organizational Scaling
- 2. Yelp’s Mission Connecting people with great local businesses.
- 3. Lots of Services, Lots of Teams More than 500 engineers Service Service Service Service Service
- 6. Shared Data Provides Context
- 7. Giant ETL Team? “This isn’t useful, can you just do all the data?”
- 8. What we did before?
- 9. MySQL Other Data Stores Yelp-main Services SCHEMATIZER MySQL Services Yelp-main Redshift S3/Parquet Cassandra Elasticsearch Cassandra Hollow KAFKAKAFKA • Paastorm • Python • Flatmap • Flink* • Java/Scala • Advanced Primitives & Stream SQL Recursive Memcached Sources SinksStream Processing Yelp's Streaming Architecture bit.ly/kafka-talk
- 10. MySQL Other Data Stores Yelp-main Services SCHEMATIZER MySQL Services Yelp-main Redshift S3/Parquet Cassandra Elasticsearch Cassandra Hollow KAFKAKAFKA • Paastorm • Python • Flatmap • Flink* • Java/Scala • Advanced Primitives & Stream SQL Recursive Memcached Extract LoadTransform Yelp's Streaming Architecture bit.ly/kafka-talk
- 11. The Rest of this Talk
- 12. Documentation, Discovery and Ownership What does this column mean?
- 13. Schema Registration and Model Extraction
- 14. Schema Registration and Model Extraction
- 15. Schema Registration and Model Extraction
- 16. Which data is available? Which data should I use? {data}
- 17. Where did it come from? Data Lineage
- 18. Where did it come from? Derived Schemas in Registration
- 19. Where did it come from? Consumer/Producer Registration v1
- 20. Where did it come from? Consumer/Producer Registration v2
- 21. Where did it come from? Consumer/Producer Registration v2
- 22. Does all this stuff actually work?
- 23. Auditors Check Output Matches Invariant
- 24. Meta Attributes on the Message Envelope
- 25. Meta Attributes on the Message Envelope
- 26. Is This Data Up-to-date? High Event-time Delay Temporary? Delay Low Event-time Delay No Delay No Delay Low Offset Delay High Offset Delay Both Event-Time Delay and Offset Delay are necessary - need to differentiate no new data v. delay. High event time delay is only an issue when there are unprocessed offsets, and that persists for some time.
- 27. Is This Data Up-to-date?
- 28. Report Consistent Dimensions From Code and Kafka
- 29. MySQL Other Data Stores Yelp-main Services SCHEMATIZER MySQL Services Yelp-main Redshift S3/Parquet Cassandra Elasticsearch Cassandra Hollow KAFKAKAFKA • Paastorm • Python • Flatmap • Flink* • Java/Scala • Advanced Primitives & Stream SQL Recursive Memcached Sources SinksStream Processing How can I get that data? Access Polymorphism
- 30. Users Add Connections from CLI
- 31. Data Lake & the Long-term Workflow S3 Data Lake CLI
- 32. DAG View for Schema ID 1 DAG View is based on the connected component of everything touching the schema. Where We're Headed
- 33. In Summary ● Context Enables Alignment - Reliable Shared Data Builds Context ● What does this column mean? - Documentation and Ownership ● What data is available? What data should I use? - Watson and Curated Data Sets through Tags ● Is this data accurate? ○ Where did it come from? - Data Lineage and Consumer/Producer Registration ○ Did all that stuff work? - Data Auditing ○ Is this data up-to-date? - Event-time and Offset monitoring ● How can I get that data? - Declarative Data Connections