Near real-time anomaly detection at Lyft
- 1. Near real-time anomaly detection at Lyft Mark Grover | @mark_grover Thomas Weise | @thweise go.lyft.com/streaming-at-lyft
- 2. Agenda ● Data at Lyft ● 3 problems in streaming ● Conclusion
- 3. Data at Lyft
- 4. Lyft: Fastest ride sharing company in the US
- 5. Data platform users 5 Data Modelers Analysts Data Scientists General Managers Data Platform Engineers ExperimentersProduct Managers Analytics Biz ops Building apps Experimentation
- 6. 6 Data Platform architecture Custom apps Services (e.g. ETA, Pricing) Operational Data stores (e.g. Dynamo) Models + Applications (e.g. ETA, Pricing) Flyte
- 7. Data platform users 7 Data Modelers Analysts Data Scientists General Managers Data Platform Engineers ExperimentersProduct Managers Analytics Biz ops Building apps Experimentation
- 8. 8 Data Platform architecture Custom apps Services (e.g. ETA, Pricing) Operational Data stores (e.g. Dynamo) Models + Applications (e.g. ETA, Pricing) Flyte
- 9. How can streaming help build better applications?
- 10. 1. Engineer Responsibility Build great products Alerting Business metrics Requirements Anomaly detection on business metrics
- 11. Anomaly Detection use cases Security Ops Payment fraud Customer service Accident detection
- 12. 2. Data Scientist Responsibility Extract knowledge and insights from data (To build better products) Prototype in a language of choice (Python, R, SQL) Quick and simple ways of “cleaning” data Requirements Prototype in a language of choice (Python, R, SQL) Quick and simple ways of cleaning data
- 13. Data Science use cases - Driver app
- 14. Data Science use cases - Pricing
- 15. Historical architecture State Store Model 1 Model 2 Model 3 t=60s t=60s t=60s t=63s t=65s t=66s
- 16. New architecture - Flink State Store Model 1 Model 2 Model 3 t=60s t=63s t=68s t=63s t=68s t=74s
- 17. Today’s focus on 3 streaming use cases 1 Anomaly Detection 2 3 Making Data Prep Easy Support non-JVM Languages
- 19. What is the problem? Security Ops Payment fraud Customer service Accident detection Business metrics alerting
- 20. 20 Anomaly detection architecture Services (e.g. ETA, Pricing) Operational Data stores (e.g. Dynamo)
- 21. Impact Business metric alerting Financial line items alerting
- 22. Challenges ● Barrier to entry is pretty high ○ Takes a long time to ingest and tune alerts
- 23. 2. Making Data Prep Easy
- 24. What is the problem? ● Data preparation - everyone needs it, examples: ○ Write raw data from stream to S3 for batch consumers ○ Filter, aggregate, … the usual ETL stuff ● Enable teams to focus on business problems, don’t worry about “getting data in” ● Data ingress still is surprisingly difficult ○ Really? ○ Give our users a service that shields them from infrastructure complexity
- 25. Dryft fully managed data processing engine, powering real-time features and events ● Need - Consistent Feature Generation ○ The value of your machine learning results is only as good as the data ○ Subtle changes to how a feature value is generated can significantly impact results ● Solution - Unify feature generation ○ Batch processing for bulk creation of features for training ML models ○ Stream processing for real-time creation of features for scoring ML models ● How - Flink SQL ○ Use Flink as the processing engine using streaming or bulk data ○ Add automation to make it super simple to launch and maintain feature generation programs at scale https://www.slideshare.net/SeattleApacheFlinkMeetup/streaminglyft-greg-fee-seattle-apache-flink-meetup-104398613/#11
- 26. Dryft Program Configuration file decl_ride_completed.sql { "source": "dryft", "query_file": "decl_ride_completed.sql", "kinesis": { "stream": "declridecompleted" }, "features": { "n_total_rides": { "description": "All time ride count per user", "type": "int", "version": 1 } } } SELECT COALESCE(user_lyft_id, passenger_lyft_id, passenger_id, -1) AS user_id, COUNT(ride_id) as n_total_rides FROM event_ride_completed GROUP BY COALESCE(user_lyft_id, passenger_lyft_id, passenger_id, -1)
- 27. Dryft Program Execution ● Backfill - read historic data from S3, process, sink to S3 ● Real-time - read stream data from Kinesis/Kafka, process, sink to DynamoDB SinkS3 Source SQL SinkKinesis/Kafka Source SQL
- 28. Bootstrapping ● Read historic data from S3 ● Transition to reading real-time data ● https://data-artisans.com/flink-forward/resources/bootstrappin g-state-in-apache-flink S3 Source Kinesis/Kafka Source Business Logic Sink < Target Time >= Target Time
- 29. When to Dryft • Feature Generation as original driver • Declarative Streaming ETL ‒ Stream to Table / Stream • SQL - Simplicity <> Power tradeoff ‒ Flink SQL supports UDFs (written in Java) ‒ A UDF could also do a service call, but..
- 30. When we need Programming https://ci.apache.org/projects/flink/flink-docs-stable/concepts/programming-model.html
- 31. Flink Streaming Options • SQL - Dryft • Java DataStream API - the usual starting point ‒ Sources, Sinks, Windowing, Implicit State Management ‒ Fluent style, high abstraction level • ProcessFunction for advanced logic ‒ User code controlled state and timers • Nice fit when Java is already established ‒ Forced language switch is hard sell, time to value long and less predictable ‒ Initial Flink Deployments at Lyft ‒ But we do a lot of stuff in Python..
- 33. What is the problem? ● Flink API primarily target Java developers ○ Most of our teams that want to solve streaming use cases don’t work with Java ● Enable streaming native to the language ecosystem ○ Python is the primary option for ML ○ (Use cases not addressed by Dryft/Flink SQL)
- 34. Streaming Options for Python • Jython != Python ‒ Flink Python API and few more • Jep (Java Embedded Python) • KCL workers, Kafka consumers as standalone services • Spark PySpark ‒ Not so much streaming, different semantics ‒ Different deployment story • Faust ‒ Kafka Streams inspired ‒ No out of the box deployment story
- 35. Apache Beam 1. End users: who want to write pipelines in a language that’s familiar. 2. SDK writers: who want to make Beam concepts available in new languages. 3. Runner writers: who have a distributed processing environment and want to support Beam pipelines Beam Model: Fn Runners Apache Flink Apache Spark Beam Model: Pipeline Construction Other LanguagesBeam Java Beam Python Execution Execution Cloud Dataflow Execution https://s.apache.org/apache-beam-project-overview
- 36. Beam Python Example def pipeline(root): input = root | ReadFromText("/path/to/text*") | Map(lambda line: ...) scores = (input | WindowInto(FixedWindows(120) trigger=AfterWatermark( early=AfterProcessingTime(60), late=AfterCount(1)) accumulation_mode=ACCUMULATING) | CombinePerKey(sum)) scores | WriteToText("/path/to/outputs") MyRunner().run(pipeline) ( What, Where, When, How )
- 37. Python on Flink via Beam • Beam model and Flink go well along ‒ Flink Runner most advanced OSS option for Beam Java SDK • Python SDK already available on Dataflow • Beam Language Portability allows Python (and Go) SDK to work with JVM-based runners ‒ Flink Runner is first to support portability • Flink Deployment Story ‒ Extend to run Python via Beam on Flink
- 38. Python on Flink via Beam Job Service Artifact Staging Job Manager Fn Services SDK Harness / UDFs Provision Control Data Artifact Retrieval State Logging Cluster Runner Dependencies (optional) python -m apache_beam.examples.wordcount --input=/etc/profile --output=/tmp/py-wordcount-direct --experiments=beam_fn_api --runner=PortableRunner --sdk_location=container --job_endpoint=localhost:8099 --streaming https://s.apache.org/streaming-python-beam-flink
- 39. 3.5 But, how do we deploy all this?
- 40. Deployment 40 Streaming Application (Dryft, Java, Beam, ...) Stream / Schema Registry Deployment Tooling Metrics & Dashboards Alerts Logging Amazon EC2 Amazon S3 Wavefront Salt (Config / Orca) Docker Source Sink
- 41. Future of Deployment • Flink embraces containerization ‒ Reactive vs. Active Flink Container Mode (resources supplied externally vs. actively requested) • Kubernetes Operator ‒ Resource Elasticity ‒ Improved Resource Utilization ‒ Auto-Scaling Support ‒ Automate (stateful) upgrade
- 42. Learnings • Integration ‒ Things work well in isolation, but.. ‒ Flink Kinesis Consumer ‣ Connectors that work reliably at scale are easy hard • Things we find at scale ‒ Intermittent AWS service errors (Kinesis, S3) ‣ Retry vs. topology reset ‒ S3 hotspotting with Flink checkpointing for large jobs (FLINK-9061) ‒ Naive pubsub consumption can lead to massive state buffering ‣ Align watermarks across source partitions
- 43. Conclusion
- 44. ● Data at Lyft ● 3 problems in streaming ○ Anomaly Detection - Anodot ○ Easy data prep - Dryft ○ Non-JVM language support - Apache Beam Conclusion
- 45. We are hiring! lyft.com/careers https://goo.gl/RsyLkS go.lyft.com/streaming-at-lyft Images from the Noun Project Mark Grover | @mark_grover Thomas Weise | @thweise