How Lyft Drives Data Discovery
2 likes299 views
The document discusses Lyft's data discovery tool called Amundsen. It provides an overview of Amundsen's architecture including its use of a graph database and Elasticsearch for metadata storage and search. It describes the challenges of data discovery that Amundsen addresses like time spent searching for data. The document outlines Amundsen's key components like its databuilder, metadata and search services. It discusses Amundsen's impact and popularity at Lyft and its open source community. Future roadmap plans include additional metadata types and deeper integrations with other tools.
![Search Results - Striking the balance
20
Relevance Popularity
● Names, Descriptions, Tags, [owners, frequent
users]
● Different weights for different metadata, e.g.
resource name
● Querying activity
● Dashboarding
● Lower weight for automated querying
● Higher weight for adhoc querying](https://image.slidesharecdn.com/zg2sqpyxqoksfaioxoyb-signature-83e341d1c550243d04fe7f4a982e6a5c434a2b92126328f7df65e0a62c4f55b7-poli-190920161718/85/How-Lyft-Drives-Data-Discovery-20-320.jpg)
How Lyft Drives Data Discovery
- 1. Wednesday, September 18th 2019 Tamika Tannis | Software Engineer, Lyft go.lyft.com/datadiscoveryslides Disrupting Data Discovery
- 2. Agenda • Data Ecosystem at Lyft • Challenges with Data Discovery • Data Discovery at Lyft • Amundsen’s Architecture • What’s Next? 2
- 3. Data Ecosystem at Lyft 3
- 4. 4 Core Data Infrastructure (High Level) Custom Applications Architecture Applications Mobile App Services Services Data Streaming Frameworks (Kafka / Kinesis) Flink
- 6. Data is used to make informed decisions 6 Analysts Data Scientists General Managers Engineers ExperimentersProduct Managers Data-driven decision making process: 1. Search & find data 2. Understand the data 3. Perform an analysis/visualisation 4. Share insights and/or make a decision Make data the heart of every decision
- 7. • Goal: What new data-driven policies can we enact to reduce driver insurance fraud? • Idea: Let’s take a deeper look into insurance claims from drivers who have given less than 𝑥 rides. • Next Step: I’ll first get all drivers who have given less than 𝑥 rides...but where do I look? Hi! I’m a new Analyst in the Fraud Department ! 7
- 8. • Ask a friend/manager/coworker • Ask in a wider Slack channel • Search in the Github repos Step 1: Search & find data 8 We end up finding tables: driver_rides & rides_driver_total
- 9. • What is the difference: driver_rides vs. rides_driver_total • What do the different fields mean? ‒ Is driver_rides.completed different from rides_driver_total.lifetime_completed? ‒ What period of time does the data in each table cover? • Dig deeper: explore using SQL queries Step 2: Understand the data 9 SELECT * FROM schema.driver_rides WHERE ds=’2019-05-15’ LIMIT 100; SELECT * FROM schema.rides_driver_total WHERE ds=’2019-05-15’ LIMIT 100;
- 10. Data Scientists spend upto 1/3rd time in Data Discovery 10 Data Discovery • Data discovery is a problem because of the lack of understanding of what data exists, where, who owns it, & how to use it. • It is not what our data scientist should focus on: they should focus on Analysis work Data-based decision making process: 1. Search & find data 2. Understand the data 3. Perform an analysis/visualisation 4. Share insights and/or take a decision
- 12. User Personas - (1/2) 12 Analysts Data Scientists General Managers ExperimentersEngineersProduct Managers • Frequent use of data • Deep to very deep analysis • Exposure to new datasets • Creating insights & developing models
- 13. User Personas - (2/2) 13 Power User - Has been at Lyft for a long time - Knows the data environment well: where to find data, what it means, how to use it Pain points: - Needs to spend a fair amount of their time sharing their knowledge with the new user - Could become “New user” if they switch teams New User - Recently joined Lyft or switched to a new team - Needs to ramp up on a lot of things, wants to start having impact soon Pain points: - Doesn’t know where to start. Spends their time asking questions and cmd+F on github - Makes mistakes by mis-using some datasets
- 14. 3 complementary ways to do Data Discovery 14 Search based I am looking for a table with data on “cancel rates” - Where is the table? - What does it contain? - Has the analysis I want to perform already been done? Lineage based If this event is down, what datasets are going to be impacted? - Upstream/downstream lineage - Incidents, SLA misses, Data quality Network based I want to check what tables my manager uses - Ownership information - Bookmarking - Usage through query logs
- 15. Data Discovery at Lyft 15 Product named after Roald Amundsen ● First expedition to reach the South Pole ● First to explore both North & South Poles
- 16. Landing Page - Optimized for search
- 17. Search Results - Ranked on relevance & popularity
- 18. Relevance - search for “apple” on Google 18 Low relevance High relevance
- 19. Popularity - search for “apple” on Google 19 Low popularity High popularity
- 20. Search Results - Striking the balance 20 Relevance Popularity ● Names, Descriptions, Tags, [owners, frequent users] ● Different weights for different metadata, e.g. resource name ● Querying activity ● Dashboarding ● Lower weight for automated querying ● Higher weight for adhoc querying
- 22. Data Preview 22
- 24. Computed Column Metadata Statistics Disclaimer: these stats are arbitrary.
- 28. 28 Postgres Hive Redshift ... Presto Github Source File Databuilder Crawler Neo4j Elastic Search Metadata Service Search Service Frontend ServiceML Feature Service Security Service Other Microservices Metadata Sources
- 30. 30 Postgres Hive Redshift ... Presto Github Source File Databuilder Crawler Neo4j Elastic Search Metadata Service Search Service Frontend ServiceML Feature Service Security Service Other Microservices Metadata Sources
- 32. Why choose a graph database? 32
- 33. 33 Why Graph database? (1/2)
- 34. 34 Why Graph database? (2/2)
- 35. 35 2. Metadata Service • A thin proxy layer to interact with graph database ‒ Currently Neo4j is the default option for graph backend engine ‒ Work with the community to support Apache Atlas • Support Rest API for other services pushing / pulling metadata directly
- 36. Neo4j is the source of truth for editable metadata 36
- 37. Why not propagate the editabled metadata back to source 37
- 38. Why not propagate the editabled metadata back to source 38
- 39. Why not propagate the editabled metadata back to source 39
- 40. Why not propagate the editabled metadata back to source 40
- 42. 42 Postgres Hive Redshift ... Presto Github Source File Databuilder Crawler Neo4j Elastic Search Metadata Service Search Service Frontend ServiceML Feature Service Other Services Other Microservices Metadata Sources
- 43. 43 Metadata Sources @ Lyft
- 44. Metadata - Challenges • No Standardization: No single data model that fits for all data resources ‒ A data resource could be a table, an Airflow DAG or a dashboard • Different Extraction: Each data set metadata is stored and fetched differently ‒ Hive Table: Stored in Hive metastore ‒ RDBMS(postgres etc): Fetched through DBAPI interface ‒ Github source code: Fetched through git hook ‒ Mode dashboard: Fetched through Mode API ‒ … 44
- 45. Databuilder 45
- 47. How is the databuilder orchestrated? 47 Amundsen uses Apache Airflow to orchestrate Databuilder jobs
- 49. 49 Postgres Hive Redshift ... Presto Github Source File Databuilder Crawler Neo4j Elastic Search Metadata Service Search Service Frontend ServiceML Feature Service Security Service Other Microservices Metadata Sources
- 50. 3. Search Service • A thin proxy layer to interact with the search backend ‒ Currently it supports Elasticsearch as the search backend. • Support different search patterns ‒ Normal Search: match records based on relevancy ‒ Category Search: match records first based on data type, then relevancy ‒ Wildcard Search 50
- 51. How to make the search result more relevant? 51 • Experiment with different weights, e.g boost the exact table ranking • Collect metrics ‒ Instrumentation for search behavior ‒ Measure click-through-rate (CTR) over top 5 results • Advanced search: ‒ Support wildcard search (e.g. event_*) ‒ Support category search (e.g. column: is_line_ride) ‒ Future: Filtering, Autosuggest
- 53. 53 Postgres Hive Redshift ... Presto Github Source File Databuilder Crawler Neo4j Elastic Search Metadata Service Search Service Frontend ServiceML Feature Service Security Service Other Microservices Metadata Sources
- 54. Web Application
- 55. Web Technologies 55 Develop Build Test
- 56. What’s Next? 56
- 57. Amundsen’s Impact • Tremendous success at Lyft ‒ Used by Data Scientists, Engineers, PMs, Ops, even Cust. Service! ‒ 90% penetration among Data Scientists ‒ +30% productivity for the Data science org. 57
- 58. Amundsen is Open Source! • github.com/lyft/amundsen • Growing and active community ‒ c.150 github stars, 10+ companies contributing back ‒ Slack w/ 30+ companies and c.100 people ‒ Presented at conferences in San Francisco, Barcelona, Vilnius, Moscow by Lyft employees and community ‒ Featured in blog posts and interviews • Net positive impact for Lyft through external community contributing ‒ Integration with open source backend ‒ Integration with new data sources (BigQuery, Redshift, Postgres), lifting them from our roadmap 58
- 60. Roadmap PeopleDashboards Data sets Phase 1 (Complete) Phase 2 (In Progress) Phase 3 (In Scoping) Streams Schemas Workflows More Metadata Deeper integration with other tools (e.g. Mode) Privacy Governance
- 63. Roadmap PeopleDashboards Data sets Phase 1 (Complete) Phase 2 (In Progress) Phase 3 (In Scoping) Streams Schemas Workflows More Metadata Deeper integration with other tools (e.g. Mode) Privacy Governance
- 64. Roadmap 64
- 65. Roadmap 65
- 66. Tamika Tannis | @ttannis | /in/tamika-tannis Project Code @ github.com/lyft/amundsen Blog Post @ go.lyft.com/datadiscoveryblog Icons under Creative Commons License from https://thenounproject.com/ 66
Editor's Notes
- #2: Name & Role working on an open-source data discovery tool at Lyft. It’s called “Amundsen” -- more on that name later. It leverages Neo4j, glad to share how we’ve been using Neo4j at Lyft to achieve goals of our product Amundsen.
- #3: On the agenda for this talk
- #5: The data infrastructure at Lyft can be visualized by this diagram Events are fires into streaming frameworks (Apache Kafka / Amazon Kinesis) Apache Flink injests that data into Amazon S3, first layer of storage, persistent storage Data stored in Amazon S3 is further transformed and stored in various other datastores Initially started with Redshift. Introduced new datastores with different strengths that better serve specific purposes, Hive for long running queries/ETLs Presto for quick analysis & as-needed queries Druid for faster interactive queries The takeaway from this slide: Lots of data (~10PB), lots of places it can be (thousands of tables), and lots of tools/people trying to use the data on a regular basis.
- #6: Now onto challenges with data discovery
- #7: Effective data discovery is important because data is at the heart of every decision we make. It is the only way to make informed, objective decisions. Applies to many roles Data-driven decision making process Search & find data Understand the data Perform an analysis Share insights or make a decision
- #8: To highlight some data discover pain points that occur without the proper tools, let’s walk through a hypothetical example
- #9: Your experience searching and finding data may involve doing all of the following 3 things.
- #10: Your experience understanding the data doesn’t get any easier.
- #11: ⅓ of time on data discovery Difficult to find what exists, understand whether or not it’s what you are looking for, or trust that it is the source of truth for that information We can significantly increase productivity and impact if we can reduce this time...
- #12: Let’s start to think of what a helpful tool would look like. Complication: What audience to serve? Who are they and what do they need?
- #13: What audience to serve?
- #14: Second level of personas to consider.
- #15: Lastly, what by what means do they want to perform discovery? There are 3 complementary ways to do Data Discovery Search based: most common and top priority Lineage based: callback to the data ecosystem, if there is a hiccup in that system, what does it impact? Datasets must be trustworthy Network based: helpful on the job to know what others are using for what purpose
- #16: We’ve talked about some pain points of data discovery and why it’s important, let’s talk about our solution -- Amundsen.
- #17: Disclaimer Representative data Amundsen circa March 2019 Our landing page is optimized for search Most common method of data discovery, presented with search bar & help text for some advanced search features We also want the landing page to be able to help users that don’t know what to search for. Created this concept of popular tables
- #18: Users presented with ranked search results Not like page-rank but based on relevance and popularity
- #19: This is what we mean when we say relevance
- #20: This is what we mean when we say popularity
- #21: Striking the balance between the two is an interesting challenge Relevance is based on metadata Popularity is not click through rate but through query access patterns.
- #28: Now that I’ve demonstrated what Amundsen is and how it can be used, let’s talk about how it was built.
- #29: Microservice architecture, services are divided by domains: ui/frontend, search, metadata Walk from top to bottom & highlight “pluggability”
- #30: I’ll now dive deeper into each of the Amundsen services presented in the previous graph, starting with metadata service, which is backed by Neo4j.
- #32: As you may remember from the application walkthrough, Amundsen surfaces resource metadata and that is what we are storing in Neo4j
- #33: However graph databases are not common for many web applications, and so one might ask why choose a graph database.
- #34: Well if you remember the diagram of the data ecosystem at Lyft from the beginning of the talk, that can be modeled as a graph. This is a very powerful feature because the alternative to created these kinds of relationships with a RDBMS is joins A NoSQL database isn’t set up for this
- #35: Let’s take a note of some of the features from the table detail page again and see how this is represented in Neo4j Walk through features What’s very beneficial about this is that when we have a new use case and a new piece of metadata to represent, we just have to create the new node and relationship.
- #36: It’s worth noting that one key architectural decision made for this service and others is that it is a proxy to interact with Neo4j Which means its can also interact with anything else that can store this data. This choice is key for us as an open source project
- #37: Another key characteristic of our system is that neo4j is the source of truth for our editable metadata
- #38: This was actually not our original intent, we ran into a roadblock when we were first implementing the description editing feature. We originally had a setup like this
- #39: Then we realized we forgot to account for something. Tables can get rebuilt using the source code that generated the table and descriptions will be overwritten
- #40: The we thought about whether or not we could do this, update them both!
- #41: The answer was no. ...And that’s how Neo4j became the source of truth for editable metadata
- #42: Now onto the data builder service
- #43: It is the layer that ingests metadata from the sources. Which sources exactly?
- #44: Many sources. Not just tables but dashboards, different kinds of resources This creates some complexity
- #46: This is what databuilder helps to address, it is a data ingestion framework similar to Apache Gobblin It functions as an ETL engine Each part is modularized, and can be reused (e.g same transformer) or swapped out A publisher leverages a transaction to make the data ingestion atomic -- it is not the case that there is partially updated data
- #47: Here is a more solid example
- #48: How is this all orchestrated? With Airflow dags, these are jobs that run to execute each piece of the puzzle End with elastic search.
- #49: Elasticsearch is what sit behind our search service...
- #50: ... in the same way that Neo4j stands behind the metadata service. Similar data is loaded into both there are some minor differences, for example data that won’t be searchable like col stats
- #51: Also similar to the metadata service the search service acts as proxy
- #52: What I find most interesting about the search service is actually the biggest problem that we struggle with, “how to make the search results more relevant”.
- #53: Lastly we have our Amundsen’s frontend service.
- #56: Obligatory slide
- #60: T