Empowering Zillow’s Developers with Self-Service ETL
1 like250 views
The document discusses Zillow's efforts to empower its developers with self-service extract, transform, load (ETL) tools. It describes two key components: Zetlas, a user-friendly tool that allows non-coding users to automate SQL workflows through a graphical interface; and Zagger, a developer-focused service that automates data engineering tasks and integrates with common ETL tools. The tools were developed to meet different user needs, lower the barrier to data work, and provide modular platforms that can be expanded over time. Zillow aims to continue growing adoption of these self-service ETL tools and unifying their backends.
Empowering Zillow’s Developers with Self-Service ETL
- 1. 1 1 Derek Gorthy Senior Software Development Engineer Yuan Feng Software Development Engineer Empowering Zillow’s Developers with Self-Service ETL
- 2. 2 Who We Are Zillow Offers Data Engineering Team @ Zillow Derek Gorthy Senior Software Development Engineer, Big Data Yuan Feng Software Development Engineer, Big Data
- 3. 3 Agenda ● How We Think About Self-Service ETL ● Core Components ● Self-Service ETL in Action at Zillow ○ Zetlas ○ Zagger ● Next Steps and Takeaways
- 4. Zillow
- 5. About Zillow ● Reimagining real estate to make it easier to unlock life’s next chapter ● Offer customers an on-demand experience for selling, buying, renting and financing with transparency and nearly seamless end-to-end service ● Most-visited real estate website in the United States * As of Q4-2020
- 6. How We Think About Self-Service ETL
- 7. Zagger Integrations Zagger Pipeline Utilities Package User Interaction Zagger Managed Service Integrations Execution Zetlas DQ Module API Parser 1 Parser N Airflow Renderer ... ... Kafka Renderer
- 8. What Is Self-Service ETL? User Interaction Pipeline Configuration File ?
- 9. How We Think About Self-Service ETL User Interaction Pipeline Interpret Pipeline Metadata Render Configuration File Opinionated Unopinionated
- 10. Core Components
- 11. User Interaction User Interaction Pipeline Interpret Pipeline Metadata Render Configuration File Opinionated Unopinionated
- 12. Interpret User Input User Interaction Pipeline Interpret Pipeline Metadata Render Configuration File Opinionated Unopinionated
- 13. Pipeline Metadata User Interaction Pipeline Interpret Pipeline Metadata Render Configuration File Opinionated Unopinionated
- 14. Render Pipeline User Interaction Pipeline Interpret Pipeline Metadata Render Configuration File Opinionated Unopinionated
- 15. Data Pipeline & Shared Integrations User Interaction Pipeline Interpret Pipeline Metadata Render Configuration File Opinionated Unopinionated
- 16. Self-Service ETL in Action at Zillow
- 17. Applied Self-Service ETL - Zetlas Motivation Features Target Users ● Modernized and reliable self-service tool to automate SQL based workflows ● No coding experience needed to create ETL workflows ● UI-driven ● Rapid prototyping and deployment ● Job monitoring/alerting ● Automated validation ● Integration with multiple internal services ● Scalable and expandable ● Data scientists ● Data analysts
- 18. Zetlas UX Design
- 19. Applied Self-Service ETL - Zagger Motivation Features Target Users ● Provide a developer-friendly abstraction from ETL tools ● Create a service that automates data engineering ancillary tasks ● Create common processing patterns for fast pipeline development ● Integrates with Terraform ● Exposes create/delete endpoints for other access patterns ● Allows for custom interpreter creation ● Integration with multiple internal services ● Data engineers ● Data producer teams
- 20. Zagger Integrations Zagger Pipeline Utilities Package User Interaction Zagger Managed Service Integrations Execution Zetlas DQ Module API Parser 1 Parser N Airflow Renderer ... ... Kafka Renderer
- 21. Next Steps and Takeaways
- 22. Development Timeline 2019 2020 2021 Pipeler shared Spark processing library development Zetlas official launch in Zillow Zagger Managed Service and Pipeline Utilities Package library User Growth for Zagger and Zetlas ZETL retirement Zetlas and Zagger backend unification
- 23. Takeaways ● UI must be designed to meet the needs of its users ● Self-service ETL isn’t just for non-data engineers ● Modular platform design allows for capabilities to be developed in piecemeal ● Abstraction from tool-specific implementation gives flexibility
- 24. More From Zillow Democratizing Data Quality Through a Centralized Platform 5/27 @ 3:15 PM PST Scaling AutoML-Driven Anomaly Detection With Luminaire 5/27 @ 5:00 PM PST