Dagster @ R&S MNT
- 1. Dagster @ Rohde & Schwarz MNT Community Meeting May, 2021
- 2. Introduction Simon, Data Engineer & Author Working at Rohde & Schwarz, Writing at sspaeti.com. Early user of Dagster Rohde & Schwarz, Company SmartAnalytics, Product sspaeti.com, Blog Specialized in electronic test equipment, broadcast & media, cybersecurity, radio monitoring and radiolocation, and radio communication. Actionable benchmarking, optimization and monitoring intelligence from drive test data in mobile network testing (MNT) Genuine news about the data ecosystem. Topics: #dataengineering #bigdata #python #opensource #ETL
- 3. What Do We Do? Our tools help to improve the quality and performance of mobile networks Article Hello Africa! R&S®Freerider 4 Backpack QualiPoc Android SmartAnalytics Source: Iberdrola.com
- 4. Architecture - Where We Come From SmartAnalytics Custom ETL (C# and SQL)
- 5. Motivation for using Dagster Bringing the ETL into the cloud and manage at a central place. Being #bigdata ready. ● on-prem → cloud ● scale-up → scale-out ● and generally overcoming limits in ETL processing and query time
- 6. Architecture - Cloud-Native with Dagster
- 7. Event-Driven with Sensors → Run-History of Sensors
- 8. Event-Driven with Sensors → Listening on S3-Folder
- 9. Import-Pipeline File-Upload ⇒ ETL ⇒ Delta ⇒ Druid
- 10. Assets - Link Data to Computations ● ETL file size, duration & time overview ● Assets for persistent Delta & Druid tables to see what pipelines affected changes
- 11. Example of adding Assets → Simply yield the Metadata-Entries
- 12. Advantages in using Dagster Replaced Custom Built Engine We could replace our own created engine. Implications: ● Stable and tested ● Massive out-of-the-box features ○ Re-start capabilities, backfill, dependency management, statemangement of running jobs, support different modes, easy testable Feature rich UI - Dagit Beautiful UI with supports the user and engineers to get a fast overview and do operations. Implications: ● Everyone can sees what’s going on in the system: ○ Current jobs ○ State in the past ○ Rich Metadata
- 13. Advantages in using Dagster Problem solving Problems and errors are straightforward to spot, even given the complex big data architecture. Implications: ● Error fixing during development are fast and easy ● Error reporting are coming with good amount of context Easy to learn Dagster User which haven’t used Dagster, can get started fast. Concepts behind make sense to new users. Implications: ● Developers up to speed fast ● It’s pleasant to write pipelines
- 14. Advantages in using Dagster Self-Documented Pipelines are documented directly within Dagit. Each step is explained by the solids and rich metadata e.g. adding SQL-Stmt or Assets. Implications: ● Users and customers can easily understand what’s going on ● Easy to model pipelines Reusable code Existing Microservices in Python could be easily transferred with minimum effort. With `resources` and `solids` we can re-use all our code in an easy way. Implications: ● Easy to consolidate code into Dagster ● No code duplication (DRY-principle) ● Stable and tested functions ● Reduce of boiler-plate compared implement multiple microservices ● Functional by design
- 15. Example of Re-usable Code with Resources Define once And use everywhere with context
- 16. Advantages in using Dagster Kubernetes deployment Easy way to schedule pods from our pipelines. Implications: ● Based on dockerfiles which allows us to run SQL-Server pods and at the same time pod with Spark configured Python based (& SQL supportive) Python is the language of data and easy to understand for analysts and engineers. With prepared easy to inject SQL-statements. Implications: ● Easier for non Engineers to adapt ● Possible to use wide range of Python packages, especially for ML
- 17. Next Steps Testing ● Add Unit and Smoke Tests to improve stability Documentation ● Use Assets more intensively / automated(?) ● Integrate with new data lineage feature Guidelines ● Extend our Dagster guidelines and best practices to align on common patterns Pipelines ● Try dynamic orchestration for overall pipeline ● Add partitions by file_name
- 18. Questions? Thanks for listening! Feel free to reach out to me on Dagster-Slack or anywhere else. SmartAnalytics Mobile Network Testing - MNT sspaeti.com Simon Späti @sspaeti