[DSC DACH 24] Cost efficient alternative to databricks lock-in - Georg Heiler
Download as PPTX, PDF0 likes70 views
The document explores cost-efficient alternatives to Databricks, emphasizing the need for flexible data pipelines and cost reduction strategies, achieving up to 43% savings. It highlights challenges such as vendor lock-in, runaway expenses, and the importance of software engineering practices in managing data effectively. The suggested approach advocates for using orchestrators like Dagster to enhance control and observability while ensuring scalability in data engineering.
[DSC DACH 24] Cost efficient alternative to databricks lock-in - Georg Heiler
- 1. COST EFFICIENT ALTERNATIVE TO DATABRICKS Georg Heiler Exploring Alternatives for Cost-Effective and Flexible Data Pipelines bit.ly/efficient-spark
- 2. Data expert Academia & Industry (telco) Specialties data architecture, multimodal and complex data challenges Thought leader Meetup organizer & speaker
- 3. • Rising importance of understanding and shaping supply chains (covid, Ukraine war) • No fine-grained clean data accessible • Abundant un- and semistructured data sophisticated cleaning & parsing required • Extract and classify links based on semantic context
- 4. Results at a glance • 43% Cost Reduction • Software Engineering practices • Future proof flexibility • Single pane of glass for pipelines
- 5. History • Mainframe • Data warehouse • Big Data (Hadoop) • SQL on large data (Hive, Spark) • Cloud DWH (Snowflake, bigquery)
- 7. PaaS Solution Comparison Databricks (DBR) • Easy to use • Can be expensive • Lock-in features (permissions, catalog) • Proprietary Photon engine AWS Elastic Map Reduce (EMR) • Price efficient • Many tuning knobs available (& required) • OSS Spark managed (scaled)
- 8. Challenges • Runaway expenses (usage-based pricing) • Missing software engineering best practices (notebooks) • Developer productivity reduced • Vendor lock-in
- 9. Vision • 0-cost switch • Software engineering practices • Cost & lock-in reduction Orchestrator (dagster) Runtime local Runtime remote DBR Runtime remote EMR
- 10. Spark at a glance
- 11. Dagster introduction X No distributed monolith of CRON strings Asset aware event based orchestration
- 12. Observed challenges • Remote execution • Parameter injection • Logging • Opaque SaaS tools • Single pane of glass • Dependency bootstrap • Missing testability in notebooks • Large-scale compute & orchestrator native development Orchestrator (dagster) Runtime local Runtime remote DBR Runtime remote EMR
- 13. Dagster-pipes
- 15. Dagster-pipes - Sample External code (with metadata) Internal asset shim orchestrating the execution of external script
- 16. Results & Demo
- 18. Partitioned UI
- 19. Implementati on time of DBR is lower
- 20. Implementation complexity of DBR is lower more & more frequent commits for EMR integration
- 21. Median cost of DBR is higher than EMR
- 22. Variability of execution time of DBR is lower
- 23. Implementation lessons • Complexity of AWS EMR: Many low level details about AWS, spot instances, networking required (master on spot instance => 💥💥) • Abstracting the PaaS requires deep understanding of their APIs Tips • maximizeResourceAllocation • LZO • Delta zorder on partition • spark.databricks.delta.vacuum.parallelDelete.enabled=true
- 24. Summary • Money saved – 43% • Bring back software engineering best practices for data • Flexibility • Data PaaS as a commodity • Take back control • Best in breed • Single pane of glass for pipelines
- 25. Takeaway – if you have a small data problem • Pipes allows to quickly bring in existing scripts whilst retaining observability • High code engineering practices scales well • Full control • Compute technology can easily be changed (i.e. duckdb, daft, …) data-engineering.expert/2023/12/11/da gster-dbt-duckdb-as-new-local-mds
- 26. COST EFFICIENCY FOR DATA Georg Heiler bit.ly/efficient-spark (data-engineering.expert/2024/06/21/cost-efficient-alternative-to-databricks-lock-in arxiv.org/abs/2408.11635 github.com/ascii-supply-networks/ascii-hydra/tree/main/src/pipelines/ascii_library_demo )
Editor's Notes
- #1: Mention talk modality interactive ask questions during the talk
- #2: Supply Chain, Text analytics & data architecture & pipelines, graphs, spatial time series
- #4: Physical goods Software 400TiB Commoncrawl, AIS, Satellite, OSINT, …),
- #5: Easy GPU/accelerator prototyping AWS is migrating to daft https://www.getdaft.io/ https://aws.amazon.com/de/blogs/opensource/amazons-exabyte-scale-migration-from-apache-spark-to-ray-on-amazon-ec2/ as we are in control of how the individual steps in the pipeline relate to each other we can relatively easily switch out the compute framework (like AWS did it)
- #9: PaaS solutions offer big benefits. Easy scalability Single centerpiece of the data engineering strategy not just an implementation detail Runaway expenses due to usage-based pricing High CI costs for spinning up resources Prioritizes simplicity over best practices All-notebook environments: Limited code reuse Limited testability Limited VCS integration Developer productivity hampered by VM spin-up times Single central platform dependence
- #10: Pass as implementation detail Containerization, CI/CD, testability flexibility
- #11: Img https://intellipaat.com/blog/tutorial/spark-tutorial/spark-architecture/
- #13: Single pane of glass for operative monitoring of opaque saas tools. Allows us to get standard software best practices like testing, modularity, DRY, ... and maintain these in the SaaS tools. It even allows us to abstract SaaS vendors (Databricks vs. EMR) and substitute one against the other one to save money for large-scale pipelines where we just need compute without wanting to pay for all the enterprisey extra features. 1) boostrap of the remote execution environments 2) centralized logging 3) single/simple start/stop of all pipeline in dagster 3) integration of upstream/downstream pipeline steps in one place PaaS solutions offer big benefits. Easy scalability Single centerpiece of the data engineering strategy not just an implementation detail Runaway expenses due to usage-based pricing High CI costs for spinning up resources Prioritizes simplicity over best practices All-notebook environments: Limited code reuse Limited testability Limited VCS integration Developer productivity hampered by VM spin-up times Single central platform dependence
- #20: The volume of trial runs required to achieve stability on EMR is high. It shows the complex setup and optimization demands of these platforms. Yet, once set up now, it proves hugely beneficial for us.
- #21: EMR was labor-intensive. This is shown by more failed and successful trials. They happened before the product was ready. In fact we required almost twice as many trial runs for EMR as for Databricks. The increase was mainly due to the complexity of setting up EMR systems. They had to handle large datasets well and safely. This setup needs lots of customization and tuning. Databricks provided these features out of the box. EMR demanded more frequent code changes. They were sometimes extensive. This reflected a steeper learning curve and higher complexity. But, it was in exchange for lower costs.
- #24: Teams need to learn the specifics of each platform. This includes API differences, setup quirks, and best practices. This learning curve can delay initial deployment and require additional training.
- #25: For small-ish to medium workloads EMR works fine out of the box => we can save money (cheaper compute) For very large workloads, we use EMR with fine-tuning. It lets us save money with cheaper compute. For special workloads, Photon is very profitable. We use Databricks to save money and time again.
- #26: https://aws.amazon.com/de/blogs/opensource/amazons-exabyte-scale-migration-from-apache-spark-to-ray-on-amazon-ec2/
- #27: Mention talk modality interactive ask questions during the talk