ABC of Distributed Data Processing
1 like1,185 views
The document discusses the principles of distributed data processing, emphasizing the distinction between descriptive, predictive, and prescriptive data analysis. It covers the challenges of data storage, architecture (including star and snowflake schemas), and the complexities involved in data relationships over time. The author, Piyush Verma from Oogway Consulting, provides insights into best practices and considerations for effective data engineering.
ABC of Distributed Data Processing
- 1. ABC of Distributed Data Processing. Achieving Buzzword Compliance. 1 Piyush Verma Oogway Consulting
- 3. When will my Data become Big Data?
- 4. Hive Data Will Save.
- 5. How did we reach here? 5
- 9. When do I call it Big Enough?
- 10. Why bother with Data Engineering? 10
- 11. Why do analysis at all?
- 12. Descriptive - Historical. - Deterministic. - Inferential. - Managers make pretty graphs.
- 13. Predictive - Future. - Probabilistic. - Based on Descriptive. - This is how pundits predict stocks.
- 14. Prescriptive
- 16. What does data look like?
- 17. Storage Choice 1
- 18. Storage Choice 2
- 19. Oh no
- 21. Scaling
- 22. Archival Policy
- 24. All related entities end up in complex joins
- 25. All Relationships complicate over Dimension of time
- 26. Anatomy 26
- 27. Anatomy
- 29. Star Schema
- 30. Snowflake Schema
- 31. De-Duplication
- 32. Bloom Filters Cuckoo Filters - Does not exist for sure. - May or may not exist.
- 36. Cubes ● Efficiency of Retrieval ● Warehouse:Cube :: DB:Table ● View: Dimension + Measure ● Slice, Dice & Rotate
- 38. Sample Solution