Demystifying big data
0 likes154 views
This document provides an overview of big data and how to approach it. It discusses how data has become more distributed and large-scale, necessitating new tools and approaches. It introduces data parallel processing which distributes data, moves computation close to data, and aggregates results. This allows scaling beyond a single machine. It also discusses challenges like data distribution, fault tolerance, and abstraction. The document then outlines the major components of a big data ecosystem, including distributed data storage, processing systems like Hadoop and Spark that handle distribution and fault tolerance, and coordination systems. It provides examples of using big data for recommendations and inventory predictions.
![Data Parallel Processing
❏ Distribute the data [ With replication]
❏ Move Computation close to Data
❏ Process each section of Data separately
❏ Aggregate the results.](https://image.slidesharecdn.com/demystifyingbigdata-151002120517-lva1-app6891/85/Demystifying-big-data-13-320.jpg)
Demystifying big data
- 5. Result
- 6. What’s next? Unanswered question of lifetime.
- 7. Unquenchable thirst of improvement ❏ How to Sell more? ❏ How to optimize inventory? ❏ How to engage customer more? ❏ What do my customer Like? ❏ How to reduce Operation Cost?
- 8. Torture the data, and it will confess to anything Ronald Coase
- 9. How to get Data? Humans…..
- 10. Ever Growing Data ❏ Historical data plays important role. ❏ Data explodes while processing. ❏ More data beats better algorithms.
- 11. So What is Big Data? When data has tendency to grow more than what one machine can process.
- 13. Data Parallel Processing ❏ Distribute the data [ With replication] ❏ Move Computation close to Data ❏ Process each section of Data separately ❏ Aggregate the results.
- 14. Advantages of Data Parallel Model ❏ No Hardware restriction. e.g Memory, CPU. ❏ No Scalability Issue ❏ Cost effectiveness. ❏ No Single point of failure.
- 15. That’s nice, So problem solved. But Presentation says Hadoop,Spark?
- 16. Challenges of Data-||-sim ❏ Data partitioning, distribution and accumulation ❏ Fault Tolerance. ❏ Distributed Coordination and management. ❏ Abstraction with the distributed complexity.
- 17. Big Data Ecosystem ❏ Distributed Data Storage System: ❏ Data distribution. ❏ Data Replication. ❏ High throughput with no single point of failure. ❏ Distributed Data Processing System: ❏ Distributing Code close to data. ❏ Abstracting distributed complexity from programmer. ❏ Fault tolerance and handling computation failure. ❏ Aggregating results. ❏ Distributed Coordination and Resource management. ❏ Resource allocation. ❏ Distributed configuration management.
- 18. Distributed Data Storage System
- 19. Distributed Data Processing System
- 20. Distributed Coordination and Resource management.
- 22. How to Sell more? Recommendation.
- 23. Speed Layer 2. Product Views 1. Web Log 3. Similar Product 4. Update user product recommendation
- 25. Batch Layer 1. User Data 2. Location Cluster per item 3. Location Cluster per item Data 3. Current Warehouse inventory 4. Inventory transfer.