Introduction to Big Data
0 likes274 views
The document provides an overview of big data, defining it as data that exceeds the capabilities of conventional database systems due to its size, speed, or complexity. It emphasizes the importance of addressing specific business challenges and starting small to leverage big data technologies effectively. Additionally, it highlights the potential benefits such as scalability, robustness, and predictable performance while cautioning against the hype and challenges associated with big data implementation.
Introduction to Big Data
- 1. big data So What? 12 October 2016 1
- 2. Who am I? • Software guy • Technology leader with experience in software development as CTOs and development managers of mid-sized teams. • Doing big data hands-on since 2009 • Running http://meetup.com/bigdatabe since 2011 (1700 members!) 2 @wimvanleuven wim@bigboards.io
- 3. 3
- 4. 4
- 5. “Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures.” 5 –Edd Dumbill, O’Reilly What is big data? http://radar.oreilly.com/2012/01/what-is-big-data.html
- 7. … moves to fast … 7
- 8. 8
- 9. … doesn’t fit … 9
- 10. What is Big Data not? • not a delivery model (on-premise vs hosted vs cloud vs IaaS/PaaS/SaaS vs serverless) • not a deployment model (private, public, hybrid) • not a revenue model (license vs subscription vs Pay-as-you-Go) • not software architecture 10
- 11. “We don’t do Hadoop because we have Big Data; we do Big Data because we have Hadoop.” 11 –Unknown developer, Facebook What is Big Data? — revisited
- 12. New tools and technologies to capture and process data on a cluster of commodity hardware so that the system acts as one, is resilient to failures and scales linearly. 12 What is Big Data? — revisited
- 13. Big Data is no panacea 13 • First decide what problem you want to solve; pick a real business problem to add immediate value • Start small, the technology is made for linear scalability (a 3-node cluster is a cluster!) • Then become lean: learn through experimentation
- 14. Big Data challenges • Beware of hype, Big Data - washing and fad • Tech infancy • IT | Biz • Data is hard • Lack of skills! 14
- 15. Benefits • Scalability of course • Collect more and more data • Robustness inherent to the setup • More predictable performance 15
- 16. 16 Questions?
- 18. DFS 18 1 2 3 4 5 2 4 5 1 2 5 1 3 4 2 3 5 1 3 4 Node A Node B Node C Node D Node E
- 19. MapReduce 19 4 5 3 2 1 Node A Node B Node C Node D Node E Map Shuffle Reduce x y z
- 20. 𝛌 20
- 21. 𝛋 21 3 1 2 45
- 22. 22 Q&A