Big data – a brief overview
Download as PPTX, PDF7 likes2,330 views
Big data comes from a variety of sources like social networks, sensors, and financial transactions. It is characterized by its volume, velocity, and variety. Hadoop and NoSQL platforms are commonly used to process and analyze big data. There are many opportunities for applications in domains like healthcare, retail, and finance. However, addressing the skills gap for data scientists remains a key challenge for fully realizing the potential of big data.
Big data – a brief overview
- 1. Big Data – A Brief Overview Petabytes, Hadoop, Analytics, Collaborative business intelligence, Data scientists, In-Memory Databases, NoSQL platforms
- 2. Big Data • What is it? • Where does it come from? • How do we process it? • What do we do with it? • Who are the players? • What are the opportunities?
- 3. What Is Big Data? Like the term Cloud, it is a bit Nebulous
- 5. Attributes of Big Data • Volume • Velocity - streaming • Variety
- 6. Where Does It Come From? It Depends
- 7. Key Drivers Spread of cloud computing, mobile computing and social media technologies, financial transactions
- 8. Sources of Big Data • Chatter from social networks, • Web server logs, • Traffic flow sensors, • Satellite imagery, • Broadcast audio streams, • Banking transactions, • MP3s of rock music, • The content of web pages, • Scans of government documents, • GPS trails, • Telemetry from automobiles, • Financial market data • ….
- 11. How Do We Process It?
- 13. Hadoop A distributed processing Framework based on Map/Reduce
- 14. Pig A platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs.
- 15. Mahout A machine learning library with algorithms for clustering, classification and batch based collaborative filtering that are implemented on top of Apache Hadoop.
- 16. Hive Data warehouse software built on top of Apache Hadoop that facilitates querying and managing large datasets residing in distributed storage.
- 17. Pegasus A Peta-scale graph mining system that runs in parallel, distributed manner on top of Hadoop
- 18. Sqoop A tool designed for efficiently transferring bulk data between Apache Hadoop and structured data stores such as relational databases.
- 19. Flume A distributed service for collecting, aggregating, and moving large log data amounts to HDFS.
- 20. Yahoo S4 S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data.
- 21. Twitter Storm Storm can be used to process a stream of new data and update databases in real time.
- 22. Trends Funding, Companies, Applications, Jo bs, IPOs
- 23. Funding & IPO • Cloudera, (Commerical Hadoop) more than $75 million • MapR (Cloudera competitor) has raised more than $25 million • 10Gen (Maker of the MongoDB) $32 million • DataStax (Products based on Apache Cassandra) $11 million • Splunk raised about $230 million through IPO
- 26. Big Data Application Domains • Healthcare • The public sector • Retail • Manufacturing • Personal-location data • Finance
- 27. A Few Examples
- 31. Market and Market Segments Research Data and Predictions
- 34. Market for big data tools will rise from $9 billion to $86 billion in 2020
- 37. Future of Big Data • More Powerful and Expressive Tools for Analysis • Streaming Data Processing (Storm from Twitter and S4 from Yahoo) • Rise of Data Market Places (InfoChimps, Azure Marketplace) • Development of Data Science Workflows and Tools (Chorus, The Guardian, New York Times) • Increased Understanding of Analysis and Visualization http://www.evolven.com/blog/big-data-predictions.html
- 39. Opportunities
- 40. Skills Gap • Statistics • Operations Research • Math • Programming • So-called "Data Hacking"