Big Data Science at the Digital Catapult
Download as PPTX, PDF3 likes817 views
The document discusses the concepts of big data and data science, comparing big data to crude oil and data science to an oil refinery where data is refined into insights. It highlights the importance of big data's volume, velocity, and variety, and outlines the roles of various data professionals such as data engineers and data scientists. Additionally, it addresses the challenges and potential of big data across different sectors, emphasizing the need for agility and capability in managing data effectively.
![BIG DATA SCIENCE
Chandan Rajah [ @ChandanRajah ]
“The price of light is far less than the cost of darkness”](https://image.slidesharecdn.com/cdecbigdatascience20140918-140922054432-phpapp01/85/Big-Data-Science-at-the-Digital-Catapult-1-320.jpg)
![Data Analytics to Data Discovery ?
data you know
data you don’t know
questions you’re asking
questions you’re not asking
Data Analyst
Data Scientist
Data
Analytics
Data Discovery
DATA MODELLING
Y F( X, random noise, parameters)
ALGORITHMIC MODELLING
Y [ BLACK BOX ] X](https://image.slidesharecdn.com/cdecbigdatascience20140918-140922054432-phpapp01/85/Big-Data-Science-at-the-Digital-Catapult-7-320.jpg)
Big Data Science at the Digital Catapult
- 1. BIG DATA SCIENCE Chandan Rajah [ @ChandanRajah ] “The price of light is far less than the cost of darkness”
- 2. COST SPEED BENEFITS OF BIG DATA AGILITY CAPABILITY
- 3. WHAT WHY Steps to the EPIPHANY WHERE DEMO
- 4. What is Big Data ? Big Data ≠ Data Volume Big Data = Crude Oil Think of data like ‘Crude Oil’ Big Data is about extracting ‘crude oil’; transporting it in ‘pipelines’; storing it in ‘mega tanks’
- 5. What is Data Science ? Data Science ≠ Statistical Analysis Data Science = Oil Refinery Data science is about ‘treating’ data; applying ‘science’ to the data; Refine the data ‘results’; and combine to form ‘insight’
- 6. Knowns, Unknowns & DIKUW FTW! known knowns we know we know known unknowns we know we don’t know unknown unknowns we don’t know we don’t know D DATA I INFORMATION K KNOWLEDGE W WISDOM U UNDERSTANDING PAST FUTURE Data Engineer Data Analyst Data Miner Data Scientist raw what how to why when numbers description experience cause & effect prediction letters context tested proven what’s best symbols relationship instruction known knowns known unknowns unknown unknowns signals reports programs models
- 7. Data Analytics to Data Discovery ? data you know data you don’t know questions you’re asking questions you’re not asking Data Analyst Data Scientist Data Analytics Data Discovery DATA MODELLING Y F( X, random noise, parameters) ALGORITHMIC MODELLING Y [ BLACK BOX ] X
- 8. DIVIDE SCATTER Split Data in Block Replicate and Store Petabytes of Resilience CONQUER EXPLORE 1000s of Parallel Threads Explore Every Path Machine Learning INSIGHT GATHER Real Time Action Periodic Dashboards Iterative Evolution What is the Big Idea ?
- 9. Divide = HDFS Name Node Client 1. Create Metadata 2. Put Blocks 1 2 3 Control / Monitoring 2 2 1 1 Data Nodes 3 3 WRITE Name Node Client 1. Get Metadata Control / Monitoring 1 1 1 2 2 2 4 3 3 3 4 4 2. Fetch Blocks Data Nodes READ
- 11. Insight = Functional Paradigm
- 12. WHAT WHY Steps to the EPIPHANY WHERE DEMO
- 13. Why is Big Data needed ? VOLUME VELOCITY VARIETY Exponential growth; 2x in 2 yrs PB (1000 TB) is now common Event streams; never at rest 640k GB per internet minute 100s of data sources 85% not in a table
- 14. Where in the Value Chain ? Generation Transport Knowledge Output Value BIG DATA SCIENCE Straddles all four Challenge Areas
- 15. WHAT WHY Steps to the EPIPHANY WHERE DEMO
- 16. Big Data Heat Map – Gartner 2012
- 17. Big Data Potential by Sector – McKinsey for USBLS, 2011
- 18. Big Data Investment by Industry – Gartner, 2012
- 19. Top Big Data Challenges – Gartner, 2012
- 20. Survey on Big Data Investments – IDG Survey, 2013
- 21. Survey on Main Drivers to Invest – IDG Survey, 2014
- 22. WHAT WHY Steps to the EPIPHANY WHERE DEMO
- 23. DEMO
- 24. COST SPEED RECAP OF BENEFITS AGILITY CAPABILITY
- 25. TIME VALUE OF DATA KNOWLEDGE IS POWER LAST WORDS OF WISDOM NOT ALL ROADS LEAD TO ROME I AM AN INDIVIDUAL
- 26. “The price of light is far less than the cost of darkness”
Editor's Notes
- #3: COST – 20x less per TB v/s Teradata, Netezza, Oracle – 75% less average marginal cost per capacity SPEED – 10x faster than Teradata, Netezza AGILITY – 115% lesser average cost per data source v/s Oracle SCIENCE – Machine learning, prediction
- #4: WHAT - What is Big Data Science? WHY - Why is it needed? WHERE - Where is it being used? HOW - How will it evolve?
- #13: WHAT - What is Big Data Science? WHY - Why is it needed? WHERE - Where is it being used? HOW - How will it evolve?
- #16: WHAT - What is Big Data Science? WHY - Why is it needed? WHERE - Where is it being used? HOW - How will it evolve?
- #23: WHAT - What is Big Data Science? WHY - Why is it needed? WHERE - Where is it being used? HOW - How will it evolve?
- #25: COST – 20x less per TB v/s Teradata, Netezza, Oracle – 75% less average marginal cost per capacity SPEED – 10x faster than Teradata, Netezza AGILITY – 115% lesser average cost per data source v/s Oracle SCIENCE – Machine learning, prediction
- #26: TIME VALUE - Yesterday’s data is less valuable than today’s data - Historical data is more valuable than just now alone POWER - Get from unknown unknowns to known unknowns or known knowns is powerful LEAD TO ROME - Exploring with no direct business impact is not a bad thing INDIVUDUAL - Treat every customer as an individual not an aggregate and analyse - Aggregate only individual insights