Big Data
2 likes326 views
This document discusses big data and provides motivation for why companies need to learn how to manage large amounts of data. It defines big data as data that exceeds the processing capacity of traditional databases due to its large volume, velocity, or variety. The document outlines the characteristics of big data using the "5 Vs" - volume, velocity, variety, viability and value. It then provides an overview of how big data works by presenting the big data supply chain. The key points are that big data enables new insights but also poses challenges for storage, analysis and use due to its scale and complexity.
![The
Core
of
Hadoop
-‐
MapReduce
44
Input
Output
Reduce
Reduce
Shuffle
Map
e
…
com
a
mulher
e
o
filho…
(
1,
e)
(3,
com)
(2,
a
)
(6,
mulher)
(5,
e
)
(7,
o)
(4,
filho)
(1,
e)
-‐>
2
(3,
com)
-‐>
1
(2,
a
)
-‐>
1
(6,
mulher)
-‐>
1
(7,
o)
-‐>
1
(4,
filho)
-‐>
1
e
[4,2,6]
a
[3,1,2]
com
[5,1,2]
filho
[2,1,1]
mulher
[3,1,10]
O
[7,1,9]](https://image.slidesharecdn.com/bigdata-150512222453-lva1-app6892/85/Big-Data-44-320.jpg)
![The
Core
of
Hadoop
-‐
MapReduce
45
Input
Output
Reduce
Reduce
Shuffle
Map
e
…
com
a
mulher
e
o
filho…
(
1,
e)
(3,
com)
(2,
a
)
(6,
mulher)
(5,
e
)
(7,
o)
(4,
filho)
(1,
e)
-‐>
2
(3,
com)
-‐>
1
(2,
a
)
-‐>
1
(6,
mulher)
-‐>
1
(7,
o)
-‐>
1
(4,
filho)
-‐>
1
e
[4,2,6]
a
[3,1,2]
com
[5,1,2]
filho
[2,1,1]
mulher
[3,1,10]
O
[7,1,9]
e
-‐>
12
a
-‐>
6
com
-‐>
8
filho
-‐>
4
mulher
-‐>
14
o
-‐>
17](https://image.slidesharecdn.com/bigdata-150512222453-lva1-app6892/85/Big-Data-45-320.jpg)
Big Data
- 1. Big Data Ins+tuto Superior Técnico Technical University of Lisbon Bruno Lopes Catarina Moreira João Pinho
- 2. Mo#va#on 2 220 2 PetaBytes Of data that people create every day!
- 3. Mo#va#on 3 90 % of Data UNSTRUCTURED Which is difficult to manage and interpret!
- 4. Mo#va#on 4 2 hours per day Spent searching for the right informa#on
- 5. Mo#va#on 5 $5 million per year Money lost due to data-‐related problems
- 6. Mo#va#on 6 1 in 3 Business leaders make decisions based on informa#on they don’t trust or don’t have 1 in 2 Business leaders say they don’t have access to the informa#on they need to do their jobs 60% CEO’s need to do a beQer job capturing and understanding informa+on rapidly in order to make swiS business decisions Of world’s data is unstructured 80 % Organiza+ons Need Deeper Insights … Informa+on is at the Center of a New Wave of Opportunity Source: Big Data by Ami Redwan Haq, Founder at Sen:nel Solu:ons Ltd 44 % as much Data and Content Over the Coming Decade 2009 800 000 PetaBytes 2020 35 ZeQaBytes
- 7. What is Big Data? 7 Big Data (video)
- 8. What is Big Data? Data that exceeds the processing capacity of a conven+onal database. The data is too big, moves too fast or doesn’t fit the structures of your database architectures. 8 Source: J. Hurwitz, A. Nugent, F. Halper & M. Kaufman, Big Data for Dummies, Willey & Sons, 2013
- 9. Why Big Data? Big data enables organiza+ons to store, manage and manipulate vast amounts of data at the right speed and at the right #me to gain the right insights. 9 Source: J. Hurwitz, A. Nugent, F. Halper & M. Kaufman, Big Data for Dummies, Willey & Sons, 2013
- 10. Why Big Data? • The value of Big Data falls into 2 categories: 1. Analy#cal Use. • Reveal new insights that were hidden in too costly to process massive amounts of data; • Example: Peer influence among customers; 2. Enabling New Products. • Combining large number of signals from user’s ac+ons and friends • Example: Facebook 10
- 11. The V’s Big Data is defined as any kind of data source that has at least three shared characteris+cs: 11 Velocity Variety Volume Viability Value
- 12. Velocity Increasing rate at which data flows into organiza#ons 12
- 13. Velocity The Large Hadron Collider at CERN generates 13 Scien+sts are forced to 25 PetaBytes per Year! Discard Massive Data In order to keep requirements prac+cal storage
- 14. Velocity The way we deliver and consume products and services is genera+ng a massive data flow to the provider • It’s possible to stream fast-‐moving data into bulk storage for later batch processing. • The importance lies in taking data from input through decision. 14
- 15. Volume Ability to store massive amounts of (un)structured data 15
- 16. Volume 16 No longer a problem!
- 17. Volume 17 Don’t forget the cloud!
- 18. Volume -‐ Challenges • How to determine relevance within large data volumes • How to use analy+cs to create value from relevant data. 18
- 19. Variety Managing, merging and governing different varie+es of data. 19
- 20. Variety • Source data is diverse and doesn’t fall into neat rela+onal structures • Data comes in the form of emails, photos, videos, monitoring devices, PDFs, audio, etc. • This variety of (un)structured data creates problems for storage, mining and analyzing data 20
- 21. Viability Quickly and cost-‐effec#vely test and confirm a par+cular variable’s relevance before inves+ng in the crea+on of a fully featured model. 21
- 22. Value Gain new insights about data by analyzing variables that were hidden 22
- 23. Big Data 23 So, how does it work?
- 24. The Big Data Supply Chain 24 Source: Edd Dumbill, Planning for Big Data, O’Reilly, 2012
- 25. Summary • Mo+va+on of why companies need to learn how to deal with Big Data! • Presented a defini+on of Big Data and analyzed the V’s. • Presented an analysis of the Big Data Supply chain for enterprises. 25
- 26. 26 Technologies for Big Data
- 27. The Reach of Big Data • So far, the analy+cal use of Big Data has been in reach only to leading coopera+ons: 27 But at a fantas+c cost!
- 28. The Reach of Big Data But now, there are several open source applica+ons than can tackle Big Data related challenges! 28
- 29. Apache Hadoop • Developed and released by Yahoo! • Open source framework for storage and large scale parallel processing of datasets on clusters. • Ability to cheaply process large amounts of data, regardless of its structure. 29
- 30. Apache Hadoop 30
- 31. Apache Hadoop 31
- 32. (Video -‐ MapReduce) • IBM – MapReduce • Health Care -‐ Real Time Alerts 32
- 33. HDFS – Hadoop Distributed File System 33
- 34. 34 • Detec+on of faults and quick, automa+c recovery • Emphasis on high throughput of data access • Tuned to support large datasets • Write-‐once-‐read-‐many access model: once a file is created, wriQen and closed, needs not to be changed HDFS – Hadoop Distributed File System
- 35. 35 • Moving computa+on is cheaper than moving data – minimizes network conges+on – Increases the overall throughput of the system • Portability Across Heterogeneous Hardware and SoSware Plaqorms HDFS – Hadoop Distributed File System
- 36. HDFS – Hadoop Distributed File System 36
- 37. The Core of Hadoop -‐ MapReduce 37 • Created by Google for web search indexes • Ability to take a query over a dataset, divide it and run it in parallel over mul+ple nodes • Consists in three stages: – Map – Shuffle – Reduce
- 38. The Core of Hadoop -‐ MapReduce 38 Input Output Reduce Shuffle Map
- 39. The Core of Hadoop -‐ MapReduce 39 Input Output Reduce Reduce Shuffle Map
- 40. The Core of Hadoop -‐ MapReduce 40 • Imagine that you want to apply MapReduce to word coun+ng in books
- 41. The Core of Hadoop -‐ MapReduce 41 Input Output Reduce Reduce Shuffle Map E … com a mulher e o filho…
- 42. The Core of Hadoop -‐ MapReduce 42 Input Output Reduce Reduce Shuffle Map E … com a mulher e o filho… ( 1, E) (2, com) (3, a ) (4, mulher) (5, e ) (6, o) (7, filho)
- 43. The Core of Hadoop -‐ MapReduce 43 Input Output Reduce Reduce Shuffle Map e … com a mulher e o filho… ( 1, e) (2, com) (3, a ) (4, mulher) (5, e ) (6, o) (7, filho) (1, e) -‐> 2 (2, com) -‐> 1 (3, a ) -‐> 1 (4, mulher) -‐> 1 (5, o) -‐> 1 (6, filho) -‐> 1
- 44. The Core of Hadoop -‐ MapReduce 44 Input Output Reduce Reduce Shuffle Map e … com a mulher e o filho… ( 1, e) (3, com) (2, a ) (6, mulher) (5, e ) (7, o) (4, filho) (1, e) -‐> 2 (3, com) -‐> 1 (2, a ) -‐> 1 (6, mulher) -‐> 1 (7, o) -‐> 1 (4, filho) -‐> 1 e [4,2,6] a [3,1,2] com [5,1,2] filho [2,1,1] mulher [3,1,10] O [7,1,9]
- 45. The Core of Hadoop -‐ MapReduce 45 Input Output Reduce Reduce Shuffle Map e … com a mulher e o filho… ( 1, e) (3, com) (2, a ) (6, mulher) (5, e ) (7, o) (4, filho) (1, e) -‐> 2 (3, com) -‐> 1 (2, a ) -‐> 1 (6, mulher) -‐> 1 (7, o) -‐> 1 (4, filho) -‐> 1 e [4,2,6] a [3,1,2] com [5,1,2] filho [2,1,1] mulher [3,1,10] O [7,1,9] e -‐> 12 a -‐> 6 com -‐> 8 filho -‐> 4 mulher -‐> 14 o -‐> 17
- 46. Who Uses Hadoop? 46
- 47. The Social Genome product: • Reach customers, or friends of customers, who have men+oned product and include a discount • Combines public data from the web, social data and proprietary data 47 Walmart (Real Time)
- 48. The Social Genome product: • Helps Walmart to understand the context of what their customers are saying online • When a person tweets I love Salt, Walmart can understand that she is talking about the movie Salt and not the condiment. 48 Walmart (Real Time)
- 49. Apache Hive 49 • Data warehouse soSware that facilitates querying and managing large datasets residing in a distributed storage • Uses HiveSQL to analyze data • Allows programmers to use their own map reduce func+ons
- 50. Apache Cassandra 50 • Database with high scalability and high availability without compromising performance. • Offers – Denormaliza+on – Materialized views – Powerful built in cache
- 51. Google Big Query 51 Built on GoogleFS
- 52. Summary 52 • Apache Hadoop techhnology – HDFS file system – MapReduce paradigm • Apache Hive • Apache Cassandra • Google BigQuery
- 54. Project HEIDI 54 How to index high dimensional data?
- 55. Project HEIDI 55 • Mul+media datasets are growing! – Ex: Audio, video, medical images, photos, etc. • We need techniques to efficiently manage and access informa+on in such large datasets! • Mul+media datasets cannot be searched like tradi+onal databases (e.g. Text search). They are searched in metric spaces.
- 56. Project HEIDI Given a mul#media object as input, return the most similar objects from a mul+media database 56
- 57. Project HEIDI 57 • In content-‐based image retrieval, images are described by their proper+es: • Colors • Texture • Shape • Shadows • etc
- 58. Project HEIDI 58 • The process of conver+ng a mul+media object into a vector with its contents/features is called feature extrac,on.
- 59. 59
- 60. Project HEIDI 60 The problem in high dimensional indexing is the
- 61. Project HEIDI 61 The problem in high dimensional indexing is the OF DIMENSIONALITY!!!
- 62. The Curse of Dimensionality 62 • A phenomena that arises when analyzing and organizing data in high dimensional spaces. • It does not occur in lower dimensions!
- 63. The Curse of Dimensionality 63 • When dimensionality increases, the volume of the space increases so fast that the data become sparse. • Tradi+onal tree indexing structures do not provide any advantages in the index process (outperformed by linear scan)…
- 64. The Curse of Dimensionality 64 • Consequences: – (hyper) cubes – (hyper) spheres • Volume increases exponen#ally with growing dimension (while the edge length remains constant).
- 65. The Curse of Dimensionality 65 How can this curse affect datasets in metric spaces?
- 66. The Curse of Dimensionality 66 • In metric spaces, there is the Nearest Neighbor paradigm. • The NN-‐sphere contains one point. • For higher dimensions, the radius of the NN-‐sphere will be larger than the size of the database.
- 67. The Curse of Dimensionality 67 Literature addresses this issue in two ways: 1. Approximate Nearest Search 2. Reduce the dimensionality of the mul+media objects
- 68. Project HEIDI 68 The algorithm that is proposed in this project does not suffer from the curse of dimensionality!
- 69. Project HEIDI 69 The Linear SubSpace Indexing Algorithm • Based on the idea of subspaces! – Feature extrac+on func+on that maps the high dimensional objects into low dimensional equivalent objects.
- 70. Project HEIDI 70 The Quick and Dirty Paradigm • Discard data objects in lower dimensional spaces • Massive comparison opera+ons are less expensive in lower dimensions
- 71. Project HEIDI 71 However, this approach has
- 72. Project HEIDI 72 The Quick and Dirty Paradigm • Can produce lots of false hits! • The mapping to lower dimensions can lead to closer data points
- 73. Project HEIDI 73 Are false hits really a
- 74. Project HEIDI 74
- 75. Project HEIDI 75 • Just perform a final comparison in the original space! • All false hits will be discarded! • The computa+on is quick, because the massive amount of objects were discarded in the lower dimensions.
- 76. Project HEIDI 76
- 77. Project HEIDI 77 But… How to map high dimensional objects into lower dimensional spaces?
- 78. Project HEIDI 78 The Quick and Dirty Paradigm • Lower bonding lemma: – Distances between objects in the feature space are always smaller or equal that in the original space
- 79. Project HEIDI 79 The Quick and Dirty Paradigm • The Principal Component Analysis algorithm maps objects into lower dimensions • Sa+sfies the lower bounding lemma
- 80. Project HEIDI 80 The Principal Component Analysis
- 81. Project HEIDI 81 The Principal Component Analysis 1. Compute Co-‐Variance Matrix
- 82. Project HEIDI 82 The Principal Component Analysis 2. Compute eigen vectors and eigen values from co-‐variance matrix 3. Get the most significant eigen vectors and create a projec+on matrix 4. Project data
- 83. Project HEIDI 83 The Hierarchical Linear SubSpace Indexing Method 1. Index Phase (off-‐line) – Par++on large dataset into chunks of data – Itera+vely apply the Principal Component Analysis to project the dataset into lower dimensions – This ac+on can be parallelized in many servers
- 84. Project HEIDI 84 The Hierarchical Linear SubSpace Indexing Method
- 85. Project HEIDI 85 The Hierarchical Linear SubSpace Indexing Method Original Space Lower Space
- 86. Project HEIDI 86 The Hierarchical Linear SubSpace Indexing Method Original Space Lower Space PCA
- 87. Project HEIDI 87 The Hierarchical Linear SubSpace Indexing Method Original Space Lower Space PCA
- 88. Project HEIDI 88 The Hierarchical Linear SubSpace Indexing Method Original Space Lower Space PCA PCA
- 89. Project HEIDI 89 The Hierarchical Linear SubSpace Indexing Method Original Space Lower Space PCA PCA
- 90. Project HEIDI 90 The Hierarchical Linear SubSpace Indexing Method Original Space Lower Space PCA PCA PCA
- 91. Project HEIDI 91 The Hierarchical Linear SubSpace Indexing Method Original Space Lower Space PCA PCA PCA
- 92. Project HEIDI 92 The Hierarchical Linear SubSpace Indexing Method Original Space Lower Space PCA PCA PCA PCA
- 93. Project HEIDI 93 The Hierarchical Linear SubSpace Indexing Method Original Space Lower Space PCA PCA PCA PCA
- 94. Project HEIDI 94 The Hierarchical Linear SubSpace Indexing Method Original Space Lower Space PCA PCA PCA PCA PCA
- 95. Project HEIDI 95 The Hierarchical Linear SubSpace Indexing Method Original Space Lower Space PCA PCA PCA PCA PCA
- 96. Project HEIDI 96 The Hierarchical Linear SubSpace Indexing Method Original Space Lower Space PCA PCA PCA PCA PCA PCA
- 97. Project HEIDI 97 The Hierarchical Linear SubSpace Indexing Method
- 98. Project HEIDI 98 The Hierarchical Linear SubSpace Indexing Method
- 99. Project HEIDI 99 The Hierarchical Linear SubSpace Indexing Method
- 100. Project HEIDI 100 The Lower Bounding Lemma is sa#sfied Euclidean Distance Most Similar Images Retrieved
- 101. Project HEIDI 101 The Hierarchical Linear SubSpace Indexing Method
- 102. Project HEIDI 102 The Hierarchical Linear SubSpace Indexing Method 2. Query Phase (on-‐line) – Project queries using projec+on matrices from step 1 – Star+ng in the lowest dimensional space, itera+vely discard all objects that are different from the query – Keep doing this un+l the original space is reached and the false hits are discarded.
- 103. Project HEIDI 103 The Hierarchical Linear SubSpace Indexing Method
- 104. Summary • Important concepts in Big Data for High Dimensional Datasets • The curse of dimensionality • Projec+on techniques • The Lower Bounding Lemma • The Quick and Dirty paradigm (some rela+on to the MapReduce paradigm) 104
- 105. Summary Mathema#cs is more important than engineering! 105
- 106. Now… Time to See This Working for REAL!!! 106