L18 Big Data and Analytics
1 like206 views
The document discusses the evolution and importance of big data, emphasizing its exponential growth and its role as a new growth platform for businesses. It highlights various big data applications and examples from companies like Macy's, Wal-Mart, and American Express, showcasing how they leverage data analytics for improved efficiency, real-time decision making, and customer insights. The document also explains the concept of big data, its characteristics, and the need for specialized processing techniques to extract valuable insights from vast amounts of data.
L18 Big Data and Analytics
- 1. LECTURE L18 BIG DATA AND ANALYTICS
- 2. Big Data With the computer revolution, digital data becomes possible Over the years, data has grown exponentially “Big Data” has become a platform by itself with new possibilities
- 3. Global Data is Growing Fast Data in Digital Universe vs. Data Storage Cost, 2010-2015 Source: Mary Meeker, KPCB
- 4. Evolution of Data Platform Source: Mary Meeker, KPCB
- 5. Data is a New Growth Platform The Network The Software The Infrastructure The Data Large investments in fibre optic & last-mile cable create connectivity that facilitated the early Internet growth Optimising the network with software became far more capital efficient than additional capital expenditure buildouts, ultimately resulting in the creation of pervasive networks (Siloed DCs -> AWS) and pervasive software (Siebel -> Salesforce) Emergence of pervasive software created the need to optimise the performance of the network and store extraordinary amounts of data at extremely low prices Next Big Wave: Leveraging this unlimited connectivity and storage to collect / aggregate / correlate / interpret all of this data to improve people’s live and enable enterprises to operate more efficiently
- 6. Data Generators Source: Mary Meeker, KPCB
- 7. “Data is moving from something you use outside the workstream to becoming a part of the business app itself.” — Frank Bien, CEO of Looker
- 8. Improve people’s live and enable enterprises to operate more efficiently
- 10. Big Data Examples Macy's Inc. and real-time pricing The retailer adjusts pricing in near-real time for 73 million items, based on demand and inventory. Source:Ten big data case studies in a nutshell
- 11. Big Data Examples Tipp24 AG, a platform for placing bets The company uses software to analyse billions of transactions and hundreds of customer attributes, and to develop predictive models that target customers and personalise marketing messages on the fly. Source:Ten big data case studies in a nutshell
- 12. Big Data Examples Wal-Mart Stores Inc. and search The mega-retailer's latest search engine for Walmart.com includes semantic data. A platform that was designed in- house, relies on text analysis, machine learning and even synonym mining to produce relevant search results. Wal-Mart says adding semantic search has improved online shoppers completing a purchase by 10% to 15%. Source:Ten big data case studies in a nutshell
- 13. Big Data Examples PredPol Inc. and repurposing The Los Angeles and Santa Cruz police departments, a team of educators and a company called PredPol have taken an algorithm used to predict earthquakes, tweaked it and started feeding it crime data. The software can predict where crimes are likely to occur down to 500 square feet. In LA, there's been a 33% reduction in burglaries and 21% reduction in violent crimes in areas where the software is being used. Source:Ten big data case studies in a nutshell
- 14. Big Data Examples American Express and business intelligence AmEx started looking for indicators that could really predict loyalty and developed sophisticated predictive models to analyse historical transactions and 115 variables to forecast potential churn The company believes it can now identify 24% of Australian accounts that will close within the next four months Source:Ten big data case studies in a nutshell
- 15. Big Data Examples A Bank and IBM A large US bank uses IBM machine learning technologies to analyse credit card transactions. Using machine learning and stream computing to detect financial fraud
- 16. TEDxUofM - Jameson Toole - Big Data for Tomorrow
- 18. What is Big Data?
- 19. What is Big Data? Big data is high-volume, high-velocity and/or high-variety information assets that demand cost-effective, innovative forms of information processing that enable enhanced insight, decision making, and process automation. Gartner
- 20. What is Big Data? Big data refers to a process that is used when traditional data mining and handling techniques cannot uncover the insights and meaning of the underlying data. Data that is unstructured or time sensitive or simply very large cannot be processed by relational database engines. This type of data requires a different processing approach called big data, which uses massive parallelism on readily-available hardware. Techopedia
- 21. “Big data is the oil of the 21st century and analytics is the combustion engine.” —Peter Sondergaard, Gartner Research What is Big Data?
- 22. How do you measure numbers at large scale? What is Big Data?
- 23. What is a Yottabyte?
- 24. 1.000.000.000.000.000.000.000.000 What is a Yottabyte?
- 25. Byte: one rice David Wellman: What is Big Data? What is Big Data?
- 26. Byte: one rice Kilobyte: handful of rice David Wellman: What is Big Data? What is Big Data?
- 27. Byte: one rice Kilobyte: handful of rice Megabyte: Big pot of rice David Wellman: What is Big Data? What is Big Data?
- 28. Byte: one rice Kilobyte: handful of rice Megabyte: Big pot of rice Gigabyte: Truck full of rice David Wellman: What is Big Data? What is Big Data?
- 29. Byte: one rice Kilobyte: handful of rice Megabyte: Big pot of rice Gigabyte: Truck full of rice Terabyte: Containership full of rice David Wellman: What is Big Data? What is Big Data?
- 30. Byte: one rice Kilobyte: handful of rice Megabyte: Big pot of rice Gigabyte: Truck full of rice Terabyte: Containership full of rice Petabyte: Covers Manhattan David Wellman: What is Big Data? What is Big Data?
- 31. Byte: one rice Kilobyte: handful of rice Megabyte: Big pot of rice Gigabyte: Truck full of rice Terabyte: Containership full of rice Petabyte: Covers Manhattan Exabyte: Covers the west coast of US David Wellman: What is Big Data? What is Big Data?
- 32. Byte: one rice Kilobyte: handful of rice Megabyte: Big pot of rice Gigabyte: Truck full of rice Terabyte: Containership full of rice Petabyte: Covers Manhattan Exabyte: Covers the west coast of US Zettabyte: Fills the Pacific Ocean David Wellman: What is Big Data? What is Big Data?
- 33. Byte: one rice Kilobyte: handful of rice Megabyte: Big pot of rice Gigabyte: Truck full of rice Terabyte: Containership full of rice Petabyte: Covers Manhattan Exabyte: Covers the west coast of US Zettabyte: Fills the Pacific Yottabyte: Earth size riceball David Wellman: What is Big Data? What is Big Data?
- 34. Byte: one rice Kilobyte: handful of rice Megabyte: Big pot of rice Gigabyte: Truck full of rice Terabyte: Containership full of rice Petabyte: Covers Manhattan Exabyte: Covers the west coast of US Zettabyte: Fills the Pacific Yottabyte: Earth size riceball David Wellman: What is Big Data? Big Data Internet Computers Early computers What is Big Data?
- 35. Big Data is not about the size of the date, it’s about the value within the data This value can be used for marketing, businesses optimisation, getting insights, improving health, security etc. What is Big Data?
- 36. Data Analytics
- 37. Why Big Data Analytics? Understand the data the company has Process data to see patterns, corrections and information that can be used to make better decisions Obtain insights that are otherwise not known
- 38. Data Analytics TRADITIONAL APPROACH Structured and Repeatable Analyses BIG DATA APPROACH Iternative and Exploratory Analyses Business users Business users Determine what questions to ask IT Structures the data to answer the question IT Delivers a platform to enable creative discovery Explores what questions could be asked
- 39. Tools for Data Analytics NoSQL databases: MongoDB, Cassandra, Hbase, Hypertable Storage: S3, Hadoop Distributed File System Servers: EC2, Google App Engine, Heroku MapReduce: Hadoop, Hive, Pig, Cascading, S4, MapR Processing: R, Yahoo! Pipes, Solr/Lucene, BigSheets,
- 40. Two Types of Data Analysis Problems Supervised Learning: Learn from data but we have labels for all the data we’ve seen so far Example: Determining Spam Emails Learn from data but we don’t have any labels Example: Grouping Emails, AlphaZero Unsupervised Learning: Learning is about discovering hidden patterns in data
- 41. Clustering One of the oldest problems in unsupervised data analysis In clustering the goal is to group data according to similarity Algorithms such as K-means are used for clustering
- 42. For each artefact found, the location to N and E from the Marker is recorded That is a Data Set Before the dig, a historian has said that three families lived in the location Clustering
- 43. Similar: close in physical distance You assign each data point to one and only one group The groups are called clusters Clustering
- 44. Clustering them is the unsupervised learning problem where you take your data and assign each data point to exactly one group, or cluster Uses unlabelled data Clustering
- 45. We may have collection data but we don’t know what to do with it We might want to explore the data without a particular end goal in mind Perhaps the data will suggest interesting avenues for further analysis In this case, we say that we're performing exploratory data analysis Clustering
- 46. Exploratory data analysis We don’t know what we are looking for Data point = colour of pixel and location of pixel Dissimilarity is the distance in colour
- 47. In some cases labelling is too expensive For example, news change every day and there are too much of them Exploratory data analysis
- 48. Using Big Data to Influence People
- 49. Alexander Nix, CEO Cambridge Analytica
- 51. Data Analysis as a Platform THEN NOW Complex tools operated by Data Analysts Chaos of data silos accross the company Real-time data analytics platform like Looker
- 52. Customer Data as a Platform Difficult to customise, lack of automated customer insights Real-time Intelligent that automatically tracks and analysis interaction with customer THEN NOW
- 53. Mapping Data as a Platform Difficult and expensive to collect data Limited in-app digital map usage Mapping platforms like Mapbox THEN NOW
- 54. Cloud Data Monitoring as a Platform Expensive and clunky point solution Lengthy implementation cycles Only used by System Administrators Cloud monitoring platforms like Datadog THEN NOW
- 55. Next Lecture L19 Network Platforms