Big Data : Risks and Opportunities
19 likes5,421 views
This document discusses big data, including opportunities and risks. It covers big data technologies, the big data market, opportunities and risks related to capital trends, and issues around algorithmic accountability and privacy. The document contains several sections that describe topics like the Internet of Things, Hadoop, analytics approaches for static versus streaming data, big data challenges, and deep learning. It also includes examples of big data use cases and discusses hype cycles, adoption curves, and strategies for big data adoption.
![Financial Model Quizzes
44
Big Data
Technology
Provider
Big Data
Solution
Integration
Big Data
As A Service
Fixed cost
BEP
Fixed cost
BEP
Fixed cost
BEPsales
cost
cost
cost
salessales
A B C
[ ] [ ] [ ]
*BEP : Breakeven Point](https://image.slidesharecdn.com/bigdatav1-150605010536-lva1-app6892/85/Big-Data-Risks-and-Opportunities-44-320.jpg)
Big Data : Risks and Opportunities
- 1. Big Data Risks and Opportunities Kenny Huang, Ph.D. 黃勝雄博士 Executive Council Member, APNIC 亞太網路資訊中心董事 huangksh@gmail.com 2015.06.09
- 2. Agenda 2 1 2 3 4 Big Data Technologies Big Data Market Opportunities, Risks,& Capital Trends Algorithmic Accountability & Privacy
- 3. Internet of Things Definition 3 The Internet of Things (IoT) is the network of physical objects or "things" embedded with electronics, software, sensors and connectivity to enable it to achieve greater value and service by exchanging data with the manufacturer, operator and/or other connected devices. Each thing is uniquely identifiable through its embedded computing system but is able to interoperate within the existing Internet infrastructure. IoT = Devices (RFID tags, Sensors, ..) + Networks + Services + Data + Analytics “In the world of IoT, even cows will be connected” Source : The Economist 2010
- 4. IoT Size & Potential 4 Source : CISCO Urban Planning Smart Cities Sustainable Environments Healthcare Emergency Response Waste Management Intelligent Shopping Smart Product Management Smart Meters Smart Homes Smart Automobiles Smart Agriculture (cows) Smart Grid Intelligent Business Decisions IoT Potential Applications
- 5. Examples of IoT 5 Google Glass Weight Scale Sense Mother Jawbone Up SkyBell Light bulb Nest Thermostat Belkin Wemo Firefox iKettle
- 6. IoT Architectural Design 6 Question : How to build systems that work well ? Breaking them into tractable components. “Modularity based on abstraction is the way things get done.” – Liskov If you can’t manage, evolve, or understand a system, probably don’t have the right abstraction. Cloud Data Centre Core Network Access Network Things Network Web server hosting IP/MPLS Ethernet; Mobile; WiFi RFID; NFC; Bluetooth
- 7. Energy Administration Architecture 7 Application servers; Database servers IP/MPLS WiFi 3G Asset managementReliability improvement Smart metering
- 8. Water Administration Architecture 8 Application servers; Database servers IP/MPLS WiFi 3G Field devices Smart metering Quality monitoring and Water control
- 9. Smart City Architecture 9 Application servers; Database servers IP/MPLS WiFi 3G Public lighting Intelligent transportation Public safety management
- 10. IoT Technologies 10 detect anomalies analyze a mix of structured, semi-structured and unstructured data Sensor Analytics Stream Analytics Big Data Analytics Real-time Analytics Machine Learning Analytics Statistical Analytics enables business users to get up-to-the-minute data by directly accessing OS enables developers to combine streams of data with historic records to derive business insights find patterns and identify trends building of the predictive models by using machine learning techniques
- 11. Big Data 11 ASA 2005 – Roger Magoulas uses the term “Big Data” McKinsey Global Institute Big Data: The next frontier for innovation, Competition, and productivity. White House Big Data Initiative : $200 Million in New R&D Investment on Big Data for scientific discovery, environmental and biomedical research, education, and national security The New Oil As far back as 2006, market researcher Cliver Humby declared data “the new oil.” Just as oil once fired dreams a century or more ago, data is today driving a vision of economic and technical innovation. If “crude” data can be extracted, refined, and piped to where it can impact decisions in real time, its value will soar. International Year of Statistics - 2013 McKinsey Global Institute, May 2011 Press Release. White House of OSTP. March 29 , 2012 CISCO ISBG, June 2012
- 12. Big Data : Size 12
- 13. The 3Vs of Big Data 13 90% of the data in the world today was created within the last two years People to People People to Machine Machine to Machine 2.9 emails sent every second 20 hours of video uploaded every minute 50 million tweets per day Volume Variety Velocity
- 14. Big Data Ecosystem 14 Generation Data Types Structured (relational) Unstructured (adhoc) Data Classes Human Machine Data Velocity Batch Streaming Data Class Types Data Mgmt. & Storage Store Secure Access Network Engines Hadoop MapReduce Apache Tools Cloudera/IBM/EMC Visualization Prepare Data For Analytics ETIL / Data Integration Workflow Scheduler System Tools Data Analytics Algorithmics automation In Real Time Business Analytics Visualization Interoperate with SQL -RDBMs BI/EDW Business Analysis Decision Support Just in Time Business Model Business User Market Penetration Enhancement Cash Flow/ROI Operational IT Store Access Prepare Analytics Analyze Visualize Analyze Business Usage Source : Sybase
- 15. 15 Analytics: Static Data vs. Streaming Data Static Data Streaming Data Multiple Passes Single Pass Persistent Inherently Temporal Offline Analytics Online as well as Offline Analytics Analytics Based on All the Data Analytics Based on a Subset of Data Only the current state is relevant Consideration of the order of the input Relatively low update rate Potentially high update rate Little or no time requirements Real-time requirements Assumes exact data Assumes outdated/inaccurate data Plannable query processing Variable data arrival and data characteristics DBMS (Database Management System) DSMS (Data Stream Management System) Persistent relational data Volatile transient data streams Random access Sequential access One-time queries Continuous queries Unlimited secondary storage Limited main memory Only the current state is relevant Consideration of the order of the input Relatively low update rate Potentially high update rate Little or no time requirements Real-time requirements Assumes exact data Assume outdated / inaccurate data Standing queries Ad-hoc queries
- 16. Big Data Challenges & Data Life of Cycle 16 Input Raw Data Collection Cleaning, Validation and Serialization Transformation & Augmentation Output Interpretation & Presentation Mining & Analytics DB Storage & Management Sensor data brings numerous challenges with it in the context of data collection, storage and processing. This is because sensor data processing often requires efficient in-network and real-time data stream processing from massive volumes of possibly uncertain data from various sources. The data generated from these sensors arrives in the form of streams. At every phase of the big data life cycle, there are research issues along each steps To handle these streaming sensor data model-based techniques are employed, such as : statistical, signal processing, regression-based, machine learning, probabilistic, time series.
- 17. 17
- 18. Example of Model-based Technique : Kalman Filter 18 Probabilistic Models: In sensor data cleaning, inferring sensor values is perhaps the most import task, since systems can then detect and clean dirty sensor values by comparing raw sensor values with the corresponding inferred sensor values. The Kalman filter is perhaps on of the most common probabilistic models to compute inferred values corresponding to raw sensor values.
- 19. 19 In the sliding window model, only the recent past is the objective concern of stream processing. The fundamental sliding windows are of fixed size, which are similar to first-in, first-out data structure. The input is still a stream of data values or elements. A data value arrives at each time instant; it later expires after a number of time stamps equal to the window size n The current window at any time instant is the set of data elements that have not yet expired. The Sliding Window Model
- 20. Hadoop 20 Processing Platform for Big Data Processing Using the “MapReduce” processing technique MapReduce is the processing part of Hadoop HDFS is the data part of Hadoop Attributes Highly scalable Commodity HW-based Open source: low cost Batch processing centric MapReduce HDFS Machine Hive HBase Mahout Pig Oozie Flume Scoop Projects Set of open source projects
- 21. Map->Reduce and HDFS Architecture 21 TaskTracker DataNode Machine JobTracker NameNode TaskTracker DataNode Machine TaskTracker DataNode Machine JobTracker keeps track of jobs being run NameNode keeps information on data location Master Slave Slave Slave
- 22. 22 1. The network is reliable 2. Latency is zero 3. Bandwidth is infinite 4. The network is secure 5. Topology doesn’t change 6. There is one administrator 7. Transport cost is zero 8. The network is homogeneous The Eight Fallacies of Distributed Computing Source: Peter Deutsch
- 23. 23 Source : Ray Kurzweil 10−5 1 105 1010 1015 1020 1025 1030 1035 1040 1045 1050 1055 1060 1900 1920 1940 1960 1980 2000 2020 2040 2060 2080 2100 Year CalculationsperSecondper$1,000 Exponential Growth of Computing Logarithmic Plot By 2020s, computers have the same power as the human brain
- 24. Deep Learning 24 Iterative Algorithm Learning at different levels of abstraction Non-linear transforms Typically neural nets Genetic programming Neural networks Quantum computers Wisdom of Crowds Examples of Iterative Algorithm What is Deep Learning
- 25. Google First Quantum Computer 25 “We actually think quantum machine learning may provide the most creative problem- solving process under the known laws of physics.” – Google Blog
- 26. 26 Ginni Rometty, CEO of IBM “In the future, every decision that mankind makes is going to be informed by a cognitive system like Watson.” Deep Learning Application Areas
- 27. 27 1 2 3 4 Big Data Technologies Big Data Market Opportunities, Risks,& Capital Trends Algorithmic Accountability & Privacy
- 28. Rainmaker I Prophet 28 Viktor Mayer-Schönberger Eric Schmidt “we now uncover as much data in 48 hours – 1.8 zettabytes – as humans gathered from the dawn of civilization to the year 2003”
- 29. Rainmaker II Knowledge Marketer 29 Big Data Hype Circle Big Data Maturity Model
- 30. Hype Create Perception with Correlation 30 source: Google * correlation doesn’t prove causation
- 33. Problem Whose Problem ? 33 Avoid Fallacy of Irrelevancy Questions : 1 You want to solve IT giants' (Google/FB) problems ? 2 You want to solve future problems with today’s technologies and price ? 3 Forging illusive needs immediately to leverage technology trends ? “Excel is very powerful. The fact is that programmers generally don't realize this.” (Jay, LinkedIn)
- 34. 34 1 2 3 4 Big Data Technologies Big Data Market Opportunities, Risks,& Capital Trends Algorithmic Accountability & Privacy
- 35. 35 Create new service offerings Satisfy customers Provide contextual relevance Information based differentiation Sell raw information Provide benchmarking Deliver analysis and insights Information based brokering Foster marketplaces Drive deal making Enable advertising Information based delivery networks Big Data Business Model (HBR, 2012) What happened ? (Reporting) How and why did it happen? (Modeling experimental design) What is happening ? (Alert) What’s the next best action? (recommendation) What will happen ? (Extrapolation) What’s the best/worst than can happen? (prediction, optimization) Information Insight Past Present Future Questions Addressed by Data Analytics (Harris & Morrison)
- 36. Target used data mining to predict buying habits of customer going through major life events Target was able to identify 25 products that when analyzed together helped determine a “pregnancy prediction” score Sent baby-related promotions to women based score Case Studies 36 Outcome Sales of Target’s Mom and Baby products sharply increased soon after advertising campaigns Privacy concerns: Target had to adjust how it communicated the new promotions General Electric using Big Data to optimize the service contracts & maintenance. Netflix used Big Data to predict if a TV show will be successful – “House of Cards” series, Director & promotion. LinkedIn used Big Data to develop “People You May Know” products - 30% higher click-thru-rates
- 37. 37 Buying opportunity #2 VisibilityinMedia Time Technology Tigger Peak of Inflated Expectations Trough of Disillusionment Slope of Enlightenment Plateau of Productivity Education Buying opportunity #1 Danger Zone Reality Check RealValue 2 years 5 years Source : Gartner; Dr. Kenny Huang Revised Gartner Hype Cycle
- 38. 38 Buying opportunity #2 VisibilityinMedia Time Buying opportunity #1 Danger Zone 2 years 5 years Source : Dr. Kenny Huang Revised Time Innovators 2.5% Early Majority 34% Late Majority 34% Laggards 16% Early Adapters 13.5% Chasm Technology Tigger Peak of Inflated Expectations Trough of Disillusionment Slope of Enlightenment Plateau of Productivity Hype Cycle and Technology Adoption Cycle Plotted Together
- 39. Big Data Visibility and Demand 39 “Big Data” Google Trends @2015.06.04 US TW 26% piloting 11% may invest in 1 year 7% may invest in 2 years 2015 Gartner research on adoption of Hadoop Technology ”Future demand for Hadoop looks fairly anemic over at least the next 24 months“. Merv Adrian, Gartner Research. (2015)
- 40. Big Data Buying Opportunity for Taiwan 40 VisibilityinMedia Time Danger Zone Source : Gartner; Dr. Kenny Huang Revised Big Data Visibility as of June 2015 Next Buying Opportunity 2017 < Time * * Ref revised Hype cycle diagram, Google trends 2015, Gartner research 2015
- 41. 41 Risk (Standard Deviation) ExpectedReturn Government Bonds Corporate Bonds Common Stock Real Estate Futures Big Data ? New Innovation !! Source : Dr. Kenny Huang Illustration Risk Acceptance Risk-Return Tradeoff
- 42. Investment Risking Model 42 Risk Acceptance Risk Mitigation Risk Avoidance Startup; Series A Due Diligence Change Investment Objects Risk Acceptance Risk Mitigation Risk Avoidance Don’t Use Taxpayers’ Money Pilot Projects; Research Change Technology Policy Business Entity Government Institution
- 43. Big Data Adoption Strategy 43 Focus on your own business Adopt and separate; or Adopt keep internal; or Attack back and disrupt the disruption Focus on your own business Attack back and disrupt the disruption; or Embrace the innovation and scale it up Source : MIT Sloan Motivation To Response AbilityToResponse Low High LowHigh
- 44. Financial Model Quizzes 44 Big Data Technology Provider Big Data Solution Integration Big Data As A Service Fixed cost BEP Fixed cost BEP Fixed cost BEPsales cost cost cost salessales A B C [ ] [ ] [ ] *BEP : Breakeven Point
- 45. Global Capital Market Trends 45 Source : Designed by Dr. Kenny Huang MM USD
- 46. 46 IPOs and Private Financing Deals in the Tech Sector since 2000 (United States) Source: PwC Source: Techcrunch If there is a bubble, investors would recover their investment and perhaps walk away with positive return, the biggest losers for sure would be the employees and founders. Game Rule : You Pick The Valuation, I Pick The Terms
- 47. 47
- 48. 48 1 2 3 4 Big Data Technologies Big Data Market Opportunities, Risks,& Capital Trends Algorithmic Accountability & Privacy
- 49. Algorithms Rule The World 49 We should interrogate the architecture of cyberspace as we interrogate the code of Congress. - Lawrence Lessig, Code is Law, 2000
- 50. Algorithmic Accountability 50 Algorithms Are Everywhere Algorithmic Accountability How can we characterize the bias or power of an algorithm? When might algorithms be wronging us, or making consequential decisions? What role should be involved in holding algorithmic power to account ? Algorithmic Confusing Algorithms are not transparent Technical complexity is a barrier (Nick Diakopoulos)
- 51. Algorithmic Power : Decisions 51 3 2 1Prioritization Classification Association Filtering
- 52. 52 Input Output Input / Output of An Algorithm Algorithm Input OutputAlgorithm WSJ Price Discrimination Do different people pay different prices depending on their geography or browser history ? Yes Source: WSJ, Dec 2012 Staples.com
- 53. e 53 Transparency Voluntary incentives for self-disclosure about algorithms Trade secrets Gaming / manipulation Goodhart’s Law: “ When a measure becomes a target, it ceases to be a good measure.” Cognitive complexity Transparency information needs to be accessible and understandable
- 54. sdfdsf 54 Other Stories from Algorithms Discriminatory / Unfair Mistake that denies a service Censorship Breaks law or social norm False Prediction sdfdsf Next Step Teaching algorithmic accountability It will be messy and hard Legal issues Computer Fraud and Abuse Act Ethical implications of publishing more information Transparency policy What factors to expose, frequency, format of disclosure
- 55. Critical Considerations for Big Data Practices 55 Customers will want to know that you are collecting data why and what you are collecting that their confidentiality is preserved that their data is accessible Privacy Customers will want an unique URL where they can see what you’ve collected to know what sensors you are using that an API is interrogating the data Transparency Customers will expect to be the owner of the data & be the copyright holder. To decide who they allow access to (might not even be you) Ownership
- 56. Concern with Big Data Practices 56 Source : Whitehouse Big Data Review
- 58. Massive Surveillance vs. Human Rights 58 Article 12: No one shall be subjected to arbitrary interference with his privacy, family, home or correspondence, nor to attacks upon his honour and reputation.
- 59. 59 Source : appledaily, 2015.04.25Source : The Truman Show
- 60. 60 USA Today 2015.06.03 The New York Times 2015.06.03
- 61. 61
- 62. 62 ".... The big question is this: how do we design systems that make use of our data collectively to benefit society as a whole, while at the same time protecting people individual? Or..... how do we find a "Nash equilibrium" for data collection..........."