Dark Data: Where the Future Lies
2 likes1,380 views
The document discusses the concept of dark data, emphasizing its exponential growth and the necessity for innovation to drive economic progress. It highlights the importance of improving both the production and consumption of information extracted from dark data, alongside the need for a cultural shift in how organizations manage data. Ultimately, the future of economic growth will depend on companies' ability to leverage dark data effectively and foster collaborative environments for data utilization.
![The economic case
The global economy is now [permanently] fueled by information
Innovation is becoming the merging of human creativity and
increasingly automated information extraction
Data is growing exponentially, human creativity ‘cycles’ are not
We are going to need [novel, surprising, freaky] ways of increasing
the speed of information extraction from vast and growing data
reserves
Finally, we are going to have to develop [novel, surprising, freaky]
economic ‘infrastructure’ to foster emergent designs for turning
extracted information into wealth creation faster
2](https://image.slidesharecdn.com/darkdatavkellen-140308114249-phpapp01/85/Dark-Data-Where-the-Future-Lies-2-320.jpg)
![[Population, wealth, technology, knowledge]
Hunting and foraging
Agricultural revolution
Rise of the ‘world system’
Industrial revolution
Post-information revolution
Sources: Wikipedia; various; UN Report World Population to 2300 (2004)
Diffusion accelerates technology adoption
Communications technology accelerates diffusion](https://image.slidesharecdn.com/darkdatavkellen-140308114249-phpapp01/85/Dark-Data-Where-the-Future-Lies-3-320.jpg)
![Production and consumption of information
In order to unleash dark data, we have to worry about two
problems: better production of information from dark data reserves
and better means of applying mined information to economic
activity (consumption)
Production
• We will need new purely human, purely technical and human-technical
ways of extracting information from growing reserves of data
Consumption
• We are desperately going to need [old, new] human beings with a very
different orientation to data and decision making
16](https://image.slidesharecdn.com/darkdatavkellen-140308114249-phpapp01/85/Dark-Data-Where-the-Future-Lies-16-320.jpg)
![Summary
Information is redefining humanity in ways we still don’t understand. The
future is not certain. It will be written by winners
Economic growth depends on rates of innovation. Innovation depends on
new insights which come chiefly from data
Data is growing exponentially. Human ability to process it is not. Thus,
dark data is growing exponentially too
Firms differ widely in their [in]ability to mine data for information
(production) and apply information in decisions (consumption)
A [largely, partially] semi-automated analytic discovery and refining
capability is imminent
Winners will find new ways of organizing themselves and their
ecosystems to gain advantage, speeding up timeframes
27](https://image.slidesharecdn.com/darkdatavkellen-140308114249-phpapp01/85/Dark-Data-Where-the-Future-Lies-27-320.jpg)
Dark Data: Where the Future Lies
- 1. Dark Data: Where the Future Lies Vince Kellen, Ph.D. Senior Vice Provost Analytics and Technologies University of Kentucky Vince.Kellen@uky.edu March 5, 2014 This is a living document subject to substantial revision.
- 2. The economic case The global economy is now [permanently] fueled by information Innovation is becoming the merging of human creativity and increasingly automated information extraction Data is growing exponentially, human creativity ‘cycles’ are not We are going to need [novel, surprising, freaky] ways of increasing the speed of information extraction from vast and growing data reserves Finally, we are going to have to develop [novel, surprising, freaky] economic ‘infrastructure’ to foster emergent designs for turning extracted information into wealth creation faster 2
- 3. [Population, wealth, technology, knowledge] Hunting and foraging Agricultural revolution Rise of the ‘world system’ Industrial revolution Post-information revolution Sources: Wikipedia; various; UN Report World Population to 2300 (2004) Diffusion accelerates technology adoption Communications technology accelerates diffusion
- 4. World’s technological installed capacity to store information Hilbert M, Lopez P. 2011. The World’s Technological Capacity to Store, Communicate, and Compute Information. Science. Vol. 332 no. 6025 pp. 60-65.
- 5. The total world's information, which is 1.8 zettabytes, could be stored in about four grams of DNA. Harvard stores 70 billion books using DNA. Research team stores 5.5 petabits, or 1 million gigabits, per cubic millimeter in DNA storage medium http://www.computerworld.com/s/article/9230401/Harvard_stores_70_billion_books_using_DNA Photo: Kelvin Ma for the Wall Street Journal Dr. Church keeps a vial of DNA encoded with copies of his latest book.
- 9. Moore’s law, growth in data, IT investments 9
- 10. As data grows exponentially, so does dark data Dark data 10
- 11. Rate of innovation, pace of urban life In order to sustain exponential economic growth rates, the rate of innovation must increase. Otherwise we will not have exponential growth Information flows through human culture (cities) is akin to blood flowing through a circulatory system. • Both cities and animals conserve physical energy (molecules). As both get bigger, they conserve energy However, the two systems have two fundamentally different behavior when it comes to ‘output’ • As cities get bigger, their ‘pace of life’ and economic output increases. The rate of information flow quickens • As animals get bigger, their ‘pace of life’ and metabolic output decreases. The rate of metabolic flow decreases Bettencourt, et al. (2007). Growth, innovation, scaling and the pace of life in cities. www.pnas.orgcgidoi10.1073pnas.0610172104 11
- 12. Information rules Information quickening drives economic growth, encouraging consumption (and conservation) of molecules • While fears of a Malthusian collapse have haunted economists forever, innovation and technology has enabled growth so far • Analytics can lead to productivity increases Information’s dominance in the economy appears to be causing slowing or reversing population growth rates • Rising populations drive rates of innovation and economic growth. No population growth might be worrisome • Is rising information unexpectedly going to cool down the economy? While innovation allows both growth and efficient use of resources, to sustain growth we are going to need more innovation, not less! • Increasing stores of information and means of action will be needed • DARK DATA WILL NEED TO BE MINED 12
- 13. Bits versus Atoms Physical material exhibits limits to scale. Data does not. Computing cost- effectiveness growth enables exponential data growth 13
- 14. Two overlapping, interacting systems The two systems now interact. Less molecules create more data. Information fuels economic growth, reduces population rates, improves utilization of molecules. The rate at which dark data is applied will affect all these rates Molecules Dark data Information 14
- 15. Pause. Where are we? Data and information are very important at this point in human history. How do we take advantage of these megatrends? 15
- 16. Production and consumption of information In order to unleash dark data, we have to worry about two problems: better production of information from dark data reserves and better means of applying mined information to economic activity (consumption) Production • We will need new purely human, purely technical and human-technical ways of extracting information from growing reserves of data Consumption • We are desperately going to need [old, new] human beings with a very different orientation to data and decision making 16
- 17. Production ideas Crowd-sourced and community sourced analytics. • Skills will be scarce. Have to do a better job of matching analytics to global skill sets Dark data exchanges • Can we sell our dark data to others for their exploitation? Can we buy others dark data? Dark data reserves exploration • More use of automated means of discovering data reserves and cataloging their location. Idea generation on possible value from mining Data refineries • We need to improve the rate at which data can be refined. Automated metadata extraction, automated data quality detection, semi-automated model construction, elimination of ‘one-off’ models and better reuse of partial or complete models 17
- 18. Production ideas Make widespread use of rapid data discovery tools. The ability to go from the first question to the final answer quickly matters greatly Combine purely automated technical methods of extraction and refinement with human, collaborative processes to further refine the data Develop and use refined, automated data movement tools Increase data’s ‘surface area’ through careful model design aimed at facilitating regular analysts’ use Increase data transparency, make available to many more analysts Utilize new ways obscuring data to improve privacy and security without sacrificing pattern discovery 18
- 19. Information consumption dysfunction The No. 1 impediment for improved use of dark data is human psychology. The dominant regime for managing information and power must end This regime has the following attributes: • Define goals and try to achieve them • Maximize winning, minimize losing • Unilateral control and accountability This regime causes the following dysfunctions • Information is power, thus data is hoarded, metadata formation is guarded, ‘framing of problems’ becomes a competitive battlefield • Gamers that rely on data obfuscation to make untestable claims • Reliance on personal anecdote and sample sizes of 1 • Threat-induced reactions to difficult data, causing data suppression • Cover-ups, manipulation of others, assaults on autonomy and agency See Chris Argyris and Double Loop Learning. http://en.wikipedia.org/wiki/Chris_Argyris 19
- 20. The problems with the dominant regime It’s in our nature, all humans are highly skilled at this behavior. Part of being a child and parent It is toxic to creative, high IQ talent It inhibits team performance It creates internal political theater It limits terribly the application of insights from dark data It causes awfully bad, if not tragic public spectacles 20
- 21. Needed: A new culture of information A new cultural model needs to develop, based on the following attributes • Transparency. Provide equal access to all sides of a debate • Rapid validation. Find and use tools that let all sides of a debate analyze, validate or refute insights into data • Instead of maximizing winning and minimizing losing, encourage small, fast failure. Instead of ‘punishing’ individuals, put the focus on team rewards and multi-lateral control • Instead of empowering leaders so that accountability can be overly simple, establish more intricate performance measurement systems that stabilize the enterprise, provide better feedback to many The future of exploitation of dark data will be owned by teams that can collaborate well, challenge members productively and stay together long enough to turn the data into economic wealth 21
- 22. How can you spot the person who can’t succeed? Shine light on their data and data management processes. Ask them to document and share details about their model. See if they will allow others to independently verify their results. Engage in a conversation about their model assumptions Gamers playing under the old rules will typically do the following • Defer, delay and avoid the meeting or producing the evidence • Refer to concepts like ‘we’re the experts’ or ‘we can’t explain it to non- experts’ • Change the subject • Cite powers outside of their control that limit their ability to respond • Go undercover and hide for a while You can’t succeed with a house full of gamers 22
- 23. Building expert teams takes skill and time Expert teams share a clear and common purpose and a strong mission Expert teams share mental models • Their members anticipate each other. That can communicate without the need for overt communication They are adaptive • They are self correcting. Their members compensate for each other. They reallocate functions. They engage in a cycle of prebrief-performance-debrief, giving feedback to each other. They establish and revise team goals. They differentiate between high and low priorities. They have mechanisms for anticipating and reviewing issues and problems of members. They periodically review and diagnose team effectiveness and team vitality They have clear (but not overly clear or rigid) roles and responsibilities • Members understand their roles and how they fit together They have strong team leadership • Led by someone with good leadership, not just technical skills. They have team members who believe the leader cares about them. They provide situation updates. They foster teamwork, coordination and cooperation. They self-correct first They develop a strong sense of "collective" • Trust, teamness and confidence are important. They manage conflict well. Members confront each other effectively. They trust each others intentions They optimize performance outcomes • They make fewer errors. They communicate often enough, ensuring members have the information to be able to contribute. They make better decisions The cooperate and coordinate • They identify team task work requirements. They ensure, through staffing and development, that the team possesses the right mix of competencies. They consciously integrate new members. They distribute and assign work thoughtfully. They examine and adjust the physical workspace to optimize communication and coordination
- 24. Other consumption ideas Examining decision-making within the enterprise. Find bottlenecks to faster decisions. Draw a new line separating central from local agency. Let projects proceed with light/fast approval with follow-up and audit later More rapid or time-boxed decision making. Use agile approaches. Minimally viable products. Incremental releases Reward spontaneous collaboration. Design committees, teams, units based on collaboration IQ rather than representativeness Automate more decisions, starting with the mundane or risk-free Define new roles with complementary analysis and application skills. Hire more generalists with excellent critical thinking 24
- 25. CEO imperative Designing an organization that can take advantage of dark data is very difficult. It is a CEO problem The challenge has many layers • Understanding where to strategically apply dark data findings, how to compete on analytics • Ascertaining organizational and infrastructure readiness • Establishing executive and employee incentive models that help • Managing and monitoring progress at the technical, individual, team and enterprise level • Enforcing evidence-based decision making and changing the culture • Designing the models to be used throughout the enterprise CIOs can play a strong role, but the CEO, IMHO, has to own this 25
- 26. CEO Advisory Engagement 1. Strategic possibilities • Examine the firm’s business model, value-creating activities • Identify areas where analytics and data may help, through ideation sessions 2. Dark data inventory • Document the data assets across the enterprise • Categorize and rank by quality and availability 3. Value network assessment • Evaluate the value for upstream and downstream players • Identify potential sources, uses for dark data 4. Economic estimates • Identify use cases, evaluate potential benefit and risks • Prioritize opportunities 5. Organizational development and change management • Identify culture issues, skill gaps, org structure changes, incentives, additional resources needed, communications approach, timelines and sequencing 26
- 27. Summary Information is redefining humanity in ways we still don’t understand. The future is not certain. It will be written by winners Economic growth depends on rates of innovation. Innovation depends on new insights which come chiefly from data Data is growing exponentially. Human ability to process it is not. Thus, dark data is growing exponentially too Firms differ widely in their [in]ability to mine data for information (production) and apply information in decisions (consumption) A [largely, partially] semi-automated analytic discovery and refining capability is imminent Winners will find new ways of organizing themselves and their ecosystems to gain advantage, speeding up timeframes 27
- 28. Questions? “Get your facts first, and then you can distort them as you please.” -Mark Twain 28