How Your Data Can Predict The Future

Editor's Notes

• #2: Howdy, Becky Wang. Please use #happydata or my twitter handle – I promise to write back.
• #3: When I was 10, I loved to read CYOA adventure Books.Pioneer in children's storytelling children's gamebook series - nearly 200 - written in 2nd person and reader is the protagnoistMake a series of decisions from the beginning of the book to the end - therefore, multiple storiesFor example, bear in the woods - go to page 47, page 49Death and dismemberment in either case.I wouldn't know I didn't start from the beginning
• #4: Instead - I started at the end.  I was looking for the future that fit me. I would flip through the book backwards writing down page number to see what would lead to the ending I wanted.I didn't stop there, I would actually model backwards from the outcome I wanted to the In fact, what I discovered with choose your own adventure books would soon become the center of the latest in technology trends 2 decades later -- big data.I learned, by recording my behavior, I could model out what would likely get me to the types of endings I wanted.  In fact, after reading 50 of them, I found that in the page ranges of 40s and 70s was death and/or dismemberment (CYOA was a very dark children's series) The 100s rarely ever had good outcomes - you ended up marrying a troll.  However, if you could end up in the 90s - you got to have really cool adventures like finding a Mona Lisa or killing the space vampire Darth Edward Cullen.I found, through my model, I could predict my future.
• #5: now, 20 years later, the record of my actions has been taken off my hands and is now being recorded by the technology I carry or interact with.There are, on a given web page, 104 different vendors recording data about what we click and what's the meaning behind.  And these vendors report to Google,which might link to your Visa, and channel your data to verizon if its mobile, and through shopkick if you're at bestbuywho might shares that data with ComScore who sells a subscription to your data to an agency.  Or, you might use your gym card and the gym will likely know a month before you're going to quit and offer you a discountdata today can not only help others understand our behavior, but allows a reading of our intention, desires for the future, and empowers advertisers and politicians and people interpreting the data to create opportunities to meet them.In this digital era, data is not just the domain to big companies - the digital era we live in now has sprung up many companies and organizations who are supporting and utilizing open data standards so that we can better understand outselves and by extension, make opporunities to design the lives we want to live.  
• #6: Overview of what we're going to cover.We are going to converge the interaction of people, technology, data, knowledge, and models in big data that You should take away these 7 thingsDifferent ways to think about data - a world view - and how it shapes how we use itApproaches to generating insights, predictions, and craft opportunities to meet them - including information markets, that help us use information from data to predict the futureHow big data and analytics can be used in creative agenciesEthics surrounding the use of big dataData Concepts for description, prediction, and prescriptionA list of reading from smart people, vendors, and take-home toolsA view into what the future holds for big data and how we will use it
• #7: When we talk about a world view of data, its not just about simply understanding data.  We want to create a mini-map of understand and talk about concepts, relationships between concepts, and ways of thinking about data.
• #8: When we talk about a world view of data, its not just about simply understanding data.  We want to create a mini-map of understand and talk about concepts, relationships between concepts, and ways of thinking about data.Then, we'll talk about theories of how we look at data and how greatly that affects how we use itWe'll talk about the building blogs of data and data concepts and touch a bit on the technology that supports it.We'll talk about how we attain the goals of what we want from dataWhat we should do with the big data And where we are headed
• #9: Data is everywhere.  Doesn't need to come from the No2 pencil of a 10 yo girl.Big day is everyone and continues to be everywhere - precisely because of the dependence on big data across all industries, including advertising, science, law enforcement, journalism, politics - It is sexy and mysterious - consider this a "behind the music.”Let's start from its early definitions -- Frank Moss, former director of the MIT Media Labs, describes big data as"the societal nervous system" about people.  Everytime we perform an action, we leave behind a mountain of data that provides us a treasure trove of information about people 
• #10: More recently, NYTimes has refined the defintion to be "Big Analysis . . . looking at the information in novel ways to find new patterns for prediction
• #11: A most recently, at the Strata Conference in SF, Seattle Times called it the "Transformative Power of technology and storytelling.”So big data isn't a bank of mainframes - it is now described as something that goes to the root of distinctive human capabilities - storytelling
• #12: In fact, the root of Big data has been driven by the explosion in the web.  While massive data sets have existed in the silos of IBM, Pfizer, Citibank for years – this information, including sales, interviews, media spends, R&D results - were all encoded into structured data setsIt is has been the evolution of the web in the last 15-20 years that has given consumers the voice they have nowIn the early days of the Web, now dubbed Web 1.0 - there was one-way communication in the webpage.  The difference was, anyone could have one - there were no heavy infrastructure costs to hang a shingle on the web.Web 2.0 was the movement to community - technology and habits moved towards two-way channels. In the early days that was anonymous comments on your blog asking if you'd like your hair to have a healthier sex life.  Web 3.0, the semantic web, are all the applications made possible by the encoding of the conversations in 1.0 and 2.0 to be machine readable – what Tim Berners-Lee called Linked Data.Web 4.0 is the "avatar-based virtualization" where we will ahve the reasoning and knowledge wrapped into our experiences so that if someone sends me an invite, my application will know if I'd like to say yes, what are the best routes to take to get there, what I should bring as a gift, and who else will likely be there.  And just look at the data sizes there as we support and subsequently create more data.  As more move through ever increasing amounts of data, we have now created a knowledge layer -- generated by analytics to work alongside data.   
• #13: We've used big data in several ways.We looking at Tweets, we can actually measure happiness.
• #14: At MIT - a researcher named Alex Pentland tracked 60 families living in campus quarters via sensors and softwares on smartphonesrecording movements, relationships, moods, health, calling habits and spendinghe found not only patterns in human behavior, but also in human relationshipsthere really are influencers in groups more interestingly, it could also predict where a person was likely to be given a set of conditionsthe data revealed subtle signs of mental illness, could predict the DJIA, and chart the spread of political ideasAnother research, Nicholas Christakis, at harvard, has been using mobile data to understand how all sorts of behaviors, diseases and ideas spread through social networks.  His work, termed "social contagious" is the idea that the people around us shape our behavior and can, in part, be used to predict what our own patterns are going to be.  By his calculation, obesity is contagious.  So is loneliness.  And so is happiness.
• #15: How does big data allow us to do this?There are three basic assumptions about how this happens.First, that someone out there knows something you don't.This premise allows us to invite new models, new data into the equality.This premise can be applied to the Mosaic Theory - which is the practice of putting together many different pieces of information to get the big pictureGumshoe journalists have been doing it for yearsAnd we've gotten a big boost from open data sources like Scraperwiki which aims to created structured data sets out of public documents,
• #16: The second assumption is that sentiment often precedes outcome – we’ve seen this true for economists and the consumer sentiment index, the recent studies of how Twitter data can predict the market index by up to 7 days, and the use of social media sentiment to predict box office openings in film.
• #17: Finally access to data, tools, and visualization
• #18: Data as revelationThe alterative world view sees data as revelation.The data really just reveals structures of social relationships that previous existed, but we couldn't see.And therefore, data and technology are an end to itself - an expression of what it means to be human.
• #19: How we think about data will largely impact whether we think we can predict the future    Now that its here - how do we look at it? We can see data in two ways – the firstdata as revolution.  The idea is that big data is everywhere –like the way google sees the web - a way to encode all the worlds information.Big data is, by definition, petabytes of data where infinite meanings can be read into it, but needing to know which is the intent of the creator of data.How we index the data is structuring the world in the image of the web - open.  In other words, as we understand data and its relationship to one another, we change the nature of the data, its relationship, and thereforeBig data is so new that its going to re-define itself and the people who are teached by it.
• #20: There are skeptics out there are the potentials of big data.Daniel Rasmus,an expert blogger at Fast Company, published a piece called "Why Big Data won't make you smart, pretty, or rich”and discussed the ways in which big data has some serious challenges in front of it because from his perspective, data isn't some magic potion that can combat the chaos of the existential world.For him, data is therefore revolution.
• #21: His first set of concerns mostly have to do with the person operating the models.  He states that due to our overconfidence, we will come to the wrong conclusions, or that we often look to data to confirm our points of view.  He also questions our ability to mind the model - is it working - who knows?His second set of concerns have to do with how well we use the data itself, but challenging that we don't necessarily have the right models to model prediction of the future - after all, we haven't been able to do it yet in any field. The reality of life, too, is that it defies prediction so how can we consider the possibility.  In my view, big data is something wholly new, but the way in which its been created and shaped is still within the realm of people's capabilitiesI believe we can create something wholly new - in fact, that the sum is great than its individual parts.  Like cold fusion. While I appreciate Daniel's very real and very necessary questioning of big data applicability –I do think big data can make us smart, rich, and pretty - from an individual level to a society level.  We just have to treat it with the same openness and respect with which we have created and sustained the web.I am a firm believer that people's nature is not based in scarcity or greed
• #22: How have we seen this happen in real life?Netflix knows which movies you’ll likeArt.sy knows which paintings you’ll likely buyAnd Facebook suggested plays will know where you’ll likely go
• #23: Finding  Correlations - Wal-Mart discovered in 2004, that along with flashlights, batteries and other emergency supplies, Pop-Tart sales increased before a predicted hurricane.Target – A company that lives up to its name in this next anecdote.   A recent NYTimes article about Target outlined how a father found out his daughter was pregnant based on a targeted-ad from Target.  Analyzing their customer database, they were able to find patterns in the shopping habits of women while they were pregnant and then applied that model to incoming consumer behavior.  If they found someone who matched – they sent them advertisements for pregnancy gear.Li & Fung, the large Chinese supply-chain operator, to look across its operations to identify trends. In southern China, for instance, a shortage of workers and new legislation raised labourcosts, so production moved north. before it actually happened,: The company also got advance warning of the economic crisis, and later the recovery, from retailers’ orders before these trends became apparent. Investment analysts use country information provided by Li & Fung to gain insights into macroeconomic patterns. 
• #24: Nautilus SGI supercomputer, which is being used to essentially predict the future. Researcher KalevLeetaru of the University of Illinoisfed the Nautilus SGI millions of news articles, which the computer then analyzedusing various keywords and geopolitical information to determine trends in national sentiment.The computer has a pretty impressive record.Its retrospective analysis noticed deteriorating social conditions in Libya and Egypt before the "Arab Spring" revolutions occurred, and it was able to predict Osama Bin Laden's location to within 125 miles before the Al-Qaeda leader was captured and killed. At a point in time when many governments, including the United States, didn’t know.
• #25: $1B Euro project in Zurich that is looking to combine all the models in economics, psychology, genetics, politics to see if we can look at information in novel ways to predict the future.There's another finalist for this $1B Euro project to simulate the Human Brain in all its known conditions to see if we can produce a large scale mechanism of thinking
• #26: So let's talk on a scale that you and I can work with.There are 3 approaches that I'd like to talk about today.The first approach to big data is in the modeling –And how how they are taking the algorithms from electrical engineering, physics and maths and applying them to business data.Source: http://www.analytics-magazine.org/component/content/article/79-march-april-2011/281-predictive-markets-predicting-the-rise-of-prediction-markets
• #27: Let’s talk about the application of anti-aircraft guns models against a car auction for actual prices of actual cars.At car auctions, "it is not that hard to be accurate in predicting the mean price of a bunch of similar vehicles, but our goal is to be close to the actual price for every individual vehicle.The aim is to reduce the mean absolute error (a little bit like standard deviation). : "Kalmancomes in once you have priced the vehicle and it actually sells at auction. KalmanFilters were used in the 1960s to help shoot down aeroplanesusing anti-aircraft guns. When targeting a fast-moving plane, you need to be able to predict where the plane is going. This means you need a model of the object's movement and an estimate of its state, i.e., its position, velocity and acceleration. You then keep an eye of the object by making observations to update your prediction about where the aircraft will go, calibrating the model with reality. You use that observation to update the state-of-the-world model. This leads us to better prediction models. We can be 20-25 percent better than more standards approaches in rapidly adapting to change in a market." 
• #28:  In law enforcement, previous models in effect in Philadelphia and other major cities have focused on not only locations of crimes but on the suspect themselves as they have been released – this resulted in several arrests. It wasn’t until recently that changes in the model, this time in the Santa Cruz police department, that modeled its processing after the aftershocks of earthquakes that the model improved.  For more information, touch base with Laura Hermann, who is also speaking at SXSW – for more information. 
• #29: Using genetic algorithms can be particularly useful when you have an optimisation problem with lots of different potential solutions. It can be hard to find the right one. For example, a pay TV provider might be challenged to schedule its content across multiple channels. How do you keep the most people happy at any one time? What is the optimal schedule? "If you have 100 items you want to put into 20 channels, there are billions of different combinations. It's not always obvious what you need to change to make it better.A genetic algorithm would create a population where each "individual" has a genome with a string that encodes a potential schedule over the channels. You run simulations based on assumptions such as the popularity of the show, if they liked it but there was something else on at the time, etc in order to evaluate how many people would be satisfied by a particular combination. You keep the ones which perform the best. These can then be paired to produce scheduling "offspring" or mutated by randomly swapping something out to generate the next generation. This next generation tends to contain more individuals with better solutions. The process continues until, after a few hundred or so generations, you end up with a solution that is often close to optimal. 
• #30: What’s significant here is that the breakthrough wasn’t more data – it was bringing in new models through which to look at the data. Models and algorithms are not only necessary for data analytics, but are also useful in terms of being better thinkers. In fact, Warren Buffett and his business partner Charles Munger have a series of 100 Mental Models they accredit with their success. https://www.coursera.org/modelthinking/lecture/preview
• #31: The second approach to data is using – neural networks.  What Big Data analysts can learn from neural networks - In the brain, spontaneous order appears to arise out of decentralized networks of simple units (neurons).  This is also a pattern in how people rank movies.For example, you might have a load of people who have ranked a load of movies.The rank comes in from decentralized areas and it might find that certain people will watch more comedy. A learning algorithm called the The Restricted Boltzmann Machine was developed on a very complex probabilty measure for distribution of states in a system - in this case, liking comedy.   The machine can learn the distribution of data and find patterns that occur to see what people prefer.By learning patterns of movies, the Restricted Boltzmann Machine can predict what movies people might like based on how they and others have ranked movies within a system such as Netflix. In fact, this approach was used as an application for the Netflix Prize to create a better recommendation algorithm.Source: http://www.wired.co.uk/news/archive/2012-01/06/big-data-evolution-and-missiles
• #32: In finance, the stock market functions much like an information market. In theory, the stock prices are a reflection of all the knowledge of all the independent buyers and sellers.  Now, the reality of financial markets is that this, indeed, is not the case, because the ability to control the movement of prices through scale and that the stock market now has less to do with valuing the company itself and more to do with profiting off the priceBut the principles of information markets can be applied to understanding outcomes. Predictive Markets is that a large diverse group of people [a crowd],buying and selling notional shares in ‘ideas’ can predict what other people will do, just as accurately but with far greater discrimination than traditional gold standard concept testing approaches with scientifically sampled target audiences.  Predictive Markets emerged from experimental economics before being popularised in James Surowiecki’s, best selling 2004 book, Wisdom of CrowdsSource: http://www.wired.co.uk/news/archive/2012-01/06/big-data-evolution-and-missiles
• #33: This is already being used in product development efforts at large CPGs. We can use all these models in the realm of advertising.  Indeed, most of our clients, the P&Gs, Unilevers, General Mills, Coca-Colas are indeed doing this. The cloud-based system from Inkling helps Ford Motor decide which new ideas are worth pursuing. Would you like an in-car vacuum ?Vendors such as Inkling, Brain Juicer, and Concensus PointCan answer Sample Questions likeWill Company X's same-store sales in August be higher than last year?Will the China generic market [material] price be higher than X (including VAT) on January 1, 2013?Will 2013 US corn planted acreage exceed the USDA's March estimate of 86.0 million acres?Will total shipments of product X in November be higher than last year?How much will this movie take home this weekend?
• #34: Big data now provides us evidence in decision making on a social level – not just the individual level. Which means it’s worth considering how this argument of judgement and evidence of decision-making has been evolving over time and how new discoveries, technologies, and techniques could change the issues or cause lasting change in how we go about making decisions. The remit of agencies has expanded from how a communications message would change behavior to how an experience through entertainment, news, information, and utility from the communications platforms can change behavior. In other words, we have more tools today then we have ever had before.
• #35: Andrew Weingard – self-generated behavior change by pointing out what's normal and what's valuable.  We avoid mucky waters of "behavior change" and more like prompting self change.   Teaches a class called “Social Data Revolution” and Speaks to the nature of data and behavior change. The type of behavior change resulting from simply reacting to agnostic data presented is  probably the most natural or “organic” form of behavior change. Services such as 23andMe or Counsyl provide in-depth genetic information, making affordable a full profiling of someone’s underlying makeup. Seeing that her tests suggest a high chance of heart disease due to hereditary factors, a user may naturally be inclined to exercise more and moderate high cholesterol foods. Or someone using the smartphone app out of purdueUniversity to snap photos of their meals to track calories and monitor intake. This harnesses the basic principles of the feedback loop.  http://stanford2011.wikispaces.com/Social+Data+and+Behavior+Change -
• #36: This is how clever advertising thinks it is.
• #37: I’m kidding – sort of.old model of advertising just cared about motivesIt becomes more interesting when we talk about we use data as an advertiser. Whether it is marketers trying to increase loyalty to a product or advertisers trying to drive traffic to a website, we use social data to motivate participation.So, as agencies, we generally know this – how can we extend this?
• #38: As  advertisers, we also know that consumers to be ruled by habits in 40% of their product choices.  Using data, we have the ability to better understand the unconscious processes that underpin behavior.For example, researchers have amassed so much anonymized data about human behavior that they have begun to unravel the complex behavioral and environmental factors that trigger “diseases of behavior”such as type 2 diabetes, says Alex Pentland, director of the MIT’s Human Dynamics Laboratory.  In Charles Duhigg’sbook, The Power of Habits, he lays out how habits are formed. On the most basic level, Researchers have come to understand the structure of habits — cue, routine, reward. For example, in the early 1900s, only 7 percent of Americans owned toothpaste. But Claude Hopkins, who was trying to sell Pepsodent, learned that a harmless film naturally coats teeth. In ads, he told people they needed to get rid of the film if they want to have a lovely smile. The film served as a cue for tooth-brushing. A decade after Hopkins’s ads, 65 percent of Americans owned toothpaste, which was good for oral hygiene, but not for removing film (toothpaste doesn’t actually remove it). The football coach Tony Dungy instituted a series of practice drills so that, during a play, each player would look for a specific cue and then react automatically by rote. This way he didn’t have to pause and think. Starbucks instills a series of routines that baristas can use in moments of stress, say if a customer starts screaming at them. 
• #39: Make people care and aware through social proof, authority, and facts
• #40: Support visioning, guide the goal and implementation planning of the consumer.  Encourage mindfulness and a little bit of willpower (but defusing guilt and frustration).  Model the behavior, cheer them on – build knowledge, shape the NEW habit!
• #41: The Opportune Moment

We know this with with the massive data sets made available to media agencies on syndicated and custom research studies on media habits.  We are looking for breakdowns and periods of useCreate opportunities online and offline through routine, userflow interface through web and social data
• #42: The power of digital, which a lot of digital agencies subscribe to, is that through creating platform that provide utility and entertainment and education and news, we can create these cue routine reward mechanisms, or at least, create ones that will mirror real life experiences. In fact, the best way to change behavior is to measure it.  That’s the premise of gamification, a powerful tool for motivated participation. In short, gamification refers to the use of game dynamics to influence behavior. Game mechanics such as points, levels, and leaderboards are used to stoke basic human desires such as achievement, self-expression, altruism etc. Branchout, a popular Facebook application for those seeking a more career/professional oriented social networking site (think Linked-in for Facebook) is a prime example of a company using game dynamics in order to influence users to spread the word about its services. Using progress bars and a ranking feature of various attributes of people in a user’s network, Branchout creates an addictive service where people willingly provide personal data to the application. 
• #43: I remain optimistic about the applications and implications of big data on a societal level, but there are definitely items to consider. 
• #44: 6 months ago, DanahBoyd, the well respected sociologist who has authored the seminal work on social networks and Kate Crawford, delivered a paper on the sociological impact of big data and questions to ask as we move forward.Automating Research Changes the Definition of Knowledge – like Daniel Rasmus, warns us to be congizant of how this gigantic leap in data availability will change the way we understand knowing what we know.  She warns against "just looking at the data" and "allowing the data to speak for itself" because its easy to be swayed by false correlations and triggers2. Claims to Objectivity and Accuracy are Misleading3. Bigger Data are Not Always Better Data4. Not All Data Are Equivalent – periodicity matters if I'm fusing data from 2 years ago with 5 years ago - I could have some serious false positives5. Just Because it is Accessible Doesn’t Make it Ethical – nowadays, using the mosaic theory, supposedly anonymized data can be fused with available public information online to discover people's identities - before we go down that road, we need to be clear of our intentions6. Limited Access to Big Data Creates New Digital Divide
• #45: Privacy – can individuals be identified from their data, and have they consented to storage and re-use? Ownership – who owns data collected for one purpose that is then used for a different purpose? Contribution – how to encourage users to contribute and maintain up-to-date information.
• #46: Now – here are the nuts and bolts - A framework in which to think about data and prescriptive analytics  I'd like to familiarize you with some language and terminology.  We're nearing the end of our session and I wanted to call out some basic concepts in data storage that we as analytics folks will want to make sure that the technical resources understand, but that we are familiar with.
• #47: For deeper information on database architecture, the distributed systems used to store big data, application software and concepts, I suggest you check out this video with Robert Scoble + Cloudera from 2010Its still timely
• #48: Next, I'm going to outline the basic steps of setting up an environment to play with open big data.  You don't need to store all the data yourself - there are many vendors that make it available to you.A special call out to Hilary Mason, a brilliant chief data scientist at bit.ly - a favorite of mine in big data.
• #49: The first step is data obtain and clean.  This involves sourcing information - if you're lucky enough to be able to afford syndicated research - also know that you'll have to pay additional fees for data fusing and access to data fusing.
• #50: The next step, explore, is really in the hands of the analyst. Consult those who have both mathematical and mental models of how things work
• #51: Codefiy it in a formula.
• #52: Visualization is keyThere are a number of great books including Visualize ThisAnd the old bible and testamentThe Visual Display of Quantitative DataBy the godfatherEdward Tufte
• #53: And remember, there’s always the analyst to tell the story from the visualization.
• #54: We're referenced a lot of these thinkers today -
• #55: I strongly suggest you follow, google, linked to, dialogue around some of these ideas.It continues to be a field of study - don't hesitate to participate in the discussion.
• #56: A small sampling of Tools and Platforms – for a deeper list – I’ll be posting a full bibliography for the talk with references and sources – check back with me on Monday. Don’t hesitate to reach out for more information.
• #57:  What you’ll need for yourselfUnderstand how you work – how do you value rewardsMonitoring software or data entry for the cueSoftware (understand the relationship between the cue and the routine)Visualization is keyIterate
• #58: For set up in a creative agencyBusiness Intelligence and economics analystsA rolodex of model makers – anthropologists, editors, authorsOpen DataTools – Palantir, Quid, Recorded FutureVisualize to make the caseA  development environment at your desktopCreative team to prototype
• #59: SKIP
• #60: A prediction about big data - it will come back into our hands - there will be exchanges we control about our data
• #61: 1 slide - further debate on open vs close data - the data will become free - the processing will cost time and money
• #62: 1 slide - an exchange for models - we will have places where the models we use will actually be where the value lies - and we will start an exchange to first crowdsource these ideas
• #63: 1 slide - industries to be transformed:  Real Estate, Manufacturing, and Government
• #64: 1 - last thought.  Alan C. Kay,  fellow at Apple Computer IncThe best way to predict the future is to invent it.