Big Data and The Future of Insight - Future Foundation

Editor's Notes

• #3: When did you first hear the term ‘Big Data’?Do you remember what your gut told you about it. Excitement? Hype? Or just another attention seeking tweet?
• #4: And searching for data mining in Google long before big data joined the party. So what has happened here?Essentially, big data has become the lay man's umbrella term for real world data analysis. We’ve glorified data and made it ‘sexy’. The world of data science had been branded.About Google Trends (http://www.google.com/trends) Google Trends analyzes a percentage of Google web searches to determine how many searches have been done for the terms you've entered compared to the total numberof Google searches done during that time.The numbers on the graph reflect how many searches have been done for a particular term, relative to the total number of searches done on Google over time. They don't represent absolute search volume numbers, because the data is normalized and presented on a scale from 0-100. Each point on the graph is divided by the highest point, or 100. When we don't have enough data, 0 is shown. A downward trending line means that a search term's popularity is decreasing. It doesn't mean that the absolute, or total, number of searches for that term is decreasing. Trends data is relative, not absolute.Just because two regions show the same number for a particular search term doesn't mean that their absolute, or total, search volumes are the same. Data from two regions with significant differences in search volumes can be compared equally because the data has been normalized by the total searches from each region.Source : About Google Trends, 2014.
• #5: There are some fundamental differences in the data we’re able to collect now and how it can be used. There is real BIG data out there and it typically consists of the three V’sVariety.Volume.Velocity.And the allure of big data is that with all this new information we have a opportunity to draw Value.Let’s start with velocity:
• #6: The options here are nearly endless. Your geolocation through mapping services, or even walking past wifi connections. With imminent iBeacon technology your location can be tracked within feet for use in stores. Companies like Euclid are already envisioning a world where we can tell exactly what store displays you are looking at.Everything you search for online.Your credit card – what you’ve been buying and when. Your cinnamon latte morning coffeeEverything you say on Social networks... Every retweet, like and comment.Your medical records, prescriptions, aliments, genome.Everything you buy with a store card. Everything you buy online. Everything you almost buy online then decide not to at the last minute.The prices of flights, the location of flights.Every email we send and receive Where we drive, and even how we driveOur compatible traits for love... So mainly location when it comes to TinderThey way we move, the way we sit.Literature can be datafied.How much energy we use, and when. And even the things we own in our homes. The internet of things is expected to revolutionise our ability to collect personal data, allowing machines to interact without human intervention. Google’s acquisition of Nest is an obvious indicator of where the top dogs think this is heading.But... (and there’s always a but) with Big data we have messy data. It’s approximated that at least 80% of the data we have produced is unstructured. It doesn’t always fit neatly into rows and columns which doesn’t make things easy. But we want to be able to look at the unstructured data out there. This is where great opportunities for consumer insight are held. This is where REAL is. Luckily, we can profile and normalise this data and best thing about this is that our ability to normalise with machine learning algorithms is improving as we collect even more data.
• #7: One area however that does benefit from increased variety of variables is our predictive power with big data. The company I previously mentioned, EDITD, is able to make 6-8 week forecasts for popularity of different fashion trends. It’s analysis like this that is able to help Asos push up sales 37% in the last quarter of 2013. We’re also able to predict crime hot spots based on variables like past arrest patterns, paydays, sporting events, rainfall and holidays. From this stations can deploy officers there in advance.http://www.sciencedaily.com/releases/2014/01/140110142212.htmFarsite Forecast were even able to predict the Oscar winners for 2014, based on variables like nominees previous winning performances, nominations, results from other awards, as well as industry gossip and, too lesser extent for poor Leonardo DiCaprio, social media buzz.In all these examples you’ll notice that the variable inputs are things being measured today that have a real and expected effect on the future. And generally, as long as some of tomorrow is contained in a measured part of today we can make predictions. This is amazing stuff but, as with all forecasts and predictive models it's never 100%. 5 of 6 Oscars were predicted correctly in 2013. And Amazon, well they keep predicting that I would like to purchase items I already own, but just didn't buy from them. Based on the data they have, they’re doing a perfect job here.
• #26: Photo © Marc Smith 2013