Big Data: Hadoop Map / Reduce sur Windows et Windows Azure

Notes de l'éditeur

• #12: Much of that data is in traditional databases and data warehouses, and those kinds of data – product orders, new customer records, etc. – grow at a more linear rate. What’s driving the exponential growth is often less structured data. This is in the form of log files, images, video, sensor or device output, and public data. Much of this data never makes it into a relational database, and the technologies used to process this kind of data go by the names “Data Intensive Scalable Computing”, “Large Scale Data Analytics”, or the more trendy term “Big Data”.In the past, much of this kind of data was simply archived, or even thrown out after a relatively short time. But now, it’s economically feasible to not only store, but process and gain insight from this kind of data.Increasing Data VolumesAccording to Gartner, the current annual growth of WW information volume is 59% and continues to rise.  This data explosion is being driven by the full range of traditional and non-traditional sources like sensors, devices, bots and crawlers.  According to an IDC report, the volume of digital records is forecasted to hit 1.2M Zetabytes (1021 bytes) this year – and predicted to grow 44x over the next decade.   Increasing Data ComplexityHistorically, the data has been largely structured in type; however the real growth is coming from non-structured data.  The success of search engine providers and e-retailers who unlocked the value of click-stream data has debunked the myth that 80% of unstructured data has no value.  The requirement to analyze and mine unstructured and structured data together is increasingly on the agenda for many enterprises today.  Increasing Analysis ComplexityIncreasing analysis complexity comes hand in hand with Increased Data Complexity.  For example, image processing for facial recognition, search engine classification of videos and use of click-stream data for behavioral analytics.  Models for transactional data are mature and well understood and have driven the value behind the last two decades of Data Warehousing and BI.  The models governing complex data and behavioral interactions are in their infancy.   Increasing Demand for New InsightsDespite the growth in useful information, we also know that the number of users in an organization who have access to Business Intelligence tools and capabilities is less than 20%.  This fact combined with the real time nature of data is given rise to demand for real-time and predictive analytics by an increasingly larger user population.  Changing EconomicsCloud computing and commodity hardware have radically reduced the acquisition cost of computational and storage capacity. The decreasing cost of distributed compute, memory and storage is fundamentally changing the economics of data processing.  The rise of the Data Warehouse appliance has more than halved the cost per terabyte of EDW systems over the past 3yrs.  Cloud Data Warehouse systems hold the promise of a 10x TCO advantage over traditional on-premises systems. Emerging TechnologiesEasy to scale commodity hardware is being complimented by new distributed parallel processing frameworks and tools, which combined are providing a rich and inexpensive platform for tackling massive data processing tasks.  MapReduce style programming models are enabling new types of analytics that were not practical or possible with SQL.    The maturity and commercialization of several open source software products has paved the way for their inclusion in product evaluations for larger scale software projects. The cloud model puts another layer of abstraction between the user and the infrastructure and application platform layer further reducing barriers to adoption of technologies like Hadoop.
• #13: http://nosql.mypopescu.com/post/9621746531/a-definition-of-big-dataKEY POINT: Hadoop is part of the solution -
• #14: Hadoop is an AND, not an OR. But it requires a certain philosophy that MSFT has not historically embraced. A key benefit of Hadoop is the large, vibrant open source community around it. To succeed, Microsoft needs to not only acknowledge but thrive in this community.(1/3 des employé font de l'Haddop)(1/3 des employés font de l'Hadoop)Historiquement Google est a l'initiative de ce type de traitement
• #18: BIG self service BIBillions+ of data itemsUnstructured, semi-structured, log dataReal-time feedsNew analysis types leveraging large server clusters Leverage the Hadoop ecosystem and ride its momentumIW centric designGive business users direct access to the Big Data storeDeliver IW-centric experiences optimized for unstructured and semi-structured queriesCreate, enrich, visualize and share big data sets through fun and immersive experiencesDo it all in the tool they already use - ExcelIncrease the number of questions, reduce the cost of exploratory mining to zeroLeverage new class of analytics and visualizationEnable new types of questions with new types of data and visualizationsLeverage analysis of text, sentiment, clickstream, time windows, classification, clusteringVisualize big data in impactful ways: tag clouds, graphs, timelines, tree maps, etc. Natural extension of our BI platformMaintain a consistent semantic model, consistent expression languageProvide an iterative, experimental, business-driven workflow from the desktop to the Big Data clusterBuild on existing IW skills with the Microsoft BI platform (Excel, PowerPivot, Crescent)Optimized for cloudIntegrate with Azure DataMarket to connect to Bing and other public data sourcesHost big data sets on Azure , integrated with MyDataLeverage Isotope to run analytics clusters
• #20: Isotope is the all-up effort around Microsoft and Hadoop. It includes several components:A full distribution of Apache Hadoop that runs on standard windows hardware.A full version of Apache Hadoop that runs on the Azure cloudConnectors from Hadoop (any Hadoop, not just Microsoft’s) to Microsoft’s key products – SQL, Excel, PDW, etc.Jscript shell for live scripting of Hadoop from the browserAdmin, monitoring, and authoring tools to make Microsoft Hadoop best-in-class
• #23: privatestaticvoidCountTopNWords() { string sourceContent = null;Console.WriteLine("Reading source file...");using (Stream sourceStream = WebRequest.Create(context.sourceFileUrl) .GetResponse().GetResponseStream()) { using (var reader = new StreamReader(sourceStream)) {sourceContent = reader.ReadToEnd(); } } string[] words = Regex.Split(sourceContent.ToLower(), "[^a-zàäâéèëêïîöôùüûç]"); var result = words .Where(x => x.Length > 5) .GroupBy(x => x) .Select(x => new {word = x.Key, nb = x.Count()}) .OrderByDescending(x => x.nb) .Take(context.NbOfWords);Console.OutputEncoding = Encoding.UTF8;foreach(var r in result) {Console.WriteLine("{0}\\t\\t{1}", r.word, r.nb); } }