Hadoop in the Cloud: Real World Lessons from Enterprise Customers

Editor's Notes

• #10: Challenge Manage sites used for dispensing liquefied natural gas (clean fuel for commercial customers who do heavy-duty road transportation) Built LNG refueling stations across US interstate highway Stations are unmanned so they built 24x7 remote management and monitoring to track diagnostics of each station for maintenance or tuning Built internet-connected sensors embedded in 350 dispenser sites worldwide generating tens of thousands data points per second Temperature, pressure, vibration, etc. Data needs outgrew company’s internal datacenter and data warehouse Solution Chose Azure HDInsight, Data Factory, SQL Database Dashboards used to detect anomalies for proactive maintenance Changes in performance of the components Energy consumption of components  Component downtime and reliability  Future: Goal is to expand program to hundreds of thousands of dispensers How They Did It Collect data from internet-collected sensors Tens of thousands data points per second Interpolate time-series prior to analysis Stored raw sensor data in Blobs every 5 minutes Use Hadoop to execute scripts and Data Factory to orchestrate Hive and Pig scripts orchestrated by Data Factory Data resulting from scripts loaded in SQL Database Queries detect site anomalies to indicate maintenance/tuning Produced dashboards with role-based reporting Azure Machine Learning , SSRS, Power BI for O365 Provide users with customizable interface View current and historical data (day-to-day operations, asset performance over time, etc.) Leveraged Azure Mobile Notification Hub for real-time notifications, alarms, or important events
• #11: JustGiving wanted to identify what was personal and relevant to people and what they cared about, so that they could suggest further causes that may inspire continual involvement. However, with 22 million customers this meant storing and processing huge amounts of data that their existing infrastructure simply couldn’t support. HDInsight, provided the scalable, ‘on-demand’ processing and analysis ability to assist JustGiving with its goal to constantly evolve the personalised experiences it provides to customers. JustGiving is a global online social platform for giving. It's a financial service (not a charity) that lets you "raise money for a cause you care about" through your network of friends. JustGiving's goal is to become "Facebook of Giving" JG preferred not to refer to Facebook as a way of describing themselves – they are using the term “social giving” and like to refer to themselves as a “tech for good” company. i.e. harness the effect to make charity a group activity which isn't just a onetime event but rather something you stay in touch with on regular basis.   More details on charity goals in a blog post from JustGiving: http://blog.justgiving.com/creating-smart-targets-to-help-fundraisers-raise-more/ Technical Details:   Workflow: There is one a set of  daily HDInsight job that uses the data coming through SQL Server to build out the social graph and provides activity recommendations to the user. The input data is 20-30 GB/job but the output is in 'hundreds' of GBs as relationships are de-normalized/expanded. Azure Table Storage is used to serve 'News Feed' to users. The data in Table store come from two main sources: Real time activity feeds/events coming in from Azure Service Bus. (~50 events/second) Activity recommendation coming out of the daily HDInsight job There are several MR processes to create the graph; once that is done, further jobs create the denormalised activity feeds for all users. M/R jobs that do all the graph building.
• #12: Wind turbines – used in windmills to turn wind into power   These things are phenomenally complex devices They emit a ton of telemetry Every 25 milliseconds they emit 10 interesting datapoints 100x of wind mills (turbines) per windfarm They are managing about a hundred windfarms through their customers Their customers are the power companies SO yes this is a “IOT scenario” Couple of challenges How am I going to collect all this information – windfarms around the globe, some in remote areas, how to collect How to I store it cheaply – lot of data – honestly no single row interesting – only interesting in aggregate – but because there’s ns much of it putting in DW it is cost prohibitive Instead they landed data in cloud from around the world Cloud allowed storing it cheaply – just store no processing Then bring processing on top of it to distill and refine the data into something more powerful They cooked data down - refined, aggregate, duplicate detection Now they provided interesting metrics to power companies But since they had the source data around they can ask lots of interesting questions on *all* the data since they started collecting it Moved beyond charts and graphs bout how the windfarms were performing Now able to mash that together with maintenance data  To be able to say what are some of the things correlated with my turbines failing These are very expensive pieces of machinery so if we can find things that are going awry *before* the whole thing breaks its usually much cheaper Now one of the interesting things they found was if they were more observant about dust conditions .. they were saving million dollar generators by replacing four dollar filters more often And so this is a interesting example of predictive maintenance
• #17: Choose VM based on core size and memory
• #27: Choose VM based on core size and memory
• #29: Choose VM based on core size and memory
• #37: Finding the right tools to design and tune your big data queries can be difficult. Data Lake makes it easy through deep integration with Visual Studio, so that you can use familiar tools to run, debug, and tune your code. Visualizations of your U-SQL, Hive, and Storm jobs let you see how your code runs at scale and identify performance bottlenecks and cost optimizations, making it easier to tune your queries.