Lessons from handling up to 26 Billion transactions a day - The Weather Company Platform Story
2 likes611 views
The document outlines the Weather Company's platform and its capabilities to handle up to 26 billion transactions daily, emphasizing the importance of data-driven infrastructure and cloud computing in modernizing operations. It details the evolution of data management techniques, challenges in leveraging data for business insights, and the associated costs of building and maintaining such platforms. Additionally, it highlights lessons learned and insights into market trends while providing an overview of key sessions related to weather data and its applications in various industries.
Lessons from handling up to 26 Billion transactions a day - The Weather Company Platform Story
- 1. The Weather Company's Platform Story: Lessons from Handling Up to 26 Billion Transactions Per Day Mon, 24-Oct 11:00 AM - 11:45 AM
- 2. Please note IBM’s statements regardingits plans, directions,and intent are subject to changeor withdrawal without notice and at IBM’s sole discretion. Information regarding potential future products is intended to outline our general productdirection andit should not be relied on in making a purchasing decision. The information mentionedregardingpotential futureproducts is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential futureproducts may not be incorporatedinto any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion. Performance is based on measurements andprojections using standard IBM benchmarks in a controlled environment. The actual throughput or performancethatany user will experience will vary depending uponmany factors, including considerations suchas the amount of multiprogramming in the user’s job stream, the I/O configuration, thestorageconfiguration,and theworkload processed. Therefore, no assurancecan be giventhat an individual user will achieve results similar to those stated here. 10/27/16World ofWatson 20162
- 3. Your speakers Derek Baron Program Director Platform as a Service Landon Williams SVP Technology Products & Architecture
- 4. 10/27/16World ofWatson 20164 Data from connected cars are an important factor in the determination of insurance premium pricing
- 5. 10/27/16World ofWatson 20165 The problem: 25 GB / hour / connected car By 2026 that’s one billion GB / year Source: http://ww2.cfo.com/big-data-technology/2016/04/my-car-my-data-connected-car/
- 6. 62 miles separate us from space The Weather Company collects and connects the dots… … powering Billions of personalized forecasts a day 6
- 7. Hundreds of different types of data, terabytes a day coming in 162 forecast models serve as inputs to our forecast > 200,000 personal weather stations Atmospheric data from 50,000 flights per day 15 Million pressure reading devices providing readings 20 Million devices provide Location data The SUN Platform is supporting data for IoT, Analytics, and Cognitive computing 7
- 8. 2012: The big reboot to embrace cloud and transform culture #2 #5 The InformationWeek Elite 100 tracks the IT practices of the nation's most innovative IT organizations Before After • 13 maxed-out data centers • Aging apps running on one-of-everything infrastructure • Cloud-based, Cloud-agnostic • Data-driven infrastructure • API-based delivery of data • 2.2 Million weather data points 4 times per hour (2012) • 2.2 Billion weather data points 15 times per hour (2014) • 60-70% of tech effort in maintenance/ops • 20-25% of tech effort in maintenance/ops 8
- 9. 9 Transferred 1.6 PB forecast, map, and digital content 141 Billion API calls 4 PB of video in a single day SUN throughput during hurricane Matthew
- 10. Yet only 15% of organizations have the capability to leverage data and advanced analytics across their organization. HBR Insight Economy Study The advent of cognitive computing The re-invention of the world in code A world awash with data What’s changed in the world today
- 11. Source: The Battle Is For The Customer Interface, Tom Goodwin, Havas Media World’s largest transportation company… owns no vehicles World’s biggest media company… creates no content World’s most valuable retailer… has no inventory World’s largest accommodation provider… owns no real estate World’s largest video conference company… has no telco infrastructure New business models disrupt legacy players
- 12. New business models create entirely new value streams 12 Source: IBM and http://www.slideshare.net/andreasc/vision-mobile-iot-megatrends-iot-accelerate-berlin-v003
- 13. Nest harnesses the power of exogenous data 13 Source: http://www.slideshare.net/andreasc/vision-mobile-iot-megatrends-iot-accelerate-berlin-v003
- 14. 14
- 15. Lessons / cultural change 15 • lessons learned over the last few years (eg cache strategies to lower latency) • technical choices and evolution (eg hadoop to spark) • team structure and cultural changes - squads / agile etc... • Scalability and efficiency lessons
- 16. #ibminsight
- 17. The SUN Platform logical architecture 17
- 18. Imperatives of the SUN Platform 18 Powers a multi- billion dollar business Serving all data on a global basis Proven Scales to the precise load without human interaction Scaling regularly between 15 and 26 Billion transactions a day Efficient Service oriented and API first methodologies Ability to “compose your own data flow" Flexible Including: Spark, Cassandra, and Parquet Supports structured, semi- structured, unstructured datasets Latest Tech Platform has never had an outage Able to sustain failures at any level with no operational impact Fault Tolerant
- 19. Example Use Case: Forecast on Demand (FoD) 19 FoD delivers Billionsof personalized forecasts a day for 2.2 Billion locations for The Weather Company
- 20. Why do so many great companies struggle putting their data to work? 20 Challenge Risk High cost to get started Maintaining the health, scale and performance of a platform is expensive • Skilled resources are expensive and hard to find, also different mix of skills are required for each step of implementation • Technology changes fast, it’s expensive to stay up to date It’s easy to fail, especially in the first few years Modernization involves a lot of failure before you succeed • Projects often reset at least 2x before getting it right The data will never be perfect Business data requires significant cost and time to cleanse and combine with external data sources through partnerships to mine for insight. Fast changing market needs Making the investment to build your own platform is a distraction from your core business
- 21. Step 1: buy, setup and manage cloud infrastructure Data centers closeto (global) users Automatic failover betweendata centers Automaticallyadapt to workload changes– increasing or decreasing resourcesand performance 21 IaaS Type Monthly 3-year Total Compute 10k $360k Storage 0.5k $18k Database 5k $180k Networking 0.5k $18k Analytics 2k $72k Management .5k $18k Security / Identity .2k $7.2k App Services .3k $10.8k Example over3 years Labor Cost 3- year Total Setup 100k $100k Manage 1 FTE == 12.5K /mo $450k Step 1 Costs: IaaS: $684k Labor: $550k Total: $1.23M * Assume 1 TOC FTE is 100k/year, 1 DevOps is 150k/year
- 22. Step 2: design and build your initial and ongoing solution ServicesArchitectureto: – Ingest / Transform / Persist / Analyze /Distribution – Self management / logging / monitoring Cloud agnostic APIdriven Automaticallyelastic, scalable 22 Example over3 years: Step 2 costs: • 12 Months to get to Production • Labor: $3.6M Labor Costs # Months FTE Count Total Initial Dev and Setup 12 8 $1.2M Ongoing DevOps (after initial dev) 24 8 $2.4M * Assume 1 TOC FTE is 100k/year, 1 DevOps is 150k/year
- 23. Step 3: Run and manage your system, 24/7 23 Labor Costs # Months FTE Count Total TOC Operations 36 5 $1.5M * Assume 1 TOC FTE is 100k/year, 1 DevOps is 150k/year TOC Facility Monthly 3-year Total Software 5k $180k Hardware 5k $180k Space 2k $72k Networking 0.5k $18k Example over3 years: Step 3 costs: Facility: $450k Labor: $1.5M Total: $1.95M
- 24. 3 year costs to build FoD from scratch Step1:Buy,setupandmanagecloud infrastructure Step 2: Design andbuildyourinitialand ongoingsolution Step 3: Run and manageyoursystem,24/7 24 Step 1 18% Step 2 53% Step 3 29% 3 Year Cost ($6.8M) Step 1: $1.23M, 18% of 3 year total cost Step 2: $3.60M, 53% of 3 year total cost Step 3: $1.95M, 29% of 3 year total cost Cost is roughly $6.8M over 3 years
- 25. 25
- 26. Profile for our foundational customers/partners Visionaries who share our belief and are driven by achieving an "order of magnitude" improvement in their business ü willing to invest in an initial use case ü see implementation as a project Datasets: non-regulated Interested? mailto: derek.baron@weather.com
- 27. Notices and disclaimers Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproducedor transmittedin any form without written permissionfrom IBM. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initialpublication and could includeunintentional technical or typographical errors. IBM shall have no responsibility to update this information. THISDOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISINGFROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESSINTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warrantedaccording tothe terms and conditions of theagreements under which they are provided. IBM products are manufactured from new parts or new and used parts. Insome cases, a product may not be new and may have been previously installed. Regardless, ourwarranty terms apply.” Any statements regarding IBM's future direction, intent or product plans aresubject to change or withdrawal without notice. Performance data containedhereinwas generally obtained in a controlled, isolated environments. Customerexamples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operatingenvironments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to makesuch products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associatedmaterials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials anddiscussions areprovided for informational purposes only, and areneither intended to, nor shall constitutelegal or other guidance or advice to any individual participant or their specific situation. It is the customer’s responsibility to insure its own compliance with legal requirements and toobtain advice of competent legal counsel as tothe identificationand interpretation of any relevant laws and regulatory requirements that may affect the customer’s business andany actions the customer may needto taketo comply with such laws. IBM does not providelegal advice or represent or warrant that its services or products will ensurethat thecustomer is in compliance withany law. 27 10/27/16World ofWatson 2016
- 28. Notices and disclaimers continued Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third- party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml. 28 10/27/16World ofWatson 2016
- 29. Thank You
- 30. 8:00-8:45 Weather: the Most Pervasive Force in Business Breakers H 8:30-11:00 Into the Storm: Extracting Weather Data and Putting It to Use for Your Business – Hands on Lab Breakers A 9:00-9:45 IBM Watson IoT Plus The Weather Company Equals a Game-Changer for Energy and Utilities Breakers A 9:00-9:45 Actionable Insights without Dealing with Data Sources, Analytics Software or Data Scientists Islander E 11:00-11:45 The Weather Company's Platform Story: Lessons from Handling Up to 26 Billion Transactions Per Day Islander E 1:00-1:45 Spotlight Session: Weather and Climate Science: Benefits to Business and Society to Date and Future Trends Theater Level 3 2:00-2:45 How Visibility into Foot Traffic Can Transform Retail: Demos and Real Client Use Cases Jasmine B 3:00-3:20 How Watson Powers Content Personalization at The Weather Company Redefining Development Theater #957 1:00-1:20 Adventures of a Storm Chaser Monetizing Data Community Theater, Booth #465 1:00-1:45 Fighting Crime with SPSS and Weather Data South Pacific D 1:20-1:50 How Many Forecast Models Does it Take to Predict the Weather Monetizing Data Community Theater Booth #465 2:00-2:45 Putting Cities at the Center of a Growth Strategy with IBM Metro Pulse Powered by Watson Breakers H 3:30-3:50 Let’s Talk About the Weather: Predictive Analytics Uncover the Impact of Climate Events Transforming Industries Theater, Booth #726 4:00-4:45 How American Airlines Uses Weather Data and Aviation Analytics Islander E 4:00-4:45 Energy and Utilities Outage Prediction, Demand Forecasting and Field Worker Safety through Weather Jasmine B 11:00-11:45 Increase Customer Loyalty through Proactive Alerting Islander E 9:00-9:45 Think You Really Know Your Customers? With Micro- segmentation, You Can Islander E 11:00-11:45 Weather and Location Should Be the Core of Your Business Strategy Breakers A 12:00-12:45 How Weather Data and the IoT Improve Nutrition and Food Safety throughout the Supply Chain Breakers J 1:00-3:30 Weathering the Storm: A Technical Deep-Dive into How to Get (and Use!) Weather Data Hands on Lab Bayside F-09 Monday Tuesday Wednesday Thursday Don’t miss these other key weather sessions