![“[Google's] ability to build, organize, and operate a
huge network of servers and fiber-optic cables
with an efficiency and speed that rocks physics on
its heels.
This is what makes Google Google: its physical
network, its thousands of fiber miles, and those
many thousands of servers that, in aggregate, add
up to the mother of all clouds.”
- Wired
Images by Connie Zhou](https://image.slidesharecdn.com/bigqueryochdataanalysmedgooglecloudplatform-140428070610-phpapp02/85/Connecta-Event-Big-Query-och-dataanalys-med-Google-Cloud-Platform-66-320.jpg)
Connecta Event: Big Query och dataanalys med Google Cloud Platform
- 1. © Connecta - Confidential
- 2. © Connecta - Confidential Ett verkligt kundbehov – våra kunder upplever svårigheter att göra vettiga analyser En stor potential och affärsmöjlighet genom dagens enorma mängder data Innovations– och kunskapsutvecklingen går fort – och det nu händer nu! 1) 2) 3)
- 3. © Connecta - Confidential ■ What is Big Data? ■ The Google Cloud Platform ■ Big Data on the Google Cloud Platform - Big Query ■ Case study - Casual gaming ■ Demo - Swedish election with Big Query and Tableau ■ Summary - The benefits of Big Data Agenda
- 4. © Connecta - Confidential • Svenskt konsultbolag som finns till för att förverkliga punkterna på ledningens agenda. Från strategi till transformation och värdeskapande • Ca 700 konsulter inom - Digital Consulting - Management Consulting - Enterprise Consulting - AM och Infrastruktur • Omsätter ca 800 MSEK och är noterade på Nordiska börsen. • Vi gör våra kunderna mer konkurrenskraftigagenom att kombinera affärsstrategiskt tänkande, tekniska kunskaper och förmågan att gå från ord till handling.
- 5. © Connecta - Confidential “90% of the data in the world today was created in the last 2 years alone” http://www.forbes.com/sites/ciocentral/2013/01/15/big-data-get-ready-for-the-2013-big-bang/
- 6. © Connecta - Confidential
- 7. © Connecta - Confidential Big Data on the top of the agenda
- 8. © Connecta - Confidential Top technology priorityThe 2013 CIO agenda (and 2012, 2009, 2008, 2007…)
- 9. © Connecta - Confidential
- 10. © Connecta - Confidential data is the oilof the 21st century
- 11. © Connecta - Confidential What is Big Data?
- 12. © Connecta - Confidential ▪ “Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using traditional data processing applications” ▪ The 3 V’s of Big Data Introduction to Big Data
- 13. © Connecta - Confidential “Now that we have all this data we have to ask the pivotal question; can it be trusted? This is the essence of Veracity.” The 4:th V: Veracity Edd Dumbill. Planning for Big Data: A CIO’s Handbook to the Changing Data Landscape. O’Reilly Media, 2012
- 14. © Connecta - Confidential Big data is about the business value it provides ▪ Unless business needs are met the data and the plan it drives are missing the vital element of value ▪ Value comes when you find insights you wouldn’t have found otherwise and when you start making better decisions ▪ Try to quantify the value and communicate it across the organization
- 15. © Connecta - Confidential
- 16. © Connecta - Confidential
- 17. © Connecta - Confidential Challenges
- 18. © Connecta - Confidential
- 19. © Connecta - Confidential Key Challenges in Big Data Information Strategy: ■ What is your plan with Big Data? Enterprise & External Information Management: ■ Information is everywhere – volume, variety, velocity – and it keeps growing! Technical threshold and competence ■ How will you start the work and who will do it?
- 20. © Connecta - Confidential Solution
- 21. © Connecta - Confidential Information Strategy: ■ Make it a top management issue and make somebody take responsibility for the effort ■ Connect your corporate strategy with your information strategy ■ Transforming company culture to be data-driven Enterprise & External Information Management: ■ Ensuring reliable and consistent data by structured work with Master Data Management (MDM) ■ The information must be used in the organization, veracity is crucial Solution to Key Challenges in Big Data
- 22. © Connecta - Confidential Technical threshold and competence ■ Choose the technical solution that fits your needs and resources ■ Secure competences with an overall picture in order to start the work ■ Start with small pilot projects to show the business value it can bring Solution to Key Challenges in Big Data
- 23. Cloud Platform Big Data Session with Connecta, April 24 - 2014 Guillaume Leygues, Enterprise Cloud Platform Sales Engineer Benelux & Nordics André Hoekzema, Enterprise Cloud Platform Lead Benelux & Nordics
- 24. “Enabling Technology for Disruptive Business Models”
- 25. Agenda 25th, 2014 Google Cloud Platform Introduction, Gaining Momentum Big Data on Google Cloud Platform Discussion 1 2 3
- 26. - Google’s Mission Statement “Organize the world’s information and make it universally accessible and useful.”
- 27. Building Products that Scale Google Maps Gmail Google Drive
- 28. Developing at Google scale means encountering Google-sized challenges.
- 29. For the past 15 years, Google has been building out the world’s fastest, most powerful, highest quality cloud infrastructure on the planet. Images by Connie Zhou
- 30. Google has been running some of the world’s largest distributed systems with unique and stringent requirements. Images by Connie Zhou
- 31. A Network that Spans the Globe
- 32. Google's Global OpenFlow Network
- 33. Innovating Software & Driving Technology Forward SpannerDremelMapReduce Big Table Colossus 2012 20132002 2004 2006 2008 2010 GFS Compute Engine
- 34. Cloud Storage Cloud SQL Cloud Datastore Compute Compute Engine App Engine App Services BigQuery Cloud Endpoints Storage
- 35. May 2013 Google Compute Engine (Preview) PHP for App Engine (Preview) Big JOIN in BigQuery The Last Year in the Cloud Platform November 2013 Cloud Endpoints GA Dedicated Memcache GA August 2013 Layer 3 Load Balancing Encryption at Rest for Cloud Storage December 2013 Compute Engine GA Live Migration Persistent Disks July 2013 Dedicated Memcache Offline Disk Import February 2014 HIPAA Support Cloud SQL GA
- 36. Source: Google Internal Data 4.75 Million active applications
- 37. Investments in Cloud Platform
- 38. We can do better Lower and simplify pricing Make developers more productive
- 39. Prices are falling • Public cloud prices have dropped 6-8% annually Source: Google Internal Data 20142006 Public Cloud Prices
- 40. But prices are not falling fast enough • Hardware costs have dropped 20-30% annually Hardware Cost Public Cloud Prices• Public cloud prices have dropped 6-8% annually Source: Google Internal Data 20142006
- 41. Pricing Updates (Effective April 1st, 2014) 35% price drop on Compute Engine, across all sizes, regions, and classes 37% price drop on App Engine frontend instance hours, 33% on Datastore writes and 50% on Dedicated Memcache 68% price drop on Cloud Storage On Demand pricing reduced by 85% - $5/TB
- 42. You should get the best price with... No Upfront Payments No Lock-in No Complexity
- 43. 100%0% 20% 40% 60% 80% Sustained Use Previous On Demand New On Demand $0.11 $0.10 $0.09 $0.08 $0.07 $0.06 $0.05 $0.04 $0.03 Sustained-use discountsNetPricePerHour
- 44. Sustained-Use Pricing 30% net reduction on Compute Engine instances with 24x7 use
- 45. • Managed VMs • The Flexibility of Compute Engine • The productivity of App Engine • Provides best of both worlds • IaaS + PaaS Flexibility Managementand Managed VMs
- 46. Developer Productivity • Use the tools you know and love • Fast, reliable deployments • Isolate and fix issues in production with Continuous Integration Developer Productivity Time to Market and Robust Design
- 47. 1000X BigQuery Streaming • Near real-time analysis • High fidelity, low latency • Focus on results, not sharding and transforming $0.01 per 100,000 rows Real time availability of data100,000 rows per second
- 48. • Deployment Manager • Replica Pools • Cloud DNS • Windows Server, SuSE, RHEL support and so much more...
- 49. Agenda 25th, 2014 Google Cloud Platform Introduction, Gaining Momentum Big Data on Google Cloud Platform Discussion 2 3 1
- 52. • Applications at the heart of business interactions • Devices and sensors • Lower cost of storage & ingestion • New programming models • New scale and capabilities for SQL • Easily available software (Open Source) • Easy on-ramp, cost effective experimentation • Unlimited scale, low TCO • Combine Open Source software and platform services Ability to process Cloud consumption modelData availability Key drivers in the growth of Big Data
- 53. Google Cloud Storage Mix and match storage and computation from OSS and Google Cloud Platform BigQuery and Datastore Connectors BigQueryDatastore Hadoop BigQuery Connector Datastore Connector Cloud Storage Connector HBase HivePig Hadoop Applications Hadoop, Pig, HBase, and Hive are trademarks of the Apache Software Foundation.
- 54. Q3, 2012 Q4,2012 Q1, 2013 Q2, 2013 TodayQ3, 2013 Q4, 2013Q2, 2012 Launch 1000x Streaming rate Table Views Table Wildcards JSON functions SQL Improvements BigQuery Innovation Momentum Google Analytics Integration Streaming API Table Decorators Large Query Results Query Caching Analytic functions Big JOIN Big Aggregates Timestamp JSON Import Nested / Repeated Fields Datastore ImportBatch Processing Excel Connector
- 56. Ease of use • Simplified infrastructure for realtime use cases • Stream events row-by-row via simple API Use cases • Server Logs, Mobile apps, Gaming, In-App real time analytics BigQuery Streaming Low cost: $0.01 per 100,000 rows Real time availability of data100,000 rows per second Customer example:
- 57. Google Analytics + BigQuery Google Analytics Premium Platform Google BigQueryData Pipeline Native Data Pipeline to Load Data into BigQuery Project
- 58. Google Analytics + BigQuery Customers
- 59. BigQuery in Action " The interactive performance of Google BigQuery, combined with Tableau’s intuitive visualization tools, enabled our analysts to interactively explore huge quantities of data – hundreds of millions of rows – with incredible efficiency. Previously, analyses would require hours or days to complete, if they would even complete at all. With Google BigQuery it takes minutes, if that, to process. This time-to-insight was previously impossible" – Giovanni DeMeo Vice President Global Marketing and Analytics
- 60. " The simulation cluster ran for nearly two months as part of the ATLAS distributed compute grid, logging over 5 million core-hours, completing 458,000 computationally intensive jobs and processing about 214 million events. The cluster achieved sustained peak throughput of 15,000 jobs per day. “We had a great experience with Google Compute Engine … and think that it is modern cloud infrastructure that can serve as a stable, high performance platform for scientific computing”. – Dr. Panitkin CERN Atlas Project CERN Atlas Compute Grid Extended on GCE
- 61. • 1.5TB in 60 seconds • 8,412 cores • Google Compute Engine MapR Breaks Minute Record Sort
- 62. Thank You
- 63. Agenda 25th, 2014 Google Cloud Introduction, Gaining Momentum Big Data on Google Cloud Platform Discussion 1 2 3
- 64. 28 Billion requests per day on App Engine
- 65. 6.3 Trillion Cloud Datastore operations per month
- 66. “[Google's] ability to build, organize, and operate a huge network of servers and fiber-optic cables with an efficiency and speed that rocks physics on its heels. This is what makes Google Google: its physical network, its thousands of fiber miles, and those many thousands of servers that, in aggregate, add up to the mother of all clouds.” - Wired Images by Connie Zhou
- 67. © Connecta - Confidential Big Data in practice - Understanding player behavior in a Casual game - Patrik Gottfridsson
- 68. © Connecta - Confidential ■ Simple rules, easy to learn ■ Play in short bursts ■ No long-term commitment ■ Targets a mass audience What is casual gaming?
- 69. © Connecta - Confidential Very small revenue per user ● (Paid) ● In-App Purchase ● Ads Business model
- 70. © Connecta - Confidential ■ Measure 2nd day retention ■ Optimize across game versions Make it sticky Reactivate Encourage ■ Find the “stales” ■ Send a “miss you” push notification ■ Find the “spiders”, the socially connected players ■ Drop their rate of ad shows Facts based revenue optimization
- 71. © Connecta - Confidential BigData BigData BigData ■ Measure 2nd day retention ■ Optimize across game versions Make it sticky Reactivate Encourage ■ Find the “stales” ■ Send a “miss you” push notification ■ Find the “spiders”, the socially connected players ■ Drop their rate of ad shows Facts based revenue optimization
- 72. © Connecta - Confidential CSV upload Cron import Google spreadsheets High level technical solution
- 73. © Connecta - Confidential Quickly up and running Avoid upfront license costs Avoid on- premise hardware Process millions of events per day Challenges
- 74. © Connecta - Confidential Collect everything you can Segmentation of the data model Validate your analytical queries Visualize graphically (obviously) Success factors
- 75. © Connecta - Confidential Immediate discoveries about gamer behavior New campaigns launched to revive “stales” and encourage “spiders” Continous follow-up of player statistics at the board level All in all, better optimized games and an increased profitability Results
- 76. © Connecta - Confidential Demo How to make data useful using Google Cloud Platform
- 77. © Connecta - Confidential 60% potential increase in operating margins for retail
- 78. © Connecta - Confidential > 2x competitive advantage 5-6% higher productivity and profitability Significantly higher return on equity and market value Data-driven decisionmaking
- 79. © Connecta - Confidential What’s your next step?
- 80. © Connecta - Confidential Connecta offers: ■ BigQuery Quickstart - Initial analysis, workshops and a running BigQuery solution ■ Cloud Code Workshop - Get your team up to speed on the Google Cloud Platform ■ Cloud Assessment - Analysis, workshops and identification of where a Cloud solution would make your company more competitive What’s your next step?