Cloud Data Services - from prototyping to scalable analytics on cloud
1 like490 views
The document outlines a framework for leveraging cloud data services for big data analytics, emphasizing the steps of ideation, prototyping, failing fast, and moving to production. It highlights various IBM cloud offerings, such as Cloudant for NoSQL storage, DashDB for in-memory analytics, and Spark for interactive data analysis, to facilitate the process. The author encourages using these tools to analyze data for monetization and suggests combining insights with real-time conditions to enhance user experience.
Cloud Data Services - from prototyping to scalable analytics on cloud
- 2. 1. IDEAS In the beginning there is an idea: • Collect new data • Combine internal and external data • Explore data • Analyze data in new ways But how to bring the ideas to life? ENERGY HEALTH GEO SCIENCEMOBILITY INTERNAL DATA SHOPPING
- 4. Cloud-Based Systems of Engagement (NoSQL, Mobile Apps, Internet of Things, Social Media) CLOUDANT IBM’s NoSQL store • is deployed in minutes on cloud • easily integrates with applications and data sources in the Bluemix context • includes analytic capabilites(e.g. geospacial) • is flexible in scaling and replication
- 6. 2. PROTOTYPE A prototyping environment is needed • Fast and easy deployment • Various technologies • Integration with internal structures • Compatible with tools of choice
- 7. DASHDB IBM’s in-memory analytics database • is deployed in minutes on cloud • easily integrates with applications and data sources in the Bluemix context, cloud providers (e.g. Amazon) and internal IT • has a rich SQL interface and in- database analytics capabilities (e.g. statistics, geospatial) • comes with R environment included Cloud-Based Systems of Engagement (NoSQL, Mobile Apps, Internet of Things, Social Media) Schema Discovery Process IBM & Third Party Integrations (Cognos, SPSS, SAS, Tableau, ESRI ArcGIS, R) • is an analytics warehouse extension of Cloudant
- 11. 1. IDEASDATA IN MY SANDBOX • I created my own Sandbox system to analyze the data on Bluemix: an analytical database (dashDB) • I copied the data from Cloudantto dashDB Schema Discovery Process • But now I would like to analyze the data interactively with my favorite language: Python
- 12. Cloud-Based Spark environment (Object Storage, SQL, R, Python, Text Analytics, Machine Learning, Jupyter notebooks) SPARK IBM’s Spark-as-a-Service offering • is deployed in minutes on cloud • Pay-as-you-go or reserved environment • integrated object storage, scales with number of executors • easily integrates with applications and data sources in the Bluemix context • Jupyternotebooks as frontendSQL
- 16. 1. IDEASI LIKE NOTEBOOKS • I create a Spark service on Bluemix • I access the data from my Jupyter notebook • In the notebook I can create a visualization to see where runners typically do their training in Munich • Interactively I can dig deeper into the data • I look at the most active tracks and the toughest tracks
- 17. 1. IDEASI FOUND SOMETHING • The combination of popular tracks that are tough give me routes that are suitable to place ads • The points of highest inclination are the places that are best to place refreshment ads, e.g. energy drinks • Maybe, we could sell that knowledge
- 18. Cloud-Based Hadoop environment (Spark, SQL, R, Text Analytics, Machine Learning) BIGINSIGHTS IBM’s Hadoop distribution • is deployed in minutes with virtualized HW on cloud • is deployed in days with bare metal HW for high performance on cloud • easily integrates with applications and data sources in the Bluemix context • is 100% open source with IBM added value extensions • is flexible in scaling SQL
- 19. WATSON ANALYTICS IBM’s Self-service analytics for business users and experts • is deployed in minutes on cloud • automatically analyzes your data and provides questions for you to explore • provides automated data visualizations • identifies predictors of your target analysis • allows to pin your results for use in dashboards and storytelling Fully Automated Intelligence Natural Language Dialogue Guided Analytic Discovery Single Analytics Experience
- 20. DATAWORKS IBM’s Data Refinery Service that • is deployed in minutes on cloud • is a collection of cloud-based data access and refinement services • integrates with data services on Bluemixand on-premise • offers data load, data profiling, probabilistic match Systems of Record & Insight Integration (Watson Analytics, DB2, Oracle, Hadoop, dashDB, flat files)
- 22. Retrieve and Rank Natural Language Classifier Tone Analyzer Watson’s APIs are also around the corner – the cognitive building blocks that harness your data
- 23. 3. FAIL FAST If the idea is not working or gives no valuable insights • Don’t spend time on it any more • Throw your prototyping environment away • Move on to the next idea • All the IBM offerings described are cloud-based, have entry versions free of charge and are paid on a monthly basis • If you don’t need them any more, just stop the subscription
- 24. 1. IDEASWE CAN DO MORE • Bluemixdevelopment platform offers many services to build a complete solution • Use weather data in combination with location info and our knowledge while customers use the app for an activity • Push special offers to customer related to current conditions customer is at high intensity point Right Now 26.3° weather is hot Picture by Shinya Suzuki
- 25. 4. PRODUCTION • If the ideas are proved to give insights or benefits, bring them to production • But don’t invent the wheel again • Adapt the procedures implemented in the prototyping phase directly to production • The IBM offerings are not just small prototyping environments • They have production ready variants or could be deployed on-premise
- 26. 1. IDEAS 2. PROTOTYPE 3. FAIL FAST 4. PRODUCTION • IBM can support you to bring your innovations to life • Prototype ideas, fail fast, bring successful idea into production – in the cloud on-premise or hybrid • Innovations born in the cloud and matured where it fits best to your business