Digital_IOT_(Microsoft_Solution).pdf
- 1. Leveraging the Cloud to Architect Digital Solutions
- 2. Agenda • State of the art technology for IoT • Table Storage Demo • ML Clustering and Classification prototype • Cortana Analytics • Architecture for building today • Patterns and anti-patterns • Open Discussion 2
- 3. What is Digital? Digital is more than technology. It involves connecting that technology with the right data science, devices, design and business strategy. It is putting a customer, device, organization or business process at the center of real change in how businesses do things and how customers experience them. In how we engage, invent, build and buy everything. It means creating value by uniting the physical world—seamlessly, efficiently, meaningfully—to the virtual one we’re building. 3
- 4. Microsoft Azure IoT services Devices Device Connectivity Storage Analytics Presentation & Action Event Hubs SQL Database Machine Learning App Service Service Bus Table/Blob Storage Stream Analytics Power BI External Data Sources DocumentDB HDInsight Notification Hubs External Data Sources Data Factory Mobile Services BizTalk Services { } 4
- 5. >90,000 New Azure customer subscriptions/month 1.5Trillion Messages per month processed by Azure IoT >500Million Users in Azure Active Directory 777Trillion Storage Transactions per day >1.5Million SQL Databases running on Azure >40% Revenue from Start-ups and ISVs Cloud Momentum 5
- 6. Device Connectivity and Security Data Ingestion and Command & Control Stream Processing & Predictive Analytics Workflow Automation Dashboards and Visualization Preconfigured Solutions Azure IoT Suite Managing Big Data 6
- 7. Is IoT even a new thing? Depending on who you ask, IoT is either: Nothing new A unicorn Magic, and will soon change everything. We’ve been doing this for 40 years 7
- 8. Emerging Challenges for IT • Scale • # devices >> # users, and growing fast • Volume of data (and network traffic) • Pace • Innovation pressure: analysis, command and control, cost • Skill pressure: data science, new platforms • Environment • IT/OT collaboration • Security and privacy threats • Emerging standards • New competitors 8
- 9. Azure Machine Learning Conceptual Model 9
- 10. • Data scientist – A highly educated and skilled person who can solve complex data problems by employing deep expertise in scientific disciplines (mathematics, statistics or computer science) • Data professional – A skilled person who creates or maintains data systems, data solutions, or implements predictive modelling Roles: Database Administrator, Database Developer, or BI Developer • Software developer – A skilled person who designs and develops programming logic, and can apply machine learning to integrate predictive functionality into applications Machine learning roles 10
- 11. Pattern: Think big. Start small • Build to an architecture that will scale, but start prototyping with a small number of devices. • It’s hard to predict what data provides value -- which impacts which sensors and devices are necessary -- until you build something. • The options can be overwhelming: set crisp goals up front and use those to define and refine. • It’s much easier to work through device identity, management/update and security at small scale. . 11
- 12. Pattern: Telemetry first • It is very hard to predict in advance what data will be useful. • It is tempting, but likely inefficient to try for business transformation in the first step. • Think about not only device telemetry but also diagnostic telemetry. • Privacy and security implications of telemetry are generally lesser than for command and control. 12
- 13. Telemetry today • High scale data ingestion stream processing • Storage for cold-path analytics • Processing for hot-path analytics 13
- 15. Storage 15
- 16. Traditional RDBMS vs. Table Storage Value Metric 1000 kB kilobyte 10002 MB megabyte 10003 GB gigabyte 10004 TB terabyte 10005 PB petabyte 10006 EB hexabyte 10007 ZB zettabyte 10008 YB yottabyte 16
- 17. Entity Properties – Up to 1MB per entity – PartitionKey & RowKey (only indexed properties) – Uniquely identifies an entity – Defines the sort order – Timestamp – Optimistic Concurrency – Exposed as an HTTP Etag – Each property is stored as a <name, typed value> pair – No schema stored for a table – Properties can be the standard .NET types – String, binary, bool, DateTime, GUID, int, int64, and double 17
- 18. Purpose of the PartitionKey – Entities in the same partition will be stored together – Efficientquerying and cachelocality – Endeavourto include partition key in all queries – Atomic multiple Insert/Update/Delete in same partition in a single transaction – Target throughput – 500 tps/partition, several thousand tps/account – Windows Azure monitors the usage patterns of partitions – Automatically load balance partitions – Each partition can be served by a differentstoragenode – Scale to meetthe trafficneeds of your table 18
- 19. Azure Machine Learning Conceptual Model 19
- 20. Azure IoT Reference Architecture Solution Portal Provisioning API Identity & Registry Stores Stream Event Processor Analytics/ Machine Learning Data Visualization & Presentation Device State Store Gateway Storage IP capable devices Existing IoT devices Low power devices Presentation Device and Event Processing Data Transport Devices and Data Sources Cloud Gate- way Agent Libs Agent Libs Control System Worker Role Agent Libs 20
- 21. Demo: ML Clustering and Classification of Data
- 23. 23
- 24. Azure Machine Learning Conceptual Model 24
- 25. Solution Portal Provisioning API Identity & Registry Stores Stream Event Processor Analytics/ Machine Learning Data Visualization & Presentation Device State Store Gateway Storage IP capable devices Existing IoT devices Low power devices Presentation Device and Event Processing Data Transport Devices and Data Sources Cloud Gate- way Agent Libs Agent Libs Control System Worker Role Agent Libs
- 27. Data, Information, Knowledge, Wisdom Hierarchy 27
- 29. Cortana Analytics Suite Transform data into intelligent action 29
- 30. Azure Machine Learning Conceptual Model 30
- 31. Azure IoT Reference Architecture Solution Portal Provisioning API Identity & Registry Stores Stream Event Processor Analytics/ Machine Learning Data Visualization & Presentation Device State Store Gateway Storage IP capable devices Existing IoT devices Low power devices Presentation Device and Event Processing Data Transport Devices and Data Sources Cloud Gate- way Agent Libs Agent Libs Control System Worker Role Agent Libs 31
- 32. Visualizing ML Results in Power BI 32
- 33. The need to know what could be… 33
- 34. Deriving Business Value from Big Data 34
- 35. Vision Analytics Using past data to predict the future Recommenda- tion engines Advertising analysis Weather forecasting for business planning Social network analysis Legal discovery and document archiving Pricing analysis Fraud detection Churn analysis Equipment monitoring Location-based tracking and services Personalized Insurance Machine learning & predictive analytics are core capabilities that can help create added business value 35
- 36. Summary • Think big (architecture), but start small (experiment, learn and refine). • Start with telemetry. Address privacy, security and manageability before moving to command and control. • Don’t interrupt the fast path and create processing bottlenecks. • Build to the reference architecture to ease the move to IoT Suite. 36
- 37. Future Topics • Cortana Analytics Suite • Azure Portal • Scalable Cloud Applications • PaaS development • Event Hub • Stream Analytics • Machine Learning Studio 37
- 38. 38
- 39. Q&A