Net conf cl v2018 real time analytics
Download as PPTX, PDF0 likes39 views
The document discusses handling real-time scenarios using IoT and Power BI. It describes ingesting data from sensors and devices using Event Hubs and Stream Analytics to transform the data in real-time. The transformed data is stored in outputs like Azure Blob Storage and sent to Power BI for real-time analytics and dashboards.
Net conf cl v2018 real time analytics
- 1. Handling Real Time Scenarios - IoT + Power BI Gastón Cruz / MVP Data Platform
- 2. IoT Scenarios – Real time analytics Today, 80% of organizations adopt cloud-first strategies AI investment increased by 300% in 2017 Data will grow to 44 ZB in 2020
- 3. D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d Custom apps Sensors and devices Business apps
- 4. Custom apps Sensors and devices Ingest Business apps D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 5. Custom apps Sensors and devices StoreIngest Business apps D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 6. Prep & Train Custom apps Sensors and devices StoreIngest Business apps D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 7. Model & ServePrep & Train Custom apps Sensors and devices StoreIngest Business apps D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 8. Model & ServePrep & Train Custom apps Sensors and devices StoreIngest Data Factory (Data movement, pipelines & orchestration) Business apps D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 9. Model & ServePrep & Train Custom apps Sensors and devices StoreIngest Data Factory (Data movement, pipelines & orchestration) Event Hubs IoT Hub Business apps Kafka D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 10. Model & ServePrep & Train Custom apps Sensors and devices Store Blobs Data Lake Ingest Data Factory (Data movement, pipelines & orchestration) Event Hubs IoT Hub Business apps 10 01 Kafka D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 11. Model & ServePrep & Train HDInsight / SparkCustom apps Sensors and devices Store Blobs Data Lake Ingest Data Factory (Data movement, pipelines & orchestration) Machine Learning Event Hubs IoT Hub Business apps 10 01 Kafka D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 12. Model & ServePrep & Train HDInsight / SparkCustom apps Sensors and devices Store Blobs Data Lake Ingest Data Factory (Data movement, pipelines & orchestration) Machine Learning Cosmos DB SQL Data Warehouse Analysis Services Event Hubs IoT Hub SQL Database Business apps 10 01 SQLKafka D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 13. Model & ServePrep & Train HDInsight / SparkCustom apps Sensors and devices Store Blobs Data Lake Ingest Data Factory (Data movement, pipelines & orchestration) Machine Learning Cosmos DB SQL Data Warehouse Analysis Services Event Hubs IoT Hub SQL Database Business apps 10 01 SQLKafka D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 14. Model & ServePrep & Train HDInsight / SparkCustom apps Sensors and devices Store Blobs Data Lake Ingest Data Factory (Data movement, pipelines & orchestration) Machine Learning Cosmos DB SQL Data Warehouse Analysis Services Event Hubs IoT Hub SQL Database Business apps 10 01 SQLKafka D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 15. Model & ServePrep & Train HDInsight / SparkCustom apps Sensors and devices Store Blobs Data Lake Ingest Data Factory (Data movement, pipelines & orchestration) Machine Learning Cosmos DB SQL Data Warehouse Analysis Services Event Hubs IoT Hub SQL Database Business apps 10 01 SQLKafka D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 16. Model & ServePrep & Train HDInsight / SparkCustom apps Sensors and devices Store Blobs Data Lake Ingest Data Factory (Data movement, pipelines & orchestration) Machine Learning Cosmos DB SQL Data Warehouse Analysis Services Event Hubs IoT Hub SQL Database Business apps 10 01 SQLKafka D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 17. Model & ServePrep & Train HDInsight / SparkCustom apps Sensors and devices Store Blobs Data Lake Ingest Data Factory (Data movement, pipelines & orchestration) Machine Learning Cosmos DB SQL Data Warehouse Analysis Services Event Hubs IoT Hub SQL Database Analytical dashboards Predictive apps Operational reports Intelligence Business apps 10 01 SQLKafka D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 18. Model & ServePrep & Train HDInsight / SparkCustom apps Sensors and devices Store Blobs Data Lake Ingest Data Factory (Data movement, pipelines & orchestration) Machine Learning Cosmos DB SQL Data Warehouse Analysis Services Event Hubs IoT Hub SQL Database Analytical dashboards Predictive apps Operational reports Intelligence Business apps 10 01 SQLKafka D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 19. Model & ServePrep & Train HDInsight / Spark DATA Custom apps Sensors and devices Store Blobs Data Lake Ingest Data Factory (Data movement, pipelines & orchestration) Machine Learning Cosmos DB SQL Data Warehouse Analysis Services Event Hubs IoT Hub SQL Database Analytical dashboards Predictive apps Operational reports Intelligence Business apps 10 01 SQLKafka D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 20. Model & ServePrep & Train HDInsight / Spark DATA Custom apps Sensors and devices INTELLIGENCE Store Blobs Data Lake Ingest Data Factory (Data movement, pipelines & orchestration) Machine Learning Cosmos DB SQL Data Warehouse Analysis Services Event Hubs IoT Hub SQL Database Analytical dashboards Predictive apps Operational reports Intelligence Business apps 10 01 SQLKafka D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 21. Model & ServePrep & Train HDInsight / Spark DATA Custom apps Sensors and devices INTELLIGENCE ACTION Store Blobs Data Lake Ingest Data Factory (Data movement, pipelines & orchestration) Machine Learning Cosmos DB SQL Data Warehouse Analysis Services Event Hubs IoT Hub SQL Database Analytical dashboards Predictive apps Operational reports Intelligence Business apps 10 01 SQLKafka D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
- 22. IoT Solution components •Sensor Data - App
- 23. IoT Solution components •Sensor Data - App •Create an Azure EventHub – Ingest Data
- 24. IoT Solution components •Sensor Data - App •Create an Azure EventHub – Ingest Data •Create Stream Analytics Job
- 25. IoT Solution components •Sensor Data - App •Create an Azure EventHub – Ingest Data •Create Stream Analytics Job •Create Queries to Transform Real Time
- 26. IoT Solution components •Sensor Data - App •Create an Azure EventHub – Ingest Data •Create Stream Analytics Job •Create Queries to Transform Real Time •Create Output Sinks – IE: Azure Blob Storage
- 27. IoT Solution components •Sensor Data - App •Create an Azure EventHub – Ingest Data •Create Stream Analytics Job •Create Queries to Transform Real Time •Create Output Sinks – IE: Azure Blob Storage •Add Power BI Output
- 28. IoT Solution components •Sensor Data - App •Create an Azure EventHub – Ingest Data •Create Stream Analytics Job •Create Queries to Transform Real Time •Create Output Sinks – IE: Azure Blob Storage •Add Power BI Output •Start Stream Analytics Job
- 30. Model & ServePrep & Train HDInsight / Spark DATA Custom apps Sensors and devices INTELLIGENCE ACTION Store Blobs Data Lake Ingest Data Factory (Data movement, pipelines & orchestration) Machine Learning Cosmos DB SQL Data Warehouse Analysis Services Event Hubs IoT Hub SQL Database Analytical dashboards Predictive apps Operational reports Intelligence Business apps 10 01 SQLKafka D a t a m a n a g e m e n t s o l u t i o n f o r i n t e l l i g e n c e i n t h e c l o u d
Editor's Notes
- #2: PRESENTACION GASTON PRESENTACION PERSONAL (AÑOS MVP) EXPLICACION CHARLA
- #3: BURBUJA 1: COMENTAR LA CANTIDAD DE DATA QUE SE CAPTURA DIARIAMENTE Y EL PORCENTAJE REAL DE USO DE LA MISMA FUENTES DE INFORMACION BURBUJA 2: OLAS DE MIGRACION A LA NUBE TECNOLOGIAS NUBE BURBUJA 3: CICLO DE VIDA DE LOS DATOS CRECIMIENTO INVERSIONES INTELIGENCIA ARTIFICIAL
- #5: ORIGENES GENERALES NO CONTROL DE LOS ORIGENES MASIVIDAD ORIGENES ABSTRACCION (REAFIRMAR) ARQUITECTURA ANTES QUE NADA!!!!!! PASAJE A INGESTA DE DATOS EL CONTROL DE LA GENERACION ES DE TERCEROS, Y LA INGESTA ES EL PUNTO DE ENTRADA EN CUALQUIER ARQUITECTURA TODA PUNTA DE INFORMACION GENERA UN ENDPOINT
- #6: PRIMER PUNTO MARCAR QUE INGESTA ES EL INICIO DE LA ARQUITECTURA REAFIRMAR LA IMPORTANCIA Y EL CONCEPTO DE ARQUITECTURA LOS PUNTOS DE CONTROL SON -> EMPEZAR PASAJE (TORTE)
- #10: - CAPA TRANSVERSAL DE INGESTA, MODELAMIENTO, TRANSFORMACION, LIMPIEZA CON AZURE DATA FACTORY
- #11: INGESTA A TRAVES DE MODELOS NO RELACIONALES PARA REAL O NEAR REAL TIME ANALYTICS CONEXIÓN A SENSORES DE PLANTA A TRAVES DE EVENT HUB – IOT HUB
- #12: STORE – VOLCADO DE INFO EN FORMATOS DISTINTOS (JSON, CSV, TXT, OTROS) EN DATA LAKE (FILE STORAGE, BLOB STORAGE)
- #13: CON OPCIONES DE PREPARAR UN MODELO Y ENTRENARLO PODEMOS PROFUNDIZAR CON UN CLUSTER DE HDINSIGHT Y APLICAR LUEGO MACHINE LEARNING PARA CREAR MODELOS PREDICTIVOS
- #14: PARA SERVIR DICHA INFORMACION: SISTEMAS DE ALOJAMIENTO RELACIONAL Y NO RELACIONAL MULTI GEO – COSMOS DB SQL DB / DW MODELOS TABULARES CON AAS / PERFILAMIENTO Y ROLES ORGANIZACIONALES / SEGMENTACION
- #15: - COMPLEMENTO ENTRE SISTEMAS
- #16: - PIPELINE GENERAL DE UNA SOLUCION DE DATOS
- #19: ANALITICA DESCRIPTIVA / OPERACIONAL / PREDICTIVA MUTIPLES TIPOS DE SALIDAS DEFINIDAS POR NEGOCIO OFFICE 365
- #21: CASO COFRE: SISTEMA DE MONITOREO PRE-EXISTENTE
- #22: GENERACION DE ARQUITECTURA DE GRANDES VOLUMENES Y PROCESAMIENTO
- #23: DEFINICION BUROCATICA
- #24: Following are some examples where Azure Stream Analytics can be used: Internet of Things(IoT) Sensor fusion and real-time analytics on device telemetry Web logs/clickstream analytics Geospatial analytics for fleet management and driverless vehicles Remote monitoring and predictive maintenance of hi-value assets Real-time analytics on Point of Sale data for inventory control and anomaly detection
- #36: MOSTRAR PUNTOS PRINCIPALES (DATA, INTELLIGENCE, ACTION) MOSTRAR LAS PARTES DE LA ARQUITECTURA DATA = APLICACION INGESTA = EVENT HUB STORE = BLOB STORAGE/EVENT HUB STORAGE MODEL AND SERVE = JOB INTELLIGENCE = POWERBI APLICACION DIFERENCIA ENTRE LO HABLADO Y LA APP EXPLICAR DE QUE SE TRATA LA APP INGESTA EXPLICAR QUE ES EL EVENT HUB EXPLICAR EL ROL VER CONFIGURACION DE LA APP RECEPCION DE INFORMACION STORE MOSTARR EL STORE MSOTRAR COMO SE ALMCENAN LOS DATOS CONEXION CON EVENT HUB MODEL AND SERVE CONEXION CON PBI INTELLIGENCE POWERBI REPORT DASHBOARD