Fom io t_to_bigdata_step_by_step-final
0 likes171 views
The document discusses steps for an IoT to Big Data approach, including acquiring IoT data, edge streaming analytics, big data analytics for IoT, and hardware platforms for pilots. It describes successes with IoT projects in areas like connected healthcare. It outlines the IoT and big data landscape models and focuses on solutions for data visibility, critical situations, extracting value from IoT data, and infrastructure including networks and cloud computing. The presentation emphasizes how Dimension Data offers full-stack solutions across the IoT and big data models and can help clients do incredible things with IoT.
![IoT – Soluções de Visualização
CONTROLADORA
PaaS: Utilizando uma
plataforma pronta
Producer
Kafka
Consumer
PaaS :Desenvolvendo sua
Plataforma
JSON / REST / HTTP
JSON
ServerClient
HTTP POST
/service/weather
(REST Interface)
[{“city”:”Paris”, “units”: “C”}]
[{“low”: “16”, “high”: “23”}]
Response
Request](https://image.slidesharecdn.com/fomiottobigdatastepbystep-final-170522164558/85/Fom-io-t_to_bigdata_step_by_step-final-8-320.jpg)
Fom io t_to_bigdata_step_by_step-final
- 1. Uma Abordagem Passo a Passo Do IoT ao Big Data
- 2. • IoT – Visibilidade dos Dados • IoT – Edge Streaming Analytics • Big Data Analytics para IoT • Plataforma de Hardware para um Piloto Agenda Do IoT ao Big Data, uma Abordagem Passo a Passo
- 3. Sucessos com projetos de IoT Connected healthcare Connected building Physical security Real-time analytics USMEA UK US MEA First responder Virtual patient observation $142 milhões em oportunidades de IoT
- 4. Cenário O Landscape de Big DataO Modelo IoT World Forum Model Collaboration Business & people process Application Report, Control Data Abstraction Aggregation and access Data accumulation Storage Edge Computing Data analisys & transformation Connectivity Communication & processing Physical Devices & Controllers
- 5. Mas eu só queria ter visibilidade dos meus dados...
- 7. IoT – Aquisição dos Dados PINS DB MCU MCU CLP NUVEM Privada ou Pública CONTROLADORA
- 8. IoT – Soluções de Visualização CONTROLADORA PaaS: Utilizando uma plataforma pronta Producer Kafka Consumer PaaS :Desenvolvendo sua Plataforma JSON / REST / HTTP JSON ServerClient HTTP POST /service/weather (REST Interface) [{“city”:”Paris”, “units”: “C”}] [{“low”: “16”, “high”: “23”}] Response Request
- 9. IoT WF Model x DD Model DD Offers Full Stack Solution IoT World Forum Model Dimension Data Solution Focus Collaboration Business & people process Application Report, Control Data Abstraction Aggregation and access Data accumulation Storage Edge Computing Data analisys & transformation Connectivity Communication & processing Physical Devices & Controllers Applications Secure Conectivity
- 10. Gostaria de atuar de imediato em situações “críticas” .
- 11. CONTROLADORA MCU MCU CLP IoT – Edge Streaming Analytics Quark Edge Devices Communication Centralized Analytics Kafka MQTT Watson IOT Platform Custom Hub Custom Apps
- 12. IoT WF Model x DD Model DD Offers Full Stack Solution IoT World Forum Model Dimension Data Solution Focus Collaboration Business & people process Application Report, Control Data Abstraction Aggregation and access Data accumulation Storage Edge Computing Data analisys & transformation Connectivity Communication & processing Physical Devices & Controllers Applications Distributed Computing Secure Conectivity
- 13. Quero extrair valor dos dados do meu IoT !
- 14. • Fast Data – Capacidade de processar streams dos sensores De que Precisamos? • Big Data – Capacidade para armazenar dados dos sensores e resultados do Analytics (arquivos, tabelas, etc.) • Uma ferramenta de Analytics • Uma ferramenta de Visualização / Query • Ambiente escalável, altamente disponível e seguro
- 15. Como seria a Arquitetura da Solução Big Data para IoT ? Serving LayerTomada de Ação - Visualização Streaming Data Twiter´s Nathan Marz (2013) Discreto Contínuo Supervisionado Classificação Regressão File System Distribuído Modelo ML Speed Layer Desenv. do modelo Analítico Batch Layer Dados
- 16. Em que Ferramenta de Big Data Analytics Investir ? "Todas as principais plataformas comerciais avançadas de análise utilizam Spark. " Gartner 2016 Magic Quadrant for Advanced Analytics "Para ciência de dados utilizando aplicativos de fonte aberta e escalável, Spark é unanimidade."
- 17. Porque Spark é Unanimidade ? Python API ! Feito para Big Data, mas “rode” onde quiser... “One Shop, all the Goods !” Spark SQL structured data Spark Streaming real-time Mlib machine learning GraphX graph processing Spark Core
- 18. Enterprise Grade Big Data com Spark • Storage “Web-scale” • Alta Disponibilidade • Mirroring & Snapshots • Segurança integrada • Etc. YARN: Data Operating System 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° °N HDFS Hadoop Distributed File System Interactive Real-TimeBatch Applications Distribuição Hadoop “Enterprise Grade” • HDFS: File system robusto e escalável • YARN: Resource Manager e Scheduler
- 19. IoT WF Model x DD Model DD Offers Full Stack Solution IoT World Forum Model Dimension Data Solution Focus Collaboration Business & people process Application Report, Control Data Abstraction Aggregation and access Data accumulation Storage Edge Computing Data analisys & transformation Connectivity Communication & processing Physical Devices & Controllers Applications Distributed Computing Secure Conectivity Analytics
- 20. E a infraestrutura de rede e computação ?
- 21. Anatomia do Tráfego em Map-Reduce Many-to-Many Traffic Pattern Map 1 Map 2 Map NMap 3 Reducer1 Reducer2 Reducer3 ReducerN HDFS Shuffle Output Replication NameNode JobTracker ZooKeeper
- 22. A Fabric é Importante para o Cluster Big Data Fabric 10 GE Tráfego Horizontal Baixa Latência “Oversubscription” adequado 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 Latency(us) Time READ - Average Latency (us) QoS - READ - Average Latency (us) ~45% Read Improvement BufferCellsUsed Timeline Hbase Major Compaction Hbase (YCSB) NO QoS W/ QoS Terasoft held on Queues QOS Faz Diferença
- 23. Starter Kits com Servidores e Rede Cisco 5 3 4 5 3 4 2 5 4 MPP HADOOP NOSQL Compute IO Bandwidth Capacity Starter High Performance Designed for Performance and density for analytics engines, NoSQL databases, and entry-level Hadoop deployments Extreme performance and density for analytics engines Server UCS C220 M4 UCS C220 M4 CPU 2 x Intel Xeon E5-2620 v3 (15M Cache, 2.40 GHz) 2 x Intel Xeon E5-2680 v3 (30M Cache, 2.50 GHz) Memory 256GB 256GB Storage Controller Cisco 12-Gbps SAS Modular Raid Controller with 2-GB FBWC Cisco 12-Gbps SAS Modular Raid Controller with 2-GB FBWC Storage 8 1.2-TB 10K SAS SFF HDD 2 1.2-TB 10K SAS SFF HDD, 6 400-GB SAS SSD Network Controller Cisco UCS VIC 1227 2 10GE SFP+ Cisco UCS VIC 1227 2 10GE SFP+ Network and Cluster Scaling 2 Cisco UCS 6248UP FIs, Scale up to 32 servers with no additional switching infrastructure 2 Cisco UCS6248UP FIs, Scale up to 32 servers with no additional switching infrastructure Cisco Single SKU UCS-SL-CPA3-S (8 servers) UCS-SL-CPA3-H (8 Servers)
- 24. Starter Kits Testados e Homologados = Performance Previsível Testado e Validado
- 25. Do PoC de IoT à Integração no Data Center PoC Infra apartada Limited Production Performance, Gerencia, Continuidade, Etc. SAN/NAS Arrays Blade Servers Enterprise Platform Orquestração, IaaS, Panejamento de Capacidade
- 26. IoT WF Model x DD Model DD Offers Full Stack Solution IoT World Forum Model Dimension Data Solution Focus Collaboration Business & people process Application Report, Control Data Abstraction Aggregation and access Data accumulation Storage Edge Computing Data analisys & transformation Connectivity Communication & processing Physical Devices & Controllers Applications Distributed Computing Secure Conectivity Analytics Hybrid Cloud Computing
- 27. A Internet das Coisas ajudando nossos clientes a fazerem coisas incríveis
- 28. Luis Filipe Silva Senior Solutions Architect, Brazil Dimension Data Americas Phone: +55 (11) 3878 66549 +55 (21) 99858 9137 Filipe.silva@dimensiondata.com