Addressing Challenges with IoT Edge Management
Download as PPTX, PDF0 likes359 views
The document discusses the significance of IoT edge management, highlighting its $1.13 trillion market potential in 2021, with major spending across regions. It emphasizes the challenges of deploying edge agents and managing data flows to enable real-time insights from IoT devices. The introduction of Cloudera Edge Management (CEM) is presented as a solution to oversee edge data collection and analytics efficiently.
Addressing Challenges with IoT Edge Management
- 1. © Cloudera, Inc. All rights reserved. ADDRESSING CHALLENGES WITH IOT EDGE MANAGEMENT Dinesh Chandrasekhar Director of Product Marketing, Data-in-Motion
- 2. © Cloudera, Inc. All rights reserved. 2© Cloudera, Inc. All rights reserved. Public Sector Transportation Utilities Healthcare Manufacturing Retail COMMON IOT USE CASES BY INDUSTRY Fleet Management Connected Cars Smart Cities Predictive Analytics Inventory/ Material Tracking • IoT is a $1.13T market opportunity in 2021. • Americas - $329B IoT spending. Manufacturing and Transportation are top industries, accounting for 26% of total spending. • APAC - $500B IoT spending. Manufacturing, Utilities and Transportation are top industries. • EMEA - $264B IoT spending. Manufacturing is top industry, powered by Industry 4.0 initiatives. • Worldwide IoT Analytics and Information Management Market = $573M Top 5 Use cases Utility Monitoring Predictive Maintenance Patient Monitoring Usage-based Insurance Asset Tracking / Monitoring Edge Data Collection
- 3. © Cloudera, Inc. All rights reserved. 3© Cloudera, Inc. All rights reserved. WHAT IS THE EDGE?
- 4. © Cloudera, Inc. All rights reserved. 4© Cloudera, Inc. All rights reserved. WHY IS EDGE DATA COLLECTION IMPORTANT? Need for real-time insights ETL process is not real-time Data provenance
- 5. © Cloudera, Inc. All rights reserved. 5© Cloudera, Inc. All rights reserved. Data Flow Apps Powered by NiFi STREAMING REFERENCE ARCHITECTURE Freightliner Fleet Trucks Truck Sensors C++ Agent Mack Fleet Trucks Truck Sensors C++ Agent Tata Fleet Trucks Truck Sensors C++ Agent Subcribing Streaming Analytics Apps Analytics App 4 Analytics App 1 Analytics App 2 Micro Services Analytics App 3
- 6. © Cloudera, Inc. All rights reserved. 6© Cloudera, Inc. All rights reserved. WHAT ARE EDGE AGENTS?
- 7. © Cloudera, Inc. All rights reserved. 7© Cloudera, Inc. All rights reserved. KEY CHALLENGES AT THE EDGE Deploying edge agents Identifying classes of agents Updates Device compatibility Edge storage availability Low power
- 8. © Cloudera, Inc. All rights reserved. 8© Cloudera, Inc. All rights reserved. WHAT IS AN EDGE MANAGEMENT HUB? Management and monitoring of agents Application support OTA updates Registry Eventing
- 9. © Cloudera, Inc. All rights reserved. 9© Cloudera, Inc. All rights reserved. Data Flow Apps Powered by NiFi MANAGING EDGE DATA COLLECTION IS A HARD PROBLEM Freightliner Fleet Trucks Truck Sensors C++ Agent Mack Fleet Trucks Truck Sensors C++ Agent Tata Fleet Trucks Truck Sensors C++ Agent Subcribing Streaming Analytics Apps Analytics App 4 Analytics App 1 Analytics App 2 Micro Services Analytics App 3 Challenge #1: Developing Edge Flows How do I develop flows meant to run on the edge using NiFi-like flow based programming with no code? Challenge #2: Deploying Flows/ Over the Air Updates (OTA) How do I deploy data collection flows to the edge? How do I do updates to only the Freightliner class of trucks? Challenge #3: Monitoring Agents and Flows How do I manage hundreds of thousands of edge agents and the flows that they are running
- 10. © Cloudera, Inc. All rights reserved. 10© Cloudera, Inc. All rights reserved. KEY CHALLENGES WITH EDGE MANAGEMENT Network availability Offline storage Agent heartbeat
- 11. © Cloudera, Inc. All rights reserved.
- 12. © Cloudera, Inc. All rights reserved. 12© Cloudera, Inc. All rights reserved. CLOUDERA DATAFLOW
- 13. © Cloudera, Inc. All rights reserved. 13© Cloudera, Inc. All rights reserved. EDGE MANAGEMENT • Edge data collection powered by Apache MiNiFi • MiNiFi – smaller footprint than NiFi • Guaranteed delivery • Data buffering • Prioritized queuing • Flow-specific QoS • Data provenance • Designed for extension • C++ / Java agents • TensorFlow support • Designed for IoT
- 14. © Cloudera, Inc. All rights reserved. 14 INTRODUCING CLOUDERA EDGE MANAGEMENT (CEM) What is CEM? New product offering is an edge management solution consisting of edge agents + edge management hub. What does CEM do? Manage, Control and Monitor edge agents to collect data from edge devices and push intelligence back to the edge. How does CEM do it? Develop, Deploy, Run & Monitor edge flow apps on thousands of edge devices. N ew
- 15. © Cloudera, Inc. All rights reserved. 15© Cloudera, Inc. All rights reserved. CLOUDERA DATAFLOW
- 16. © Cloudera, Inc. All rights reserved. 16© Cloudera, Inc. All rights reserved. FLOW MANAGEMENT • Web-based user interface • Highly configurable • Out-of-the-box data provenance • Designed for extensibility • Secure • NiFi Registry • DevOps support • FDLC • Versioning • Deployment
- 17. © Cloudera, Inc. All rights reserved. 17© Cloudera, Inc. All rights reserved. 280+ PROCESSORS FOR DEEPER ECOSYSTEM INTEGRATION Hash Extract Merge Duplicate Scan GeoEnrich Replace ConvertSplit Translate Route Content Route Context Route Text Control Rate Distribute Load Generate Table Fetch Jolt Transform JSON Prioritized Delivery Encrypt Tail Evaluate Execute All Apache project logos are trademarks of the ASF and the respective projects. Fetch HTTP Syslog Email HTML Image HL7 FTP UDP XML SFTP AMQP WebSocket
- 18. © Cloudera, Inc. All rights reserved. 18© Cloudera, Inc. All rights reserved. CLOUDERA DATAFLOW
- 19. © Cloudera, Inc. All rights reserved. Streaming Analytics Reference Architecture Data Flow Apps Powered by NiFi Kafka is Everywhere. Critical Component of Streaming Architectures Kafka Producers Kafka Topics Kafka TopicsKafka Consumers & Producers Kafka Consumers US West Fleet Truck Sensors C++ Agent US Central Fleet Truck Sensors C++ Agent US East Fleet Truck Sensors C++ Agent Analytics App 1 Analytics App 2 Analytics App 5 Analytics App 3 Analytics App 4
- 20. © Cloudera, Inc. All rights reserved. Cloudera Streams Messaging Manager (SMM) What is SMM? Kafka Management and Monitoring tool Cure the “Kafka Blindness” Single Monitoring Dashboard for all your Kafka Clusters across 4 entities – Broker – Producer – Topic – Consumer REST as a First Class Citizen Alerting Schema Management Integration with Schema Registry
- 21. © Cloudera, Inc. All rights reserved. 21© Cloudera, Inc. All rights reserved. CLOUDERA DATAFLOW
- 22. © Cloudera, Inc. All rights reserved. 22© Cloudera, Inc. All rights reserved. STREAMING ANALYTICS • Pattern matching • Predictive and Prescriptive Analytics • Complex Event Processing • Continuous & Real-time Insights
- 23. © Cloudera, Inc. All rights reserved. OLAP Access PatternSQL Access Pattern Streaming Event Storage Substrate Topic A Kafka Topic Kafka Topic Topic B Kafka Topic Topic C Kafka Topic Topic D Kafka Topic Topic X 3 KafkaAnalyticsAccess Patterns Streaming Access Pattern N ew KAFKA SQL New KAFKA OLAP New
- 24. © Cloudera, Inc. All rights reserved. 24© Cloudera, Inc. All rights reserved. CLOUDERA DATAFLOW
- 25. © Cloudera, Inc. All rights reserved. 25© Cloudera, Inc. All rights reserved. ENTERPRISE SERVICES • Provisioning • Management • Monitoring • Unified Security • Single Sign-on • Audit • Compliance • Edge-to-Enterprise Governance
- 26. © Cloudera, Inc. All rights reserved. 26© Cloudera, Inc. All rights reserved. CLOUDERA DATAFLOW
- 27. © Cloudera, Inc. All rights reserved. THANK YOU
Editor's Notes
- #13: Data ingestion, transformation and routing done visually with no code using Apache NiFi & 260+ processors Build streaming apps and analytics from edge to datalake / EDW using builder Enable edge data collection and intelligence through MiNiFi agents Support massive IoT infrastructures Deliver perishable insights with pattern matching and Complex Event Processing (CEP) from real-time streams Manage, monitor, secure and govern streaming data
- #16: Data ingestion, transformation and routing done visually with no code using Apache NiFi & 260+ processors Build streaming apps and analytics from edge to datalake / EDW using builder Enable edge data collection and intelligence through MiNiFi agents Support massive IoT infrastructures Deliver perishable insights with pattern matching and Complex Event Processing (CEP) from real-time streams Manage, monitor, secure and govern streaming data
- #17: Web-based user interface Design, control, feedback & monitoring Highly configurable Loss tolerant vs guaranteed delivery Low latency vs high throughput Dynamic prioritization Flow can be modified at runtime Back pressure Data provenance Track dataflow from beginning to end Designed for extension Build your own processors Secure SSL, SSH, HTTPS, etc.
- #19: Data ingestion, transformation and routing done visually with no code using Apache NiFi & 260+ processors Build streaming apps and analytics from edge to datalake / EDW using builder Enable edge data collection and intelligence through MiNiFi agents Support massive IoT infrastructures Deliver perishable insights with pattern matching and Complex Event Processing (CEP) from real-time streams Manage, monitor, secure and govern streaming data
- #22: Data ingestion, transformation and routing done visually with no code using Apache NiFi & 260+ processors Build streaming apps and analytics from edge to datalake / EDW using builder Enable edge data collection and intelligence through MiNiFi agents Support massive IoT infrastructures Deliver perishable insights with pattern matching and Complex Event Processing (CEP) from real-time streams Manage, monitor, secure and govern streaming data
- #25: Data ingestion, transformation and routing done visually with no code using Apache NiFi & 260+ processors Build streaming apps and analytics from edge to datalake / EDW using builder Enable edge data collection and intelligence through MiNiFi agents Support massive IoT infrastructures Deliver perishable insights with pattern matching and Complex Event Processing (CEP) from real-time streams Manage, monitor, secure and govern streaming data
- #26: Web-based user interface Design, control, feedback & monitoring Highly configurable Loss tolerant vs guaranteed delivery Low latency vs high throughput Dynamic prioritization Flow can be modified at runtime Back pressure Data provenance Track dataflow from beginning to end Designed for extension Build your own processors Secure SSL, SSH, HTTPS, etc.
- #27: Data ingestion, transformation and routing done visually with no code using Apache NiFi & 260+ processors Build streaming apps and analytics from edge to datalake / EDW using builder Enable edge data collection and intelligence through MiNiFi agents Support massive IoT infrastructures Deliver perishable insights with pattern matching and Complex Event Processing (CEP) from real-time streams Manage, monitor, secure and govern streaming data