Streaming Analytics and Internet of Things - Geesara Prathap
1 like535 views
This document discusses streaming analytics and the Internet of Things. It describes some challenges of IoT such as data volume, processing speed requirements, and data storage needs. It then discusses WSO2's IoT and analytics platforms, which can perform real-time, batch, and predictive analytics on streaming IoT and other data. The analytics platform uses Siddhi to analyze event streams using techniques like filtering, windowing, pattern detection, joins, and more. Real-time analytics examples include alerts, counting, correlation, and learning models.
Streaming Analytics and Internet of Things - Geesara Prathap
- 1. October 2016 Streaming Analytics and Internet of Things Geesara Prathap(geesara@wso2.com)
- 2. Challenges 2 • How fast do we need results? • How much data to keep? • Common language ? • Do we have centralized data storage and processing units? • Knowledge of the past data only?
- 3. ** IoT is not new !! Source: http://community.arm.com/groups/internet-of-things/blog/2014/06
- 6. Analytics Platform 6 WSO2 Analytics platform uniquely combine simultaneous real time and batch analytics with predictive analytics to run data from IoT, mobile, and web apps into actionable insights.
- 8. Analytics Strategy 8 Single platform to address all analytics styles. Batch Analytics: analytics on data at-rest, running typically every hour or every day, and focused on historical analytics dashboards and reports Real time Analytics: analyze event streams in real-time and detects patterns and conditions Predictive Analytics: leverages machine learning to create a mathematical model allowing to predict future behavior. Interactive Analytics: execute queries on the fly on top of data at rest.
- 9. IoT / Edge Analytics 9
- 10. Streaming Analytics in Other Words 10 ● Gather data from multiple sources ● Correlate data streams over time ● Find interesting occurrences ● Notify
- 11. Basic Building Blocks 11 ● Receivers: Data collection point, associated to a specific data connector ● Publishers: Data publishing point, associated to a specific data connector ● Event Streams: Event data flowing through the system ● Execution Plans: Execution pipeline applied to event streams ● Siddhi: Codename for the streaming engine ● Siddiqi: SQL-like query language
- 14. Event Streams 14 ● Event stream is a sequence of events ● Event streams are defined by stream definition ● Event streams have inflows and outflows ● Inflows can be from ○ Event receivers ○ Execution plans ● Outflows are to ○ Event publishers ○ Execution plans
- 15. Data Connectors 15 ● The following connectors are available out of the box Source: Email, File, JMS, Kafka, MQTT, SOAP, Websocket, Thrift, Binary, Log and JMX receiver Sink: RDBMS, Cassandra, SMS, Email, File, HTTP, JMS, Kafka, MQTT, SOAP, Websocket, Thrift, Binary ● Incoming/ outgoing data can be mapped using XPath, regular expressions, or JSON paths ● Data connectors are common across the analytics platform
- 16. Real-time Analytics Patterns ● Simple counting (e.g. failure count) ● Counting with Windows (e.g. failure count every hour) ● Preprocessing: filtering, transformations, (e.g data cleanup) ● Alerts, thresholds (e.g Alarm on high temperature) ● Data correlation, Detect missing events detecting erroneous data( e.g detecting failed sensors) ● Joining event streams (e.g. detect a hit on soccer ball) ● Merge with data in a database, collect update data conditionally
- 17. Real-time Analytics Patterns ● Detecting event sequence patterns( e.g. small transaction followed by large transaction) ● Tracking - follow some related entity’s state in space, time etc. (e.g location of airline baggage, vehicle, tracking wild life) ● Detect trends- Rise, turn, fall, outliers, Complex trends like triple bottom etc., (e.g algorithmic trading, SLA, load balancing) ● Learning a model (e.g. predictive maintenance) ● Predicting next value and corrective actions (e.g automated car)
- 18. CEP = SQL for Real-time Analytics ● Easy to follow from SQL ● Expressive, short, and sweet ● Define core operations that covers 90% of problems ● Let’s experts dig in when they like! Let’s look at the core operation
- 19. Operators: Filters Assume a temperature stream Here weather: convertFtoC() is a user defined function. They are used to extend the language Usecases: - Alerts, thresholds, (e.g Alarm on high temperature) - Preprocessing: filtering, transformation (e.g data cleanup)
- 20. Operators: Windows and Aggregation Support many window types - Batch windows, Sliding windows, Custom windows Usecases - Simple counting ( e.g failure count) - Counting with Windows ( e.g failure count every hour)
- 21. Operators: Patterns Models a followed by relation: e.g. event AS followed by event B Very powerful tool for tracking and detecting patterns Usecases - Detecting event sequence patterns - Tracking - Detect trends
- 22. Operators: Joins Models a followed by relation: e.g. event AS followed by event B Very powerful tool for tracking and detecting patterns Usecases - Detecting event sequence patterns - Tracking - Detect trends
- 25. CONTACT US !