Time Series Analysis Using an Event Streaming Platform
0 likes124 views
The document presents a virtual meetup by Dr. Mirko Kämpf on advanced time series analysis using an event streaming platform, focusing on data preparation and processing techniques for time series research, anomaly detection, and network analysis. It discusses the advantages of stream processing over batch processing and emphasizes the integration of data streams, algorithms, and decision-making within the confluent platform. Key topics include the challenges of data integration, the role of Kafka in facilitating complex event processing, and the importance of establishing a robust architecture for real-time analytics.
Time Series Analysis Using an Event Streaming Platform
- 1. Time Series Analysis Using an Event Streaming Platform Virtual Meetup - September 8-th 2020 Dr. Mirko Kämpf - Solution Architect @ Confluent, Inc. - Team CEMEA
- 2. Abstract Time Series Analysis using an Event Streaming Platform Advanced time series analysis (TSA) requires very special data preparation procedures to convert raw data into useful and compatible formats. In this presentation you will see some typical processing patterns for time series based research, from simple statistics to reconstruction of correlation networks. The first case is relevant for anomaly detection and to protect safety. Reconstruction of graphs from time series data is a very useful technique to better understand complex systems like supply chains, material flows in factories, information flows within organizations, and especially in medical research. With this motivation we will look at typical data aggregation patterns. We investigate how to apply analysis algorithms in the cloud. Finally we discuss a simple reference architecture for TSA on top of the Confluent Platform or Confluent cloud.
- 3. This presentation is about linking: - Time-Series-Analysis (TSA) - Network- or Graph-Analysis Confluent Platform - Complex Event Processing (CEP). Research work ends often with nice charts, scientific papers, and conference talks. But, many published results can’t be reproduced - often because the setup it is simply too complicated ... Question: How can we integrate data streams, experiments, and decision making better?
- 4. Why not using batch processing? Study anatomy … ● Batch processing is fine: ○ as long as your data doesn’t change. ○ in PoCs for method development in a lab. ○ for research in a fixed scope.
- 5. Why using Kafka? Study the anatomy … Study and influence the living system ... ● Stream processing is better: ○ for real time business in changing environments. ○ iterative (research) projects. ○ repeatable experiments on replayed data.
- 6. Content: (1) Intro Typical types of event How to identify hidden events? (2) The Challenge (3) Approach Time Series Analytics & Network Analytics in Kafka Create time series from events Create graphs from time series pairs (4) Demo (5) Architecture & Building Blocks: (6) Unified Domain Model
- 7. Events - 1 Business events - transaction records - discrete observation A Sale A Trade An Invoice A Customer Experience JUTS SIMPLE OBSERVATION & DATA CAPTURING How to handle events?
- 8. Events Events - 2 Well defined events - in known context Sometimes: SIMPLE Sometimes: DATA ANALYSIS How to identify events? directly observed “observed” during analysis of an episode
- 9. Univariate TSA: single episodes are processed - Extreme values - Patterns - Distribution of values - Fluctuation properties - Long-term correlations (memory effects)
- 10. Events Events - 3 Extreme Events - “outliers” in unknown context ADVANCED DATA ANALYSIS (& ML) How to handle?
- 11. Multivariate TSA: pairs / tuples of episodes are processed - Comparison Similarity measures for link creation
- 12. 1212 Reality is Complex: We should simplify a bit! Simplification in our method can lead to isolation: - DATA SILOS - OPERATIONAL SILOS SOLUTION: GRAPHS capture structure. TIME SERIES capture properties over time (history). relations as graph or matrix: objects in groups:
- 13. WHAT? WHY? Z value is a property of the network’s topology which evolves over time.
- 14. Special procedures are established. The events which make people & the market happy :-) Complex Event AnalysisEvent Driven Architecture Recap: IT Operations - Server crash - Cyber crime Business Events - Big deal won - Technical issue solved Transactions (in business) - orders placed - products shipped - bills paid Extreme Events: - Service slow down due to emerging bottlenecks - Increased demand in a resource What events are and how to process event-data can be misunderstood or simply unclear. It all depends on our view and our goals!
- 15. The Challenge: How can you combine unbound data assets and scientific methods? ● You bring data to a place where it can be processed easily, for example to a cloud system or into special purpose systems, such a event processing platform. ● You integrate important algorithms as early as possible in your processing pipeline to get results fast, using streaming processing capabilities.
- 16. Things can become complicated: Complex Event Analysis Integration Across Domains Extraction of Hidden Event
- 17. Complex Event Analysis - time series analysis and ML reveal hidden events - multi-stage processing is usually needed METHODOLOGY
- 18. Integration Across Domains - distributed event processing systems are used - apps consume and produce events of different flavors - event-types and data structures my change over time ORGANIZATION & OPERATIONS
- 19. Problems on ORGANIZATION level: Many legacy systems can’t be integrated without additional expensive servers. Often, this data is unreachable for externals. Business data is managed by different teams using different technologies. Data scientists use data in the cloud, and they all do really L❤VE notebooks. But often, they don’t use any automation.
- 20. Extraction of Hidden Events - requires Applied Data Analysis & Data Science - embedding of Complex Algorithms in IT landscape - integration of GPU/HPC and streaming data pipelines TECHNOLOGY & SCIENCE
- 21. Kafka and its Ecosystem … - are considered to be middleware, managed by IT people: - researchers do not plan nor build their experiments around this technology yet. - don’t offer ML / AI components: - many people think, that a model has to be executed on an edge-device or in the cloud.
- 22. Yes, Apache Kafka can support agile (data)experiments! - Kafka APIs give access to data (flows) in real time. - allows replay of experiments at a later point in time - Kafka allows variation of analysis without redoing a simulation or ingestion by simply reusing persisted event-streams again. - Kafka Streams and KSQL allow data processing in place: - this allows faster iterations, for example because plausibility checks can be done in place - the streaming API gives freedom for extension of core logic - DSL and KSQL will sav you a lot of implementation time
- 23. Why not building something great using the right tools ???
- 24. ADVANCED TIME SERIES ANALYSIS & NETWORK ANALYSIS … how does it work?
- 25. Network Reconstruction & Topology Analysis: The Approach
- 26. OpenTSx Network Reconstruction & Topology Analysis: A Standardized Event Processing Pipeline
- 27. ADVANCED TIME SERIES ANALYSIS & NETWORK ANALYSIS … and how does this fit into Kafka?
- 28. 28 The following slides will show how TSA concepts and Kafka concepts fit together.
- 30. Create Time Series from Event Streams: By Aggregation, Grouping, and Sorting Events / Observations event series time series
- 31. From Table of Events to - Time Series
- 32. Table Stream Duality ⇒ Time Series and Graphs A time series is a table of ordered observations in a fixed context. A graph can be seen as a table of node- and link- properties - stored in two tables.
- 33. Create Networks from Event Streams: By Aggregation, Grouping, and Sorting Events / Observations node properties link properties
- 34. Multi Layer Stream Processing: TSA to Reveal Hidden System Properties Events / Observations event series time series Node properties Link properties Node properties Link properties Static aspects of system topology Dynamic aspects of system topology Multivariate TSA Univariate TSA
- 35. Dynamic & Static aspects of system topology Complex Event Processing: For Complex Systems System Events & Emerging Patterns Node properties Link properties Topology Analysis
- 36. Use the Table-Network Analogy: Kafka Graphs https://github.com/rayokota/kafka-graphs large durable graphs: Persisted in Kafka topic
- 37. Sliding Windows: Define the Temporal Graphs In some use cases, we don’t want to keep the node and link data in topics: - nodes aren’t always linked - changes are very fast - focus on activation patterns, rather than on underlying structure It is fine to calculate the correlation links and the topology metrics on the fly, just for a given time window. t 1 t 2
- 38. Back to Streams
- 39. 39 Demo ...
- 40. The project will provide reusable streaming apps so that developers implement just their specific logic.
- 44. Finally, we can compose a complex flow ...
- 46. Demo: OpenTSx Generate some events and some episodes. Apply some time series processing procedures on: - a stream of episodes. - a stream of events Complex procedures are composed from a set of fundamental building blocks. Visualize the flows and dependencies. https://github.com/kamir/OpenTSx
- 48. Kafka Connect UDFs: Kafka Connect Data Assets We use 4 building blocks ... Source Connectors integrate sources … Legacy and Future Systems Sink Connectors integrate external targets … Special Purpose Systems Domain specific logic is implemented in small reusable components: Domain Driven Design Data flows are no longer transient. The event log acts as single source of truth. Paradigm Shift in Data Management
- 49. Confluent Platform Kafka Consumer API Kafka Connect KSQL & Kafka Streams application KSQL UDFs Streaming Applications Primary Data Kafka Producer API Kafka Connect Derived Data … for our Time Series Processing Platform
- 50. Summary: Because Kafka is a scalable & extensible platform it fits well for complex event processing in any industry on premise and in the cloud. Kafka ecosystem provides extension points for any kind of domain specific or custom functionality - from advanced analytics to real time data enrichment. Complex solutions are composed from a few fundamental building blocks:
- 51. What to do next? (A) Identify relevant main flows and processing patterns in your project. (B) Identify or implement source / sink connectors and establish 1st flow. (C) Implement custom transformations as Kafka independent components. (D) Integrate the processing topology as Kafka Streams application: (a) Do you apply standard transformations and joins (for enrichment)? (b) Is a special treatment required (advanced analysis)? (c) Do you need special hardware / external services (AI/ML for classification)? (E) Share your connectors and UDFs with the growing Kafka community. (F) Iterate, add more flows and more topologies to your environment.