Why And When Should We Consider Stream Processing In Our Solutions Teqnation 2023

Editor's Notes

• #3: What is Stream Processing ? Why We Should Learn It ?
• #5: Developer By Day, Furniture Assembelr By NightI learned that using Right tool is the most important part of assembling
• #6: Question 1: Who has heard these technologies a lot ? Question 2: Who has used this technologies in production ? Everyday that we wake up, we hear some new Apache technologies ..
• #9: Okay, Not for me I'm not fan of complex definitions. let's get to a simple definition
• #10: reading data multiple source processing Data itself. payload itself individually or joined with other data sending out to another system
• #11: Event processing is a technique that focuses on listening for specific events or patterns of events within a system, enabling decision-making and triggering actions based on the information contained in the events.
• #12: Services communicates with Events
• #17: We need to chunk the data to make it feasible to process
• #19: Bounded Stream Example : Processing list of last month records for Train Check in – Checkout for Analysis purpose
• #20: 1 Minute : You are watching netflix on Airplane / Subway . Your actions will be synced afterward
• #21: We have three type of guarantees, no gurantee , at least one delivery, exactly once deliveryFlink -> Checkpointing Don’t forget to check learn more
• #22: Ok, wait. Hold your horse , So you said a lot of definitions, what is the usecase ..
• #23: Transforms are Filter , Map , Aggregate , Join, Custom Functions
• #24: 30s – 1 Minute
• #25: 1 minute
• #26: 1 minute We cannot carry two watermelon with one hand We need to chunk the data to make it feasible to processOk, right. We should devide . but how we are going to divide the data ?
• #27: It’s very similar to a shuttle, isn’t it ?
• #28: Let’s imagine that we are receiving request logs
• #29: Watching Video on in the Subway or during the flight Phone Call How Stream Processing can do this ? Session Window is based on Group By Key
• #31: 1 Minute : Thing that we need to learn, they are too much. So we make it easier by Examples ! How can we do it in our current applications, without Stream processing frame works ?
• #33: Some times we need to store some data, and later looking back to stored data similar to what we used to do with Redis / Database.
• #35: Key By is most commong Transformation partition the data stream similar to group by in SQL Some times we need to group some of the data together Some times it may cause a network shuffle that will partition the stream on different nodes
• #36: 5 minute
• #38: val failedLogins = p.apply("Read PubSub Messages", readFromPubSubSubscription()) val ipCounts = failedLogins .apply("Window", failedLoginWindowingStrategy()) .apply("Map to KV <IP,MSG>", mapToKVIPAddr()) .apply("Group by Key IP-Addr", GroupByKey.create()) .apply("Count per IP", countNumberOfAttempts()) val alerts = ipCounts .apply("Filter by Threshold", isCountOfAttempAboveThresholdFilter()) .apply("Enrich with Old Breaches Last Month", enrichWithOldBreachesLastMonth()) alerts.apply("Write Alerts to PubSub", publishToPubSubTopic())
• #39: val failedLogins = p.apply("Read PubSub Messages", readFromPubSubSubscription()) val ipCounts = failedLogins .apply("Window", failedLoginWindowingStrategy()) .apply("Map to KV <IP,MSG>", mapToKVIPAddr()) .apply("Group by Key IP-Addr", GroupByKey.create()) .apply("Count per IP", countNumberOfAttempts()) val alerts = ipCounts .apply("Filter by Threshold", isCountOfAttempAboveThresholdFilter()) .apply("Enrich with Old Breaches Last Month", enrichWithOldBreachesLastMonth()) alerts.apply("Write Alerts to PubSub", publishToPubSubTopic())
• #40: Stream Processing Applications and especially when you start to have Stateful functions are not really easy.
• #46: Complexity Handling out-of-order events, windowing, and state management Increased complexity compared to batch processing Implementation and Maintenance Expertise required in distributed systems, fault tolerance, and specific stream processing frameworks Maintenance effort for business logic and data flow changes Testing and Debugging Complex testing scenarios and simulation of various events and failures Difficulties in debugging due to real-time and distributed nature of processing Error Handling Managing errors and edge cases can be challenging Recovery mechanisms and failure scenarios require careful consideration Data Consistency Ensuring exactly-once processing and data consistency can be challenging Requires robust handling of distributed systems and failures Learning Curve and Project Timeline 2-3 months for a medior developer to become proficient 4-6 months for a project to reach stability from start Resource Intensiveness Real-time processing may consume more resources than batch processing Cloud services can help mitigate infrastructure costs
• #47: In Short Stream Data Integration is Map Transform Filter Enrich Stream Data Integration is also using States , Windowing , State Management, Event Pattern
• #48: Learning Curve Stream Data Integration : 1 – 2 weeks Stream Analytics: 2 – 3 months For not very basic project, expect 2-4 months from project initiation to reach stability It’s not easy to find developers with extensive stream processing experience. For most of Stream processing frameworks, there are not many step by step documentation & stack overflow questions with working answers. You need to connect the dots yourself. Decent community support available, but not as extensive as Spring or other popular frameworks Stream processing can be resource-intensive, ( Cloud services helps us here )
• #51: Case Stream Data Integration: (Map, Filter, Basic Enrichment)You are not getting much out of using Stream processing frameworks. You can achieve almost same results with other tools with possibility to scale up.Case Stream Analytics : You should start investing on your stream processing solution and building a team by help of professional consultants to lead/faciliate/boost the process. In the mean time, you can use other available tools to support part of your business requirements. ( Like BigQuery, Monitoring tools)
• #52: Case Stream Data Integration: (Map, Filter, Basic Enrichment)You are not getting much out of using Stream processing frameworks. You can achieve almost same results with other tools with possibility to scale up.Case Stream Analytics : You should start investing on your stream processing solution and building a team by help of professional consultants to lead/faciliate/boost the process. In the mean time, you can use other available tools to support part of your business requirements. ( Like BigQuery, Monitoring tools)
• #53: Case Stream Data Integration: (Real time ETL) You are not getting much out of using Stream processing frameworks. You can achieve almost same results with other tools with possibility to scale up.Case Stream Analytics : You should start investing on your stream processing solution and building a team by help of professional consultants to lead/faciliate/boost the process. In the mean time, you can use other available tools to support part of your business requirements. ( Like BigQuery, Monitoring tools)
• #59: Anomaly detection: Stream processing can help identify unusual patterns or behaviors in IoT device data, enabling early detection of potential issues or failures. For example, it can be used to monitor sensor data from industrial equipment or vehicles to detect anomalies that may indicate a malfunction or maintenance need. Smart home automation: In a smart home environment, stream processing can be used to analyze data from various sensors and devices to trigger automated actions, such as adjusting lighting or temperature based on occupancy, time of day, or user preferences. Fleet management: Stream processing can analyze data from GPS trackers, vehicle sensors, and other devices in real-time to optimize fleet operations. This may include route planning, vehicle maintenance scheduling, fuel efficiency analysis, or driver behavior monitoring. Environmental monitoring: IoT devices can be deployed to monitor various environmental parameters, such as air quality, water levels, or temperature. Stream processing can be used to analyze this data in real-time, enabling rapid response to environmental changes or potential hazards. Energy management: Stream processing can be used to analyze energy consumption data from smart meters, IoT devices, and sensors in real-time, helping to optimize energy usage and reduce costs. This can be applied to smart grids, microgrids, or individual buildings. Predictive maintenance: By analyzing IoT sensor data in real-time, stream processing can help predict when a machine or equipment may require maintenance or is likely to fail. This allows for proactive maintenance scheduling, reducing downtime and increasing operational efficiency.