Complex Event Processing: What?, Why?, How?
15 likes3,321 views
The document discusses Complex Event Processing (CEP), explaining that CEP is an emerging technology that allows processing of streaming event data in real-time to detect complex patterns and situations. Several use cases for CEP are presented, including algorithmic trading, fraud detection, and location-based services. The document also covers CEP adoption paths and debates around CEP, such as whether it is mature enough and how it relates to other technologies.
![Event Processing Simple, Streaming, Complex / Business EP Simple Route, Filter, Enrich Streaming min,max,avg Sliding data windows Complex / Business Many streams Causality A followed by B within t minutes Different rates Historical data and real time data Consumer/Producer, JMS, ESBs, EIP Stateless & Timeless How? « Event Stream Processing » Statefu-less / Timefu-less How?? Continuous joins and indexing Time as first class citizen Temporal requests Statefull / Timefull When [CEP] then […]](https://image.slidesharecdn.com/uesiv4-1226411802984305-8/85/Complex-Event-Processing-What-Why-How-6-320.jpg)
![CEP’s Friends Edge Produce zillions of events SOA Interconnect, mediate, leverage and reuse XTP Scale throughput and data volumes as required Right-time down to sub-millisecond where needed BAM & BI (or back to SOA) Operational applications CEP Situational & Actionable Awareness Rules engine When [CEP] then […] If … then […]](https://image.slidesharecdn.com/uesiv4-1226411802984305-8/85/Complex-Event-Processing-What-Why-How-7-320.jpg)
![#1: Algorithmic trading Volume Weighted Average select ticker, avg (price), count (*), sum (price) from MarketData(ticker=‘GOOG'). win:length (1000) Example CEP server performance figures 1 CEP server, Java based, TCP in/out 1000 CEP statements 500 000 evt/s [100 Mbit/s] Latency: avg 2.5 µ s, 99% < 10 µ s 4 core 2GHz, 65% load avg Simple Event Processing Stream Processing (Timefull)](https://image.slidesharecdn.com/uesiv4-1226411802984305-8/85/Complex-Event-Processing-What-Why-How-10-320.jpg)
![#2: Fraud and compliance, risk management Context FX trading Event { Trade direction, amount, user id, account, currencies traded } 200 000 evt/day, 50 rules, ~5% of triggering Two (or more) buy trades followed by ( ) a sale within the next hour between 2 parties select e1, e2, final from pattern [ every e1=FXTrade(userId in (100,101)) (e2=FXTrade(userId in (100,101), dir = e1.dir) where timer:within (8 hours ) ) final=FXTrade(userId in (100,101), dir != e1.dir) where timer:within (1 hour ) ]](https://image.slidesharecdn.com/uesiv4-1226411802984305-8/85/Complex-Event-Processing-What-Why-How-12-320.jpg)
![#3: Location based services Context: Mobile asset management (Command & Control systems) Tracking delay between real time position and planned position 20% time left to reach destination, 10% delta from planned position Continuous join with reference (passive) data select id, lat, lon, time-ref.time, Utils.distance (lat,lon,ref.lat,ref.lon) as distance from GPSPosition, sql:RefDB [ select lat, lon, time from Plan where id={id} and stage=‘JFK’ ] as ref where time<0.8*ref.time and Utils.distance(lat,lon,ref.lat,ref.lon)>1.1*SPEED*(0.8*ref.time-time)](https://image.slidesharecdn.com/uesiv4-1226411802984305-8/85/Complex-Event-Processing-What-Why-How-13-320.jpg)
![Adoption: Risks & Benefits ? “ CEP is mature? CEP is really not ESP? CEP is really event-driven SOA? CEP is really real-time BI? CEP is really low latency, high throughput, white-box COTs algo trading? CEP is really not a type of BPM? CEP is not really for detecting complex events? Complex does not really mean complex?” [as seen on blogs] Use case driven Build vs Buy SOA/BI/BAM related Orthogonal processing (discover & monitor) vs path critical processing (act upon)](https://image.slidesharecdn.com/uesiv4-1226411802984305-8/85/Complex-Event-Processing-What-Why-How-18-320.jpg)
Complex Event Processing: What?, Why?, How?
- 1. Complex Event Processing Quoi, Pourquoi, Comment? Fabien Coppens Alexandre Vasseur ©OCTO Technology – Université d’été du Système d’information
- 2. Objectives Understand what Complex Event Processing (CEP) is Where does CEP fit in SOA/EDA/BAM/BI/RDBMS Learn about CEP use cases Discover CEP adoption paths
- 3. Speaker Qualifications Fabien Coppens Senior Architect CEP user Alexandre Vasseur Middleware technology evangelist Esper co-lead & early CEP fan
- 4. The « Right-Time » Enterprise Business is data-driven More data, more sources More complex situational scenarios Right-time is A business imperative A technology challenge Strong coupling! Request/Response! Data stored and later analyzed Moving requirements Situation Action/Decision Events Time Push Loosely coupled Time & Event-Driven
- 5. « By 2011, a new generation of application platforms stemming from the convergence of extreme transaction processing (XTP) technologies will supersede Java EE and .NET as the platform of choice for large-scale, business-critical operational applications .“ Gartner, July 2007 SOA + EDA + XTP + Edge + BAM/BI Convergence: Event Processing Financial Services Fraud detection RFID BAM & monitoring Sensors Location/presence based services Security Large scale SOAs Right-Time + Events
- 6. Event Processing Simple, Streaming, Complex / Business EP Simple Route, Filter, Enrich Streaming min,max,avg Sliding data windows Complex / Business Many streams Causality A followed by B within t minutes Different rates Historical data and real time data Consumer/Producer, JMS, ESBs, EIP Stateless & Timeless How? « Event Stream Processing » Statefu-less / Timefu-less How?? Continuous joins and indexing Time as first class citizen Temporal requests Statefull / Timefull When [CEP] then […]
- 7. CEP’s Friends Edge Produce zillions of events SOA Interconnect, mediate, leverage and reuse XTP Scale throughput and data volumes as required Right-time down to sub-millisecond where needed BAM & BI (or back to SOA) Operational applications CEP Situational & Actionable Awareness Rules engine When [CEP] then […] If … then […]
- 8. CEP ~ Database 2.0 ? RDBMS Store data (a lot) Handle queries SQL (ie abstracted) Request/Response Concept of Time Right time CEP Store rules Handle data EPL: Event Processing Language Subscribe/Notify Time & causality « Continuous query » SQL Queries CEP Server EPL Queries CEP Database
- 9. Event Processing Language & CEP Server EPL: For right-time events, what SQL is to historical data Best practices, rapid prototyping, tooling, less custom code CEP server: Middleware to deal with recurrent complexity indexing, state management, scalability, management, connectivity etc ? CEP Application Source + CEP as EPL + Sink Is EPL expressive enough (build vs buy) Is EPL a standard (vs SQL) Is it a specialized RDBMS or a true engine? Does it require a RDBMS? Manageable? Scalable Interoperate? Time & Causality Filter, Aggregate Sliding windows Continuous execution State management, threading Transports In & Out Interrop. Dashboards High Availability Runtime Management Record / Replay / Test
- 10. #1: Algorithmic trading Volume Weighted Average select ticker, avg (price), count (*), sum (price) from MarketData(ticker=‘GOOG'). win:length (1000) Example CEP server performance figures 1 CEP server, Java based, TCP in/out 1000 CEP statements 500 000 evt/s [100 Mbit/s] Latency: avg 2.5 µ s, 99% < 10 µ s 4 core 2GHz, 65% load avg Simple Event Processing Stream Processing (Timefull)
- 11. #1: Build vs Buy? MarketData(GOOG, 50) MarketData(MSFT, 25) MarketData(GOOG, 30) MarketData(GOOG, 40) Avg price / last 2 ticks, per ticker GOOG:40 GOOG:35 Receive market data Output / trigger actions Functional specification? Abstraction for user defined rules? Performance & monitoring etc? list = new MyList(GOOG,2) /*2 = size*/ // on event list.add(tick) // some tick discarded // as list size was // harcoded with size 2 //TODO: do not hardcode // recompute avg //TODO: optimize this send computeAvg(list)
- 12. #2: Fraud and compliance, risk management Context FX trading Event { Trade direction, amount, user id, account, currencies traded } 200 000 evt/day, 50 rules, ~5% of triggering Two (or more) buy trades followed by ( ) a sale within the next hour between 2 parties select e1, e2, final from pattern [ every e1=FXTrade(userId in (100,101)) (e2=FXTrade(userId in (100,101), dir = e1.dir) where timer:within (8 hours ) ) final=FXTrade(userId in (100,101), dir != e1.dir) where timer:within (1 hour ) ]
- 13. #3: Location based services Context: Mobile asset management (Command & Control systems) Tracking delay between real time position and planned position 20% time left to reach destination, 10% delta from planned position Continuous join with reference (passive) data select id, lat, lon, time-ref.time, Utils.distance (lat,lon,ref.lat,ref.lon) as distance from GPSPosition, sql:RefDB [ select lat, lon, time from Plan where id={id} and stage=‘JFK’ ] as ref where time<0.8*ref.time and Utils.distance(lat,lon,ref.lat,ref.lon)>1.1*SPEED*(0.8*ref.time-time)
- 14. #4: Trading as SaaS : Strateer Algo trading as SaaS CEP is a core enabler Users can configure their own trading strategies ~20 CEP rules templates provided Each configurable by end users Power users can create their own rules Esper CEP + JMS + MarketData connectors + RIA
- 15. #4: Strateer use case Strateer is algorithmic trading for individual investors 1. Drag & Drop your own automated strategy 2. Test it the way fund managers do 3. Then run it on Strateer: 1000s of data sources, real-time engine. 4. Get trading alerts to you anytime, anywhere. 5. Share and compare results with community
- 17. Solutions (excerpt) Single use case platform (trading, compliance, BI ..) xxx Event Processing Mandates a DB? Latency / Throughput Of the shelf? Proven / Innovating Maturity Cost (Buy + Learn + Operate) Event Processing = Middleware x = Simple / Stream / Complex / Rules In-memory vs hidden DB vs DB 1 to 1000 if not more range Interoperate? Open platform? … … … ?
- 18. Adoption: Risks & Benefits ? “ CEP is mature? CEP is really not ESP? CEP is really event-driven SOA? CEP is really real-time BI? CEP is really low latency, high throughput, white-box COTs algo trading? CEP is really not a type of BPM? CEP is not really for detecting complex events? Complex does not really mean complex?” [as seen on blogs] Use case driven Build vs Buy SOA/BI/BAM related Orthogonal processing (discover & monitor) vs path critical processing (act upon)
- 19. Links http://esper.codehaus.org http://www.espertech.com http://complexevents.com And many other googling around or asking your favorite vendors (!)