Scalable Event Processing with WSO2CEP @ WSO2Con2015eu
6 likes1,790 views
This document provides an overview of complex event processing (CEP) and the WSO2 CEP product. CEP involves analyzing streams of event data for patterns that match predefined rules. It is useful when dealing with large volumes of data that are not needed long-term and time-critical analysis is required. The WSO2 CEP product supports distributed, scalable, and real-time complex event processing using the Siddhi query language. It allows defining queries with filters, windows, aggregations, and patterns to detect opportunities, threats, and anomalies. New features in version 4.0 include improved distributed processing, high availability, and support for dashboards, websockets, and MQTT. Common use cases for CEP include
![define stream StockStream
(symbol string, volume int, price double);
@name(‘Filter Query’)
from StockStream[price > 75]
select *
insert into HighPriceStockStream ;
@name(‘Window Query’)
from HighPriceStockStream#window.time(10 min)
select symbol, sum(volume) as sumVolume
insert into ResultStockStream ;](https://image.slidesharecdn.com/scalableeventprocessingwithwso2cep-wso2con2015eu3-150603105103-lva1-app6892/85/Scalable-Event-Processing-with-WSO2CEP-WSO2Con2015eu-28-320.jpg)
![define stream StockStream
(symbol string, volume int, price double);
@name(‘Filter Query’)
from StockStream[price > 75]
select *
insert into HighPriceStockStream ;
@name(‘Window Query’)
partition with (symbol of HighPriceStockStream)
begin
from HighPriceStockStream#window.time(10 min)
select symbol, sum(volume) as sumVolume
insert into ResultStockStream ;
end;](https://image.slidesharecdn.com/scalableeventprocessingwithwso2cep-wso2con2015eu3-150603105103-lva1-app6892/85/Scalable-Event-Processing-with-WSO2CEP-WSO2Con2015eu-29-320.jpg)
![define stream StockStream
(symbol string, volume int, price double);
@name(Filter Query’)
@dist(parallel= ‘3')
from StockStream[price > 75]
select *
insert into HightPriceStockStream ;
@name(‘Window Query’)
@dist(parallel= ‘2')
partition with (symbol of HighPriceStockStream)
begin
from HighPriceStockStream#window.time(10 min)
select symbol, sum(volume) as sumVolume
insert into ResultStockStream ;
end;](https://image.slidesharecdn.com/scalableeventprocessingwithwso2cep-wso2con2015eu3-150603105103-lva1-app6892/85/Scalable-Event-Processing-with-WSO2CEP-WSO2Con2015eu-30-320.jpg)
Scalable Event Processing with WSO2CEP @ WSO2Con2015eu
- 2. ● Complex Event Processing What ? Why ? ● WSO2 Complex Event Processor Scalability Edge Computing ● Usecases ● What’s new is WSO2 CEP 4.0 ?
- 6. ● Amount of data you analyse are big ● You don’t need data for long time ● You don’t want to store all the data ● Time critical analysis ○ Finding opportunities before they’re gone ○ Identify threats beforehand
- 9. • Filters & Transformations • Windows and Aggregations • Joins • Sequences and Patterns • Event Tables
- 10. Stateless • Filters & Transformations Stateful Inmemory • Windows and Aggregations • Joins • Sequences and Patterns Stateful Persisted • Event Tables
- 12. • Vertical Scaling • Horizontal Scaling
- 17. How about scaling median ?
- 18. How about scaling median ? If & only if we can partition !
- 20. Spark Streaming • Supports distributed processing • Runs micro batches • Not supports pattern & sequence detection
- 21. Spark Streaming • Supports distributed processing • Runs micro batches • Not supports pattern & sequence detection Apache Storm • Supports distributed processing • Stream processing engine
- 22. Advantages • Supports distributed processing • Supports Partitioning • Extendable • Open Source Disadvantages • Need to write Java code • Need to start from basic principles ( & data structures ) • Adoption for change is slow • No support to govern artifacts
- 23. Advantages • Supports distributed processing • Supports Partitioning • Extendable • Open Source Disadvantages • No need to write Java code (Supports SQL like query language) • No need to start from basic principles (Supports high level language) • Adoption for change is fast • Govern artifacts using Toolboxes • etc ...
- 28. define stream StockStream (symbol string, volume int, price double); @name(‘Filter Query’) from StockStream[price > 75] select * insert into HighPriceStockStream ; @name(‘Window Query’) from HighPriceStockStream#window.time(10 min) select symbol, sum(volume) as sumVolume insert into ResultStockStream ;
- 29. define stream StockStream (symbol string, volume int, price double); @name(‘Filter Query’) from StockStream[price > 75] select * insert into HighPriceStockStream ; @name(‘Window Query’) partition with (symbol of HighPriceStockStream) begin from HighPriceStockStream#window.time(10 min) select symbol, sum(volume) as sumVolume insert into ResultStockStream ; end;
- 30. define stream StockStream (symbol string, volume int, price double); @name(Filter Query’) @dist(parallel= ‘3') from StockStream[price > 75] select * insert into HightPriceStockStream ; @name(‘Window Query’) @dist(parallel= ‘2') partition with (symbol of HighPriceStockStream) begin from HighPriceStockStream#window.time(10 min) select symbol, sum(volume) as sumVolume insert into ResultStockStream ; end;
- 34. • System Monitoring (Logs, Servers) • Finance (Financial trading) • Banking (Credit card fraud detection) • Marketing (Location & time based advertising) • Logistics (Taxi services, Vehicle parking) • Healthcare • E-commerce (Online retailers) • Telecommunication (Mobile offers) • System automation (Auto scaling) • IoT (Smart home, Smart buildings) • ...
- 35. • Distributed processing with Apache Storm • Improved High Availability • Improved Siddhi Query Language • Toolbox Support • Realtime Dashboard • Websocket + MQTT Support And lots of extensions • Geo • Timeseries • NLP • PMML & WSO2 ML Try out CEP 4.0 Alpha : https://github.com/wso2/product- cep/releases