Flink Forward Berlin 2017: Dawid Wysakowicz - Looking for patterns with Flink CEP library
1 like1,154 views
The document outlines the features and capabilities of complex event processing (CEP) engines, particularly focusing on event-pattern detection, aggregation, and transformations. It includes examples of pattern specifications, nondeterministic finite automata, and SQL integration with Flink. Additionally, it refers to future enhancements such as dynamic patterns and SQL match recognition within the Flink ecosystem.
Flink Forward Berlin 2017: Dawid Wysakowicz - Looking for patterns with Flink CEP library
- 1. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
- 2. © Copyright. All rights reserved. Not to be reproduced without prior written consent. About me ■ Data Engineer at GetInData ■ Apache Flink Committer ■ Involved in Flink CEP library development Dawid Wysakowicz @dwysakowicz
- 3. © Copyright. All rights reserved. Not to be reproduced without prior written consent. CEP engines’ features ■ Event-pattern detection ■ Event abstraction ■ Event filtering ■ Event aggregation and transformation ■ Modeling event hierarchies ■ Detecting relationships (such as causality, membership or timing) between events ■ Abstracting event-driven processes David C. Luckham. 2001. The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA. Opher Etzion and Peter Niblett. 2010. Event Processing in Action (1st ed.). Manning Publications Co., Greenwich, CT, USA.
- 4. © Copyright. All rights reserved. Not to be reproduced without prior written consent. CEP engines’ features ■ Event-pattern detection ■ Event abstraction ■ Event filtering ■ Event aggregation and transformation ■ Modeling event hierarchies ■ Detecting relationships (such as causality, membership or timing) between events ■ Abstracting event-driven processes FlinkCEP
- 5. © Copyright. All rights reserved. Not to be reproduced without prior written consent. CEP engines’ features ■ Event-pattern detection ■ Event abstraction ■ Event filtering ■ Event aggregation and transformation ■ Modeling event hierarchies ■ Detecting relationships (such as causality, membership or timing) between events ■ Abstracting event-driven processes FlinkCEP Flink ecosystem
- 6. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
- 7. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Buying Home Insurance Online Property Info Offer Accept General Terms of Contract PaymentScope Change Info Change
- 8. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Simple Example - Matching Payments Notify whenever a payment for accepted offer arrives
- 9. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Patterns Pattern .begin("Offer accepted") .subtype(OfferAccepted.class) .where(...) .or(...) .followedBy("Payment received") .subtype(Payment.class) .where(...) .where(...) Basic building blocks
- 10. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Conditions Pattern .begin("Offer accepted") .subtype(OfferAccepted.class) .where(...) .or(...) .followedBy("Payment received") .subtype(Payment.class) .where(...) .where(...) Basic building blocks Condition specification
- 11. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Pattern succession Pattern .begin("Offer accepted") .subtype(OfferAccepted.class) .where(...) .or(...) .followedBy("Payment received") .subtype(Payment.class) .where(...) .where(...) Basic building blocks Condition specification Pattern succession
- 12. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Time Restrictions Pattern .begin("Offer accepted") .subtype(OfferAccepted.class) .where(...) .or(...) .followedBy("Payment received") .subtype(Payment.class) .where(...) .where(...) .within(Time.seconds(3)) Basic building blocks Condition specification Time restrictions Pattern succession
- 13. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Image source: “Continuous Analytics: Stream Query Processing in Practice”, Michael J Franklin, Professor, UC Berkley, Dec 2009 and http://www.slideshare.net/JoshBaer/shortening-the-feedback-loop-big-data-spain-external
- 14. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ■ followedBy Consuming Strategy
- 15. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ■ followedBy ■ next Consuming Strategy
- 16. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ■ followedBy ■ next ■ notFollowedBy Consuming Strategy
- 17. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ■ followedBy ■ next Consuming Strategy ■ notFollowedBy
- 18. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ■ Skip till next ■ Strict continuity ■ Skip till any ■ Not follow ■ Not next See more: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html#conditions-on-contiguity https://www.slideshare.net/DawidWysakowicz/flink-complex-event-processing Consuming Strategy
- 19. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Looping Example How many offer changes before accepting? ... ?
- 20. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Quantifiers Pattern .begin("start") .subtype(Event.class) .where(...) .or(...) .times(3) .followedBy("next") .subtype(Event.class) .where(...) .where(...) .oneOrMore().optional() .within(Time.seconds(3)) Quantifier application Basic building blocks Condition specification Time restrictions Pattern succession
- 21. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
- 22. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Nondeterministic Finite Automaton ■ Graph where: ● Vertices = States ● Edges = Transitions ■ IGNORE - event not consumed ■ TAKE - event consumed ■ PROCEED - event analyzed in the target state
- 23. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Example of NFA Pattern .begin(“A”) .followedByAny(“B”).optional() .next(“C”) .followedBy(“D”)
- 24. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Example of NFA A B? C D TAKE (A) TAKE (C) TAKE (B) PROCEED (TRUE) IGNORE (TRUE) IGNORE (TRUE) IGNORE (not D) TAKE (D) Pattern .begin(“A”) .followedByAny(“B”).optional() .next(“C”) .followedBy(“D”)
- 25. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A
- 26. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A B? C D TAKE TAKE TAKE PROCEED IGNORE IGNORE IGNORE TAKE A
- 27. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A A B? A
- 28. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream A B Example of NFA A B? C D TAKE TAKE TAKE PROCEED IGNORE IGNORE IGNORE TAKE
- 29. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A A B? A B A
- 30. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A B? C D TAKE TAKE TAKE PROCEED IGNORE IGNORE IGNORE TAKE A B
- 31. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A A B? A B A C B?
- 32. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A B? C D TAKE TAKE TAKE PROCEED IGNORE IGNORE IGNORE TAKE A B C
- 33. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A A B? A B A C B? D B? A C
- 34. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A B? C D TAKE TAKE TAKE PROCEED IGNORE IGNORE IGNORE TAKE A B C
- 35. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A A B? A B A C B? D B? D A C
- 36. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A A B? A B A C B? D B? D A C D Match: A, C, D B? Match: A, B, C, D A
- 37. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Nondeterminism ■ When? ● OneOrMore ● Times ● FollowedByAny ● Optional
- 38. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Nondeterminism ■ When? ● OneOrMore ● Times ● FollowedByAny ● Optional ■ What to do? ● Within - restrict by time ● Until - restrict by event
- 39. © Copyright. All rights reserved. Not to be reproduced without prior written consent. What brings the future?
- 40. © Copyright. All rights reserved. Not to be reproduced without prior written consent. SQL Integration - Flink 6935 select orderId from events MATCH_RECOGNIZE ( PARTITION BY userId MEASURES A.orderId PATTERN (A B) DEFINE A AS (A.type = “order”), B AS (B.type = “payment” and B.orderId = A.orderId) ) as its
- 41. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Dynamic Patterns - Flink 7129 DataStream Events Source e.g. CEP DataStream Pattern DataStream Patterns Source e.g.
- 42. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Dynamic Patterns - Flink 7129 DataStream Events Source e.g. CEP DataStream Pattern DataStream Patterns Source e.g.
- 43. © Copyright. All rights reserved. Not to be reproduced without prior written consent. FlinkCEP ■ Pattern detection ■ CEP engine with the rest of Flink ■ What to expect: ● Dynamic patterns - FLINK-7129 ● SQL’s MATCH_RECOGNIZE - FLINK-6935
- 44. © Copyright. All rights reserved. Not to be reproduced without prior written consent. FlinkCEP ■ Pattern detection ■ CEP engine with the rest of Flink ■ What to expect: ● Dynamic patterns - FLINK-7129 ● SQL’s MATCH_RECOGNIZE - FLINK-6935 ● YOUR AD IDEA HERE