Developing Configurable and High Performance Apps in Drools
- 1. Developing Configurable and High Performance Apps in Drools Ajay Mahajan Lead Architect 1
- 2. Agenda What, Where, Why, When Drools Eco-System & A Use Case Rule Definitions Usage - Deployment Modes Best Practices Value from Session 2
- 3. What, Where, Why, When 3
- 4. What is a Rule Engine 4
- 5. Where Does It Fit Parameterization Code Config Files Database Rules Engine 5
- 6. Why Should I Bother • Many ways to define Rules Flexibility & • Group rules and define priorities Configurability • Plethora of functions that help in decisioning Rule Engine • User friendly and Business Like Manageability • Better Tooling Support • Easier Understanding & Visualization • Just define your rules not execution details Declarative v/s • Execution is handled by Rules Engine Imperial • Sequencing and Re-entry 6
- 7. FUDs – Fear, Uncertainty and Doubt • True, as compared to if-else statements in code Slow • However, Rules are precompiled • In Some Cases, execution is faster if designed correctly • Yes, they do have a learning curve Difficult • Start small and Limit the features to those you really need • You don’t have to learn each feature and function offered More • Yes, rules engine do need more space than a java class file • Precompiled Rules form a Rete graph Memory • Use stateless models where use case allows • Stateful models – Follow the optimizations and mind your memory • Commercial Tools can get expensive, e.g. Blaze and Jrules Expensive • Open Source Drools has evolved over time, in its version 5.5 • Drools used in high volume, mission critical systems 7
- 8. When Should I Use a Rule Engine IF (Requirements== expressed as rules) IF (Rules == many OR complex OR changing frequently) IF (Rules == managed separately from application code) IF (memory != very low) IF (Application == evolving) IF (Developers == have skills OR ok with learning curve) IF (Business Users == like to see / experiment with rules) IF ( Additional Complexity < Benefits in Flexibility + Configurability) THEN use Rules Engine 8
- 10. Jboss - Drools Expert • Main Rules Engine Component • Have Stabilized after going through product cycles • Highly Successful and widely used Rule Flow (replaced by jBPM) • Group into Rule sets and define flow chart to execute them • Graphical Environment to define work flows • jBPM using BPMN 2.0 is the way to go for any serious BPM Guvnor • Web Based GUI to manage the Rules • Split out as separate component in V5 • Read Only in Production, but modify in test/UAT for business Planner • Resources– travelling salesman, scheduling, routing • Heuristic Rules • Relatively new Fusion • Event Processing- ESP / CEP use cases • Concept of Sliding Time window • Other products such as Esper and Twitter Storm 10
- 11. Drools Expert – Steps • DRL, Decision Define Tables, DSL Compile • To Knowledge base Create • Uses Knowledge base Session Insert • Causes Activations Facts Fire • RHS Rules Executes 11
- 12. Real Use Case Trades Matching Engine Match Statuses T1, T4, T3, T2 T1 C3 T2 C1 + C3 T3+T4 C4 Confirmation C1, C2, C3, C4 100’s of Trades and Confirmations inflowing per second at peak hr Flowing in any order, not necessarily one after the other One trade can exactly match to one confirmation A trade can match to more than one confirmation One or more trades can match to one Confirmation 12
- 13. DRL – Drools Rules Language rule "Perfect Match" salience 100 when t:Trade() c:Confirm(qty == t.qty , confirmId == t.cusip , amt == t.amt , price == t.price ) then log("Perfect Match for " + t.toString() + c.toString()); Match perfectMatch = new Match(t,c, "Perfect"); // retract perfect matches retract(t); retract(c); end Trade to Confirm Matching Why is this The base language for Rules Definition blazing All other Forms compile to this language fast ?? Rich and Versatile Entire Syntax, features and Rules definition is available Least User Friendly 13
- 14. DSL - Domain Specific Language expander Match.dsl rule "Exact Match Trade to Confirm" when Match Trade and Confirm - on cusip with confirmId - on amt - on price then Log "Perfect Match“ Create Match with Status “Perfect” Remove Matched Elements end This is real code not Pseudo code DSL combines with grammar definition and translates to DRL Very Easy to Understand for Users and Visualize Power of creating new business vocabulary Encourages reuse 14
- 15. Decision Table RuleTable TradeRequests NAME CONDITION CONDITION ACTION ACTION ACTION ACTION ACTION ACTION event:Event event:Event eventGroup $param != null TradeRequest tr = ne tr.setStartCaptureDate(DateUtils.getBusinessDate(event.get$1(), $2+1)); tr.setEndCaptureDate(DateUtils.getBusinessDate(event.get$1(), $2+1)); tr.setStartTradeDate(DateUtils.getBusinessDate(event.get$1(), $2)); tr.setEndTradeDate(Date tr.setSettlementF Trade Condition Event Start Capture End Capture Settlement (Comment Column) Name Group Date Present Trade Type Date Date Traded After Traded Before Flag TD < ED ; F = F IncOpenFail I effectiveDate Open Fails FreezeDate, 0 EffectiveDate, -1 "F" TD < ED ; SD <= FD ; F = O IncRegOpen I effectiveDate Regular Open EffectiveDate, -1 FreezeDate, 0 EffectiveDate, -1 "O" TD < ED ; FD < SD IncExtSet I effectiveDate Extended Settle EffectiveDate, -1 FreezeDate, 0 EffectiveDate, -1 ED<=TD ; SD <=FD ; CD <= FD IncShortSet I effectiveDate Short Settle EffectiveDate, 0 FreezeDate, 0 EffectiveDate, 0 TD < ED ; FD < PD IncAsOf I effectiveDate As of Trades FreezeDate, 1 EffectiveDate, -1 Corporate Actions Excel columns are designated as Conditions or Actions Top few control rows are hidden from users, Control rows help translate the excel into DRL Easy to understand for Users, once basic Excel formats are given Fit for use cases where there is need for intense parameterization 15
- 16. Learning & Best Practices Experiences, Usage Models, Performance 16
- 17. Real Life Experiences Complex Matching Engines • Multiple Engines used for various functional matching • Performed at 600 transactions / sec on one instance of execution • If the I/o (messaging / database) were commented, got 8k executions / sec • Stateful models used, but memory was conversed through optimizations Corporate Actions – Event Validations, Trade Extraction • Stateless model that evaluates each event separately through set of rules • 100’s of rules based on event types defined in Decision Tables • Increase in number of rules barely dent the performance • For 10k executions, 2 rules take 320 ms, and 100 rules take 328 ms Risk Analysis & Calculations • Calculations have lot of parameters, such as credit rating, product type, etc. • Calculations segmented into small number of individual steps • The decision of which formulae to use, was done by a Rules Engine 17
- 18. Usage Models Synchronous Execution Request – Response Style Your Rule Application Engine Can act on the decision immediately Asynchronous Pipeline Rule Messaging Style Events Actions Engine Very scalable and resilient In Process with the Application Jar file as part of the application Excellent for Stateless execution, as reduces I/o without increasing memory Stateful executions are challenge in clustered environment & need memory sizing Out of Process as a runtime component Central Deployment & Management Overheads in Communication, and hence affects performance Could become bottleneck / central point of failure Needs sophisticated scaling models (e.g. functional split based on Hash or some key) 18
- 19. Improving Performance ..1 Keep Separate Deployable Units rather than a giant rule engine component Divide and Conquer Use Stateless Sessions where Business case allows You can cluster and load balance your services seamlessly You can use in-process deployments easily Limit the number of facts in Stateful Executions The degradation is exponential beyond 400k objects in memory If higher volumes anticipated, than plan for multi deployments using sharding concepts Limit the Size of the objects checked in memory Use DTO (Data Transfer Object) pattern Use Batched Mode of Execution Check in more objects if you can in one go into the memory 19
- 20. Improving Performance ..2 Use Drools only for decisions, not performing actual actions Let the decisions be communicated to a downstream component or by the caller to Rules Engine Avoid using evals(), --- use only as a last resort The java code inside eval is difficult to optimize into rete tree Work on aggregates where possible Rather than Checking Individual facts into the memory If you want to dig deeper Read more on the Rete Algorithm 20
- 21. Development Tips Use the IDE Syntax Validations DSL conversions Drools Debugging Use events Understanding how rules activate and fire Helpful for troubleshooting Remember to turn off in production Keep individual rules small, simple and atomic Avoid cyclic triggering of rules when you update the facts Use Agenda groups & Activation groups wherever applicable 21
- 22. Ajay Mahajan ajay.mahajan@wipro.com 22