Complex Event Processing (CEP) for Next-Generation Security Event Management, Fraud and Intrusion Detection

Tim Bass, CISSP Director, Principal Global Architect Emerging Technologies Group Complex Event Processing (CEP) for Next-Generation Security Event Management, Fraud and Intrusion Detection April 17, 2007 (First Draft) London

Our Agenda Brief Overview of TIBCO Software Inc. PredictiveBusiness® and CEP SEM, FDS and IDS Reference Architecture Solutions Architecture and Case Study Wrap Up & Open Discussion

Who We Are and What We Do We help our customers… Improve operational visibility, collaboration and ability to be proactive Increase operational efficiency and effectiveness Accelerate projects, initiatives and go-to-market cycles A leading provider of business integration and process management software.

How TIBCO Delivers for Customers Accelerate projects, initiatives, and go-to-market cycles Increase operational efficiency and effectiveness. Improve operational visibility, security, collaboration and responsiveness

TIBCO is Trusted by Thousands of Companies 47 of the World’s 100 Largest Companies are TIBCO Customers * By annual revenues except for investment banking which is measured by assets Retail Banking — 17 of top 20 Consumer Package Goods — 5 of top 10 Energy — 5 of top 10 Hi-Tech Manufacturing — 15 of top 20 Investment Banking — 9 of top 10 Manufacturing (non High-tech) — 5 of top 10 Pharmaceutical — 6 of top 10 Telecommunications — 8 of top 10 Transportation — 4 of top 10

TIBCO History and Acquisitions IPO 1999 eXtensibility InConcert Staffware TIBCO Today Teknekron 2000 2002 2001 2003 2004 2005 2005 1,600+ employees Consistently profitable Worldwide presence Recognized market leader 2500+ customers Acquired by Reuters Est. 1980s Palo Alto Campus Est. 1997 2004

TIBCO Runs a Strong and Viable Business 14 consecutive quarters of yr/yr total revenue growth $284M USD invested in R&D in past 4 years $540M USD in cash + short term investments in the bank and growing Market cap of $1.9 billion (USD)

Revenue Numbers FY 2004 – 2006 (in thousands of dollars) 15.8% $61,060 $73,715 $387,220 FY2004 16.4% $73,127 $67,081 $445,910 FY2005 16.6% $85,923 $90,558 $517,279 FY 2006 R&D SPEND AS A % OF REVENUE R&D SPEND PRE-TAX PROFIT REVENUE

PredictiveBusiness TM Source: Ranadiv é , V., The Power to Predict , 2006.

Complex Event Processing " Events in several forms, from simple events to complex events, will become very widely used in business applications during 2004 through 2008 " --- Gartner July 2003

What is Complex Event Processing? Detecting Threats and Opportunities with PredictiveBusiness®

When Do You Need to Think About CEP? “ CEP applies to a very broad spectrum of challenges in information systems. A short list includes:” Business process automation Computer systems to automate scheduling and control network-based processes and processing Network monitoring and performance prediction Detection intrusion, fraud and other network attacks . The Power of Events , Addison Wesley, ISBN: 0-201-72789-7, 2002

Bloor Report on Event Processing Event Processing and Decision Making Automated Operational Decisions Automated Predictive Decisions Human Predictive Decisions Human Operational Decisions Decision Latency Event Complexity Process Complexity Pattern Matching and Inferencing Anti-Money Laundering Credit-Card Fraud Exchange Compliance Database Monitoring Algorithmic Trading Trade Desk Monitoring Customer Interaction Order Routing RFID Tariff Look-Up Rail Networks Search & Rescue Baggage Handling Liquidity Management

Firewalls, Stand-Alone or Purpose-Built Fraud and Intrusion Detection Systems, Cryptography, Access Control, are Simply Not Sufficient. Malicious Users are Using Legitimate Internet Application Protocols, such as HTTP, HTTPS and SOAP to Defraud Businesses. A 2006 CyberSource reports that $2,800,000,000 (2.8B USD) was lost to on-line fraud in the US and Canada in 2005. eCommerce online fraud continues to grow (US and Canada) at a 20% annual rate. Risk for international transactions is 3 times the average risk. Industry and Business Drivers A Sample of the Problems with Network Security and Fraud Detection

Rapidly detect threats with a low rate of false alarms and a high level of situational detection confidence … Detection-Oriented Systems - Design Goals What are the overall design goals for detection systems? (Illustrative Purposes Only)

Classification of Intrusion and Fraud Detection Systems Traditional View Before Data Fusion Approach to FDS and IDS Distributed Fraud and Intrusion Detection Systems, Logs Detection Approach Systems Protected Architecture Data Sources Analysis Timing Detection Actions HIDS NIDS Hybrid Audit Logs Net Traffic System Stats Real Time Data Mining Anomaly Detection Signature Detection Centralized Distributed Active Passive Agent Based Security “Stovepipes” Centralized

Intrusion Detection and Data Fusion (2000) Next-Generation Intrusion Detection Systems Source: Bass, T., CACM, 2000

A Business Optimization Perspective What Classes of Rule-Based Problems Do Businesses Need to Solve? Rule-Based Pattern Recognition Anomaly Detection Track and Trace Monitoring (BAM) Dynamic Resource Allocation Adaptive Resource Allocation Constraint Satisfaction (CSP) Dynamic CSP Adaptive Marketing Dynamic CRM Fault Management Impact Assessment Detection Prediction Scheduling Fraud Detection Intrusion Detection Fault Detection Rule-Based Access Control Exception Management Compliance Work Flow Risk Management Fault Analysis Impact Assessment Example PredictiveBusiness® Applications

Emerging Event-Decision Architecture Customer Profiles Purpose-Built Analytics Secure, Distributed Messaging Backbone Internet/Extranet Sensors Human Sensors Edge/POC Sensors Operations Center Other Reference Data Rule-Based Event Processors

Complex Event Processing Reference Architecture Next-Generation Functional Architecture for Fraud and Intrusion Detection 24 EVENT PRE-PROCESSING EVENT SOURCES EXTERNAL . . . LEVEL ONE EVENT TRACKING Visualization, BAM, User Interaction CEP Reference Architecture DB MANAGEMENT Historical Data Profiles & Patterns DISTRIBUTED LOCAL EVENT SERVICES . . EVENT PROFILES . . DATA BASES . . OTHER DATA LEVEL TWO SITUATION DETECTION LEVEL THREE PREDICTIVE ANALYSIS LEVEL FOUR ADAPTIVE BPM

CEP – Situation Detection Hierarchy 22 Adapted from: Waltz, E. & Llinas, J., Multisensor Data Fusion, 1990 Impact Assessment Situational Assessment Relationship of Events Identify Events Location, Times and Rates of Events of Interest Existence of Possible Event of Interest Data/Event Cloud Analysis of Situation & Plans Contextual and Causal Analysis, Rules Causal Analysis, Bayesian Belief Networks, Rules, NNs, Correlation, State Estimation, Classification Use of Distributed Sensors for Estimations Raw Sensor Data (Passive and Active) HIGH LOW MED

CEP High Level Architecture 22 Adapted from: Engelmore, R. S., Morgan, A.J., & and Nii, H. P., Blackboard Systems, 1988 & Luckham, D., The Power of Events, 2002 EVENT CLOUD (DISTRIBUTED DATA SET) KS KS KS KS KS KS KS KS KS KS KS KS KS KS

Sensors Systems that provide data and events to the inference models and humans Actuators Systems that take action based on inference models and human interactions Knowledge Processors Systems that take in data and events, process the data and events, and output refined, correlated, or inferred data or events HLA - Knowledge Sources KS KS KS

Structured Processing for Event-Decision Multi-level inference in a distributed event-decision architectures User Interface Human visualization, monitoring, interaction and situation management Level 4 – Process Refinement Decide on control feedback, for example resource allocation, sensor and state management, parametric and algorithm adjustment Level 3 – Impact Assessment Impact assessment, i.e. assess intent on the basis of situation development, recognition and prediction Level 2 – Situation Refinement Identify situations based on sets of complex events, state estimation, etc. Level 1 – Event Refinement Identify events & make initial decisions based on association and correlation Level 0 – Event Preprocessing Cleansing of event-stream to produce semantically understandable data Level of Inference Low Med High

CEP Level 0 – Event Preprocessing Cleanse/Refine/Normalize Data for Upstream Processing Calibrate Raw Event Cloud: Web Server Farm Event Stream Example - Group HTTP REQUESTS and RESPONSES Reduce and Extract Required Data from Transaction Format into Event for Upstream Processing Agent-Based Log File Event Steam Example - Parse Log File for Sensor Information Match Patterns and Convert Tokens to JMS Properties Reduces System Load by Preprocessing Events Enables Upstream to Concentrate on Most Relevant Events Focuses on Objects/Events

CEP Level 1 – Event Refinement Problem: Which Events in the Event Stream Are “Interesting”? Event Refinement Example (Association & Classification): Hypothesis Generation (HG) Processing incoming events, data and reports Hypothesis: This Group of Events May Need to be Tracked Output: Scorecard or Matrix Hypothesis Evaluation (HE) Evaluates Scorecard/Matrix for likelihood evaluation Rank Evaluation: These Events have a Higher Likelihood Output: Fills Scorecard/Matrix with relative likelihood estimation Hypothesis Selection (HS) Evaluates Scorecard/Matrix for best fit into scenario Evaluation: Provide an Estimate (Name) of the Scenario Activity Output: Assignment of scenario - activity estimate to event

CEP Level 2 – Situation Refinement What is the Context of the Identified Events? Focuses on Relationships and States Between Events Situation Refinement Event-Event Relationship Networks Temporal and State Relationships Geographic or Topological Proximity Environmental Context Example: Brand currently used by phishing site in Internet increasing probability of fraud and identity theft Event / Activity Correlation – Relational Networks Pattern, Profile and Signature Recognition Processing

CEP Level 3 – Impact Assessment Predict Intention of Subject (Fraudster example) Make changes to account identity information? Transfer funds out of account? Test for access and return at later time? Estimate Capabilities of Fraudster Organized Gang or Individual Fraudster? Expert or Novice? Estimate Potential Losses if Successful Identify Other Threat Opportunities

CEP Level 4 – Process Refinement Evaluate Process Performance and Effectiveness Exception Detection, Response Efficiency and Mitigation Knowledge Development Identify Changes to System Parameters Adjust Event Stream Processing Variables Fine Tune Filters, Algorithms and Correlators Determine If Other Source Specific Resources are Required Recommend Allocation and Direction of Resources

Database Management Examples Reference Database User Profiles Activity and Event Signatures and Profiles Environmental Profiles Inference Database Subject Identification Situation and Threat Assessment Knowledge Mining Referential Mapping Database Examples Mapping Between IP Address and Domain Mapping Between Known Anonymous Proxies

User Interface / Interaction Operational Visualization at all “Levels” Dynamic Graphical Representations of Situations Supports the Decision Making Process of Analytics Personnel Process and Resource Control Supports Resource Allocation and Process Refinement Display Control & Personalization Different Operator Views Based on Job Function and Situation

Business Optimization Summary A Simplified View of the CEP Reference Architecture Flexible SOA and Event-Driven Architecture

TIBCO’s Real-Time Agent-Based SEM Approach A Multisensor Data Fusion Approach to Security Event Management Distributed Fraud and Intrusion Detection Systems, Logs Detection Approach Systems Protected Architecture Data Sources Analysis Timing Detection Actions HIDS NIDS Hybrid Audit Logs Net Traffic System Stats Real Time Data Mining Anomaly Detection Signature Detection Centralized Distributed Active Passive Agent Based Enterprise Correlation of Security Events

Security Event Management High Level Event-Driven Architecture (EDA) for SEM (CEP and BPM) JAVA MESSAGING SERVICE (JMS) DISTRIBUTED EVENTS (TIBCO EMS) HIGH PERFORMANCE RULES-ENGINE (TIBCO BE) HIGH PERFORMANCE RULES-ENGINE (TIBCO BE) HIGH PERFORMANCE RULES-ENGINE (TIBCO BE) SENSOR NETWORK RULES NETWORK FDS BW JMS LOGFILE JMS BW LOGFILE JMS BW LOGFILE JMS BW IDS JMS BW FDS JMS BW SQL DB BW JMS ADB SQL DB BW JMS ADB MESSAGING NETWORK SYSTEM SYSTEM SYSTEM SYSTEM SYSTEM SYSTEM SYSTEM SYSTEM BPM Compliance Workflow (TIBCO iProcess)

TIBCO BusinessEvents™ Solutions Overview BusinessEvents™ Solutions Space Data: Events & Databases -Real-Time & Historical Data Models: Statistical Financial Optimization Comms: Pub/Sub Messaging Queues Topics UIs Knowledge: Facts & Rules

TIBCO BusinessEvents™ Overview High performance, low latency business rules engine. Top down business process modeling. Real-time event processing. Cross-application and cross-process integration. Analytical and predictive models. Modeling Tools, Statefulness, Business Rules and Process Integration UML Conceptual UML State Business Rules Business Users Event Analyzer

TIBCO BusinessEvents™ Overview Collection, Normalization Metric of Managed Objects, Normalized Non-Contextual Events Metadata Repository Event Management, Correlation, Aggregation, Inference and Analysis Correlated, Analyzed, Contextual Dialogue Events Rules, Knowledge, Patterns, Models Visualization, Reporting, Alert Management Application Interface Feeds Visualization: Detection Metrics Agents Synthetic Warehouse Visualization: Process View Dialogue Manager Inference Engine FDS/IDS Logfiles Edge Devs Semantic Model Events Rules Design Environment State Model Sensors

TIBCO BusinessEvents™ Awards 2006 Best Complex Event Processing Software Winner: TIBCO 2006 Event Processing General Purpose Gold Award Winner

CEP and BusinessEvents™ Case Study: Real-Time On-Line Fraud Detection Requirements Identify characteristics of fraud, such as continuous behavior changes, and identify new patterns of fraud Stop new account setups from fraudulent IP addresses Stop online registrations from fraudulent IP addresses Verify user identity in every transaction based on click-behavior Identify multiple users trying to login from same IP address Identify single user logins from multiple IP addresses within a time span Prevent phishing by tracking IP addresses that mass download institutional web pages Prevent phishing, pharming and man-in-the-middle attacks by checking against a list for fraudulent IP’s in real-time

On-Line Fraud Detection Use Case Architecture and Capacity Planning Approx. 12,000 Hits Per Second During Peak Period Across the Three Sites – One Instance Of TIBCO BusinessEvents™ Capable of Handling Maximum Hits Overall 100 Million Hits Handled Between 3PM – 4 PM Peak Approx. 250 Million Hits Per Day Across the Three Sites TIBCO EMS™ TIBCO Business Events™ Session Info Three Server Farms ~600-700 Application Servers

Characteristics of Solutions Architecture Fusion of SEM information from across the enterprise, including: Log files Existing FDS and IDS (host and network based) devices Network traffic monitors Host statistics Passive Web-stream “edge devices” Secure, standards-based JAVA Messaging Service (JMS) for messaging: Events parsed into JMS Application Properties SSL transport for JMS messages TIBCO technology for next-generation detection, prediction, rule-based intrusion response, and adaptive control TIBCO Business Works™ as required, to transform, map or cleanse data TIBCO BusinessEvents™ for rule-based IDS analytics TIBCO Active Database Adapter as required

Potential Extensions to Solutions Architecture Extension of SEM to rules-based access control Integration of SEM with access control TIBCO BusinessEvents™ for rule-based access control Extension of SEM and access control to incident response Event-triggered work flow TIBCO iProcess™ BPM for incident response TIBCO iProcess™ BPM security entitlement work flow TIBCO BusinessEvents™ for rule-based access control Extensions for other risk and compliance requirements Basel II, SOX, and JSOX - for example Other possibilities to be discussed later Extensions for IT management requirements Monitoring and fault management, service management, ITIL

TIBCO SOA and BPM Architecture

Key Takeaways Enterprise SEM requires the correlation and fusion of information from numerous event sources across the enterprise: Model all IDS Devices, Log Files, Sniffers, etc. as Sensors Use Secure Standards-based Messaging for Communications Next-Gen IDS Requires a Number of Technologies: Distributed Computing, Publish/Subscribe and SOA Hierarchical, Cooperative Inference Processing High Speed, Real Time Rules Processing with State Management Event-Decision Architecture for Complex Events / Situations Solution Expandable to Other Security, Compliance and IT Management Areas (as required)

Thank You! Tim Bass, CISSP Director, Principal Global Architect Emerging Technologies Group [email_address] Event Processing at TIBCO

Complex Event Processing (CEP) for Next-Generation Security Event Management, Fraud and Intrusion Detection

More Related Content

Viewers also liked (20)

Similar to Complex Event Processing (CEP) for Next-Generation Security Event Management, Fraud and Intrusion Detection (20)

More from Tim Bass (8)

Recently uploaded (20)

Complex Event Processing (CEP) for Next-Generation Security Event Management, Fraud and Intrusion Detection

Editor's Notes