SAP Sybase Event Streaming Processing

High Performance CEP:
The SAP Sybase Event Stream Processor
Technical Product Overview
Speaker’s Name/Department (delete if not needed)
Month 00, 2011

Insight from fast-moving data: Continuous Intelligence

Analyze events as they occur
• Continuous insight
• Immediate response

Rapid application development
• Reduce/eliminate dependence on specialist
programming skills
• Cut implementation/deployment time
• Improve business agility

Non-intrusive deployment
• Event-driven architecture
• Adapt to existing data models

© 2012 SAP AG. All rights reserved. 2

The SAP Sybase Event Stream Processor

INPUT Studio
STREAMS (Authoring)

Reporting
Sensor Data
Tools
SAP® Dashboard
?
Transactions
Sybase® Applications
Message

Events
ESP Bus
SAP HANA, Sybase IQ

Reference
Data

• Unlimited number of input • Send output to apps, dashboards
streams
• Capture data in SAP HANA, Sybase IQ,
• Incoming data is processed as it Sybase RAP for historical analysis,
arrives, according to the compliance
business logic defined using high
level authoring tools


Sample Architecture of a CEP-based Application:
Real-time P&L

Market Prices

 Reuters Live Dashboard
Market Data
Middleware
 Bloomberg
 Nasdaq
 …

Order Management Systems

Alerts
Trading Systems

Sybase Historical Analytics,
Ad-hoc Analysis,
IQ
Compliance


Two approaches to CEP

Continuous
Rules
Queries


Key Concepts

Input Streams Derived Streams,
Events arrive on input streams Windows
Apply continuous query
operators to one or more
input streams to produce a
new stream

Database Analogy Windows can Have State
• Streams are like tables • Retention rules define how many or how
• Events are rows in the table long events are kept
• Fields in an event message are • Opcodes in events can indicate
columns in the table insert/update/delete and can be
automatically applied to the window


ESP Studio: Both visual and textual authoring

Visual Dataflow Authoring Language-based Authoring

• ANALYST-LEVEL SKILLS (EXCEL, VBA) • RAPID PROGRAMMING
• EASY TO UNDERSTAND COMPLEX MODELS • EASY TO USE LANGUAGE (CCL)
• NO NEED TO LEARN LANGUAGE SYNTAX • MODULAR, PROJECT BASED APPROACH

ESP Studio supports both Visual and Textual authoring
• Visual diagram produces CCL (code-behind approach)
• User can switch back and forth between diagram and language
• Eclipse-based


CCL: Continuous Computation Language
SQL-based Event Processing Language
Leverage familiarity and simplicity of SQL

Instead of “snapshot” queries, continuous queries
Filter (WHERE), Aggregate (GROUP BY), Join, Compute

Extensions for event streams
Windows, Patterns

Example CCL Query

Calculation of 10 CREATE LOCAL WINDOW VWAP10
minute VWAP PRIMARY KEY DEDUCED
AS
SELECT
TICKER.Symbol AS Symbol ,
Sum ( TICKER.Price * TICKER.Size) / Sum ( TICKER.Size)
AS VWAP
FROM TICKER KEEP 10 MINUTES
GROUP BY TICKER.Symbol ;


Key CCL Language Features

• Stateless and stateful elements. • Full support for JOINS
– STREAMS (stateless without keys) – ANSI JOIN Syntax
– WINDOWS (stateful with keys) – Stream-Window Joins and Window-Window joins
– Windows push updates and automatically process
opcodes. • Filter

• Retention policies on Windows • Aggregate
– Named windows (windows created with “CREATE – Group By, Group Filter, Group Order
WINDOW..”)
– Unnamed windows (FROM A KEEP 5 ROWS) • Pattern Matching
– Time and Count based retention policies – Watch for specific sequence of events within time
interval
– Custom windows with EventCache
– Within or across streams
• Parameters, Variables – Pattern can include missing events
– Variables can maintain state
• Transform/Project
– Change event schema, compute new fields


SPLASH scripting adds extensibility to CCL

SPLASH (Stream Processing LAnguage SHell) was introduced in Aleri to provide
extensibility and overcome the limitations of SQL

It complements CCL, providing the ability to write custom functions and stream/window
operators using a simple, procedural language that is optimized for stream processing

Syntax is similar to C and Java, it’s spirit is closer to little languages like AWK or Perl

ESP 5.0 has full support for SPLASH

Control Structures: Data Structures: Flex Streams:
if, while, switch
if (temperature < 0) { - Variables - Custom event “handlers”
display := 'below zero'; - Vectors written in SPLASH
} else if (temperature = 0) { - Dictionaries
display := 'zero';
} else { - Event Cache
display := 'above zero';
}

For more info see the SPLASH Tutorial on sybase.com (in the Aleri Product Documentation/Manuals)


Modularity

Benefits:
CREATE MODULE
Facilitates re-use • Defines a module consisting of
windows and streams
Simplifies team development • Define inputs, outputs,
parameters
Improves manageability of
large projects
IMPORT
• Equiv. to #include – include
external .ccl file(s)
• Beyond modules, re-use
declarations, etc

LOAD MODULE
• At load point, define bindings
and parameters
• Same module can be loaded
multiple times


Support for External function libraries (UDF)

• Call out to external function libraries

• Support for C and Java

• Can be used to supplement built-in functions

• Can also be used to “trigger” external actions


ESP Studio: RUN/TEST Tools

Single click to Run a project

Rich Suite of Testing, Tuning and Debugging tools
Stream viewer
Record/playback
Manual event input
Performance Monitor
Event Tracer
Debugger with Breakpoints

Multi-server control
View/monitor projects running on multiple servers and clusters


ESP Advanced Cluster Architecture

Manager Private
Cloud
Manager Manager

Manager

Controller Controller Controller

Project Project Project
Project Project
Project Project
Project Project


ESP Cluster Architecture
Key Features

Manager
• Equi-peer management of cluster.
• Program deployment and lifecycle control.
• Program load balancing at deployment.
• Launcher failure detection.
• Container failure detection.
• Program failover.
• Publishes lifecycle updates for Managers, Controllers and Programs.
• Name resolution.

Controller
• Acts as a daemon for Managers to launch Container processes.
• Allows for launching of any configured Program type.

Container
• Process that hosts a Program (i.e. – esp_server).


Choice of high availability configurations

FT Cluster
Manager

Configurable to your level of
availability and flexibility needs
Instant failover to "hot" spare
Synchronized
• highest degree of availability Hot Spare
• recovery in seconds; no data loss

Shared resources "warm" spares
• flexible configurations
• recovery in minutes
• parallel & pipeline configurations
Parallel & Pipelined Shared
State
Shared Warm Spares
All managed by fault tolerant
cluster manager


ADAPTERS

Standard Optional Custom
(Built-in) (Add-on) (SDK)
• JMS, MQ, • Market Data • C/C++
• Reuters, NYSE Tech
TIBCO • FIX • Java
• Sockets • Rep Server • .NET
• Databases
• Files
• MS Excel
• Email (out)


Integration/Connectivity

Application with
In-Process
External Adapters Native ESP
Adapters
Support
• Entirely under the • Independent of • Embed ESP
control of the ESP ESP Server pub/sub API
server. • Uses ESP directly in
• Performance pub/sub API producer or
advantages • TCP socket consumer
• No hot-standby at connectivity to applications
the adapter level ESP Server
• C/C++ only • Auto-failover in
hot-standby
configurations
• C/C++, Java,
.NET


USING Sybase Replication Server for Change Capture
Real-time change capture: turn database transactions into event streams

Transactions Database
(insert, update, Replication Replication
Agent Server Real-time Event
delete)
Stream
SAP Sybase
ESP


High Level ESP Architecture
Cluster Manager
Security Load Balancing Failover Lifecycle/Status

Distributed Cache SDK Interface URI Resolution

Client
Launcher
SDK - .NET JAVAC++
ESP Project

Security (Kerberos, PAM, SSL)
Adapter Query Engine

Command & Control
SQL Query
Gateway SDK
Studio Window Adapter
External
Stores Framework Adapter
Tools s
Internal
Adapter
Compiler s

Log
File Data
Source/Destination


What sets SAP Sybase ESP apart

• Performance • State Management
− Scalable for extreme throughput − Unique ability to automatically apply
− Consistent low-latency incoming events as
− Only CEP vendor to submit to STAC inserts/updates/deletes to a table
benchmarking − Simplifies modeling
• SPLASH scripting − Extremely efficient
− Overcome limitations of SQL • Dynamic
− Flexibility and productivity − Add continuous queries to live system
• Authoring flexibility • High Availability
− Language or UI/IDE − Hot-Hot with Auto-Failover
• “Private Cloud” Architecture • Advanced Subscriptions
− Scalable, Dynamic − Initial state followed by updates
− Subscriptions with predicates


High Performance
STAC Certified measurement of throughput and latency

The only CEP vendor to publish STAC benchmarks

• Linear scaling with addition of more
cores:
180,000 messages/sec on 16 cores
300,000 messages/sec on 24 cores

• Consistent/predicable low latency.
Only minor increase in latency from
20,000 to 300,000 messages/sec:
Mean: 1.296 - 1.568 ms
Medium: 1.177 - 1.502 ms
Std Dev: 0.567 - 1.104 ms
Min: 0.091 – 0.115 ms
99th percentile: 2.479 – 3.165 ms
Source: STAC Research Sep 08 – conducted against Aleri 3.1; 5.0 performance is
similar


PANOPTICON EX
REAL-TIME DATA VISUALIZATION

Visually Rich Real-time Dashboards with
Historical Drill-down and Analysis


Panopticon EX Features

Visual Data Analysis
Real-time streaming from Sybase ESP
Historic Time Series data from Sybase RAP or IQ

Drag-and-Drop Designer
Rich palette of charting objects
Simple configuration
No coding required

Multi-dimensional Analysis (in-memory OLAP)

Delivered via Desktop (thick client) or Web (browser)

SDK provides APIs for ability to customize, embed


Panopticon EX for ESP

Native integration with Sybase
ESP, RAP, and IQ Panopticon EX
Reduces Insight
True real-time dashboards, Latency
subscribing to ESP streaming
output  It’s Not Just the Time it
Takes to Get the Data
Seamless drill-down into  It’s also the time it
historical time series data in takes to understand the
Sybase RAP data
Rich Palette of
Visualization Tools
Designed to Make it
Easier to Understand the
Data


Thank you

Contact information:

F name MI. L name
Title
Address
Phone number

SAP Sybase Event Streaming Processing

More Related Content

What's hot (19)

Viewers also liked (6)

Similar to SAP Sybase Event Streaming Processing (20)

More from Sybase Türkiye (20)

Recently uploaded (20)

SAP Sybase Event Streaming Processing