A Hybrid Technology Platform for Increasing the Speed of Operational Analytics

Ed Lynch – Executive Client Technical Professional
Data Warehousing and Business Analytics for System z
10 October 2012

A Hybrid Technology Platform for Increasing
the Speed of Operational Analytics

© 2009 IBM Corporation

Speaker Biography
Ed Lynch is an Executive Client Technical Professional specializing in IBM’s System z Data
Warehousing, Business Analytics, and Information Integration software products. Ed’s twenty-
eight year career with IBM has spanned many areas of IBM, and has always involved IBM's
Information Management (IM) products. His previous roles have included DB2 for z/OS
Development and Management, DB2 Technical Marketing, Development and Delivery of IM
Product Education, a Principal of Information Integration Design and Implementation
Consulting Services, and DB2 Tools & Information Integration Technical Sales.
Currently, Ed is the lead Technical Specialist for North America’s System z Data Warehousing
and Business Analytics, and Information Integration Software Technical Sales team.
Ed has worked extensively with DB2 across the various operating system platforms,
InfoSphere Data Replication, InfoSphere Classic Replication Server, DB2 Analytics
Accelerator, DB2 DataPropagator, InfoSphere Federation Server, InfoSphere Classic Federation
Server, DB2 Connect, IBM Information Server, and IBM’s Business Analytics solutions. In his
current role, he frequently works with product development in identifying and prioritizing
product requirements, and developing product strategy. He also provides software technical
sales support and works extensively with customers to create architectures using these
products.
edlynch@us.ibm.com
1-972-561-9975


Abstract

With the wealth of data available today, organizations are no longer willing to relegate
information to the back office. Modern organizations are demanding access to information.
However, it is not enough to capture information, users must be quickly able to sift through
these massive amounts of data, extract information and transform it into actionable
knowledge. Systems today are enabling organizations to anticipate risk, identify threats,
assess readiness, and match the risk assessment to the resources required to address
them; all at the time of decision. They use a platform that provides the ability to react to
changes decisively, based upon the facts of the situation, not in hours or days- but at the
moment of opportunity. They optimize decisions based upon current weather conditions,
past threats and behaviors and current resource availability to assure a successful
operation.This session will review the architecture and benefits of a hybrid system of MPP
and SMP technologies enabling the merging of fit for purpose and mixed workload
capabilities into a single system. See how this hybrid system facilitates both transaction-
oriented applications and analytics into a single platform for operational analytics. Find out
why these enhancements are the next logical steps in creating a highly optimized
environment, both in price and performance, that is designed to meet the wide range of
analytic workloads that today's organizations need to accommodate.


DB2 Analytics Accelerator V3 More insight from your data
Further extending the features
• Unprecedented response times for
“right-time” analysis

• Complex queries in seconds rather
than hours

• Transparent to the application

• Inherits all System z DB2 attributes

• No need to create or maintain indices

• Eliminate query tuning
Blending System z and Netezza
• Fast deployment and time-to-value
technologies to deliver unparalleled,
mixed workload performance for
complex analytic business needs.

4 © 2009 IBM Corporation

DB2 Analytics Accelerator
Train-of-thought Analytics

FAST Cost Saving Appliance
Complex queries run Eliminate costly No applications to
up to 2000x faster query tuning while change, just plug it
while retaining single offloading complex in, load the data,
record lookup speed query processing and gain the value


Introducing
DB2 Analytics Accelerator V3
Reducing the Cost of High Speed Analytics

Improve Productivity Lower Host Costs Consolidate
Eliminate query tuning Reduce storage costs Reduced complexity

Eliminate table indexing Offload query processing Reduced software costs

Minimize storage admin Defer system upgrades Reduced hardware costs


Fast Time to Value
IBM DB2 Analytics Accelerator
Production ready - 1 person, 2 days
Table Acceleration Setup 2 Hours
– DB2 “Add Accelerator”
– Choose a Table for “Acceleration”
– Load the Table (DB2 copy to Netezza)
– Knowledge Transfer
– Query Comparisons
Initial Load Performance
400 GB “Loaded” in 29 Min
570 million rows (Loads of 800GB to 1.3TB/Hr)
Actual Query Acceleration 1908x faster
2 Hours 39 Minutes to 5 Seconds
CPU Utilization Reduction
35% to ~0%
Actual customer results, October 2011


Performance & Savings
DB2 w ith Times
Faster
Queries run faster
DB2 O nly IDAA
Total Total • Save CPU resources
Row s Rows
Query Reviewed Returned Hours Sec(s) Hours Sec(s) • People time
Query 1 2,813,571 853,320 2:39 9,540 0.0 5 1,908
Query 2 2,813,571 585,780 2:16 8,220 0.0 5 1,644 • New Business
Query 3 8,260,214 274 1:16 4,560 0.0 6 760 opportunities
Query 4 2,813,571 601,197 1:08 4,080 0.0 5 816
Query 5 3,422,765 508 0:57 4,080 0.0 70 58
Query 6 4,290,648 165 0:53 3,180 0.0 6 530
Query 7 361,521 58,236 0:51 3,120 0.0 4 780
Query 8 3,425.29 724 0:44 2,640 0.0 2 1,320
Query 9 4,130,107 137 0:42 2,520 0.1 193 13

DB2 Analytics Accelerator: “we had this up and
running in days with queries that ran over 1000
times faster”
DB2 Analytics Accelerator: “we expect ROI in
less than 4 months”
Advance to 32 minute mark for DB2
Analytics Accelerator section of keynote

Accelerating decisions to the speed of business
8 12 October 2012
Actual customer results, October 2011 © 2009 IBM Corporation

IBM DB2 Analytics Accelerator V3 Product Components

Netezza
zEnterprise Technology
CLIENT

Data Studio
Foundation
DB2 Analytics
Accelerator Network
Admin Plug-in
OSA- Primary
Express3 10Gb
10 GbE
Backup

IBM DB2
Data Warehouse application
Analytics
DB2 for z/OS enabled for IBM
Users/ Acelerator
DB2 Analytics Accelerator
Applications


Deep DB2 Integration within zEnterprise
Applications DBA Tools, z/OS Console, ...
Application Interfaces Operational Interfaces
(standard SQL dialects) (e.g. DB2 Commands)

DB2 for z/OS

IBM
Data Buffer Log DB2
Manager Manager
... IRLM
Manager Analytics
Accelerator

Superior availability Superior
reliability, security, z/OS on performance on
Workload management System z analytic queries

Netezza


TM
Accelerator powered by Netezza 1000 Appliance
Slice of User Data
Swap and Mirror partitions
High speed data streaming
High compression rate
EXP3000 JBOD Enclosures
12 x 3.5” 1TB, 7200RPM, SAS (3Gb/s)
Disk Enclosures max 116MB/s (200-500MB/s compressed data)
e.g. 1000-12: 8 enclosures → 96 HDDs(32/128 TB)

Accelerator Server
SMP Hosts SQL Compiler, Query Plan, Optimize,
Administration
2 front/end hosts, IBM 3650M3 or 3850X5
clustered active-passive
2 Nehalem-EP Quad-core 2.4GHz per host
Snippet BladesTM
(S-Blades, SPUs)
Processor & streaming DB logic
High-performance database engine
streaming joins, aggregations, sorts, etc.


S-Blade™ Components Dual-Core FPGA
8 FPGA Processors/Blade

Netezza DB Accelerator

Intel Quad-Core
8 Cores/Blade

IBM BladeCenter Server

Eliminating the I/O Bottleneck
Move the SQL to the hardware to where the data lives

“Just send
the Answer,
not Raw
Data”

select DISTRICT,PRODUCTGRP,
The Key to the Speed sum(NRX)
from MTHLY_RX_TERR_DATA
where MONTH = '20091201'
and MARKET = 509123
and SPECIALTY = 'GASTRO'
FPGA
CPU Core
Core Zone Map

Complex ∑ Restrict, Project
Joins, Aggs, etc. Uncompress
Visibility
Slice of table
MTHLY_RX_TERR_DATA
(compressed)

sum(NRX) where MONTH = '20091201' select DISTRICT,
and MARKET = 509123 PRODUCTGRP,
and SPECIALTY = 'GASTRO' sum(NRX)

Bringing Netezza AMPPTM Architecture to DB2 for z/OS
AMPP = Asymmetric Massively
Parallel Processing

CPU FPGA

Advanced Memory
Analytics

BI SMP CPU FPGA
Host
DB2 for z/OS Memory

Legacy
Reporting

CPU FPGA

DBA Memory

Network Disk
Fabric S-Blades™ Enclosures


Query Execution Process Flow

Application Optimizer
Interface
SPU
CPU FPGA

Memory

Accelerator DRDA Requestor
SPU
CPU FPGA

SMP Host
Memory

Application SPU
Query execution run-time for CPU FPGA

queries that cannot be or should Memory
not be off-loaded to Accelerator
SPU
CPU FPGA

Memory

DB2 for z/OS DB2 Analytics Accelerator

Queries executed without DB2 Analytics Accelerator
Queries executed with DB2 Analytics Accelerator

Workload-Optimized Query Execution
• Single and unique system
DB2 for z/OS and for mixed query workloads
• Dynamic decision for most
OLTP-like query
OLTP-like query
efficient execution platform
User control and DB2 heuristic

• New special register
DB2 Native
QUERY ACCELERATION
DB2 Native
Light ODS-
Light ODS- Processing
Processing – NONE
query
query
– ENABLE
– ENABLE WITH FAILBACK
• New heuristic in DB2
Light BI Query
Light BI Query optimizer

Heavy BI Query
Heavy BI Query Optimized processing
for BI Workload


Accelerator Data Load
DB2 for z/OS Accelerator

Table A
Table B

CPU FPGA
Part 1 Unload USS Pipe
Accelerator Administrative Stored

Memory
Table C

Accelerator CPU FPGA
Procedures

Studio Table D

Coordinator
Memory

Part 1
. . .
CPU FPGA
. . .
Memory
Part 2
. . .
CPU FPGA
Part m Memory

• 1 TB / h – can vary, depending on CPU resources, table partitioning,
• Update on table partition level, concurrent queries allowed during load
• V2.1 & V3 unload in DB2 internal format, single translation by accelerator

DB2 Analytics Accelerator V3
Lowering the Costs of Trusted Analytics

What’s New? • zEnterprise EC12 Support
• High Performance Storage Version 3 will support the zEnterprise
Saver EC12, z196 and z114 System z
platforms
Store a DB2 table or partition of data
solely on the Accelerator. Removes • Query Prioritization
the requirement for the data to be
Brings System z workload
replicated on both DB2 and the
management down to the individual
Accelerator
query being routed to the Accelerator
• Incremental Update
• High Capacity
Enables tables within the Accelerator
Support has been extended to include
to be continually updated throughout
the entire Netezza 1000 line (1.28 PB)
the day.
• UNLOAD Lite
Reduces z/OS MIPS consumption, by
moving the preparation off System z.


Build a System z Trusted Analytic System
Reduce the cost of host storage for historical data by 95%!

Historical High Performance Low Latency Data
Most data in an analytic All aggregate queries run Tables and partitions that
system is historical and not at the same high speed as require updating will be
subject to change. Most any accelerator supported able to be updated by
data can be in a Storage query incremental update, table
Saver and maintain trusted load or partition load
performance and security


High Performance Storage Saver
Reducing the cost of high speed storage

Store historic data on the Accelerator only
Applications
Tables can be resident on:
1. DB2 Only
2. DB2 and Accelerator
3. Accelerator Only
SQL

When data no longer
requires updating, reclaim
DB2 DB2
the DB2 storage
Accelerator
Table A Table A Table A

Special Registers control behavior
High speed High speed CURRENT QUERY ACCELERATION
indexed aggregate
lookups, best Accelerator lookups, best for CURRENT GET_ACCEL_ARCHIVE
for OLTP
Table A complex DSS type
type queries queries Managed by zParms

21 Mixed workload type queries

Save Over 95% of Host Disk Space for Historical Data
Historical Data
Year Year -1 Year -2 Year -3 Year -4 Year -5 Year -7

1Q 1Q 1Q 1Q 1Q 1Q 1Q

2Q 2Q 2Q 2Q 2Q 2Q 2Q

3Q 3Q 3Q 3Q 3Q 3Q 3Q

4Q 4Q 4Q 4Q 4Q 4Q

Current Data

4Q One Quarter = 3.57% of 7 years of data
One Month = 1.12% of 7 years of data
One month = 2.78% of 3 years of data

High Performance Storage Saver
Reducing the cost of high speed storage
Time-partitioned tables where:
– only the recent partitions are used in a transactional context (frequent data
changes, short running queries)
– the entire table is used for analytics (data intensive, complex queries).
DB2 partitions are deleted after the High Performance Storage Saver are created
on the accelerator

DB2 No longer present on DB2 Storage
Query from
Application
#1

Or

Accelerator Accelerator Accelerator Accelerator Accelerator Accelerator Accelerator
#1 #2 #3 #4 #5 #6 #7


The Evolution of a High Performance Storage Saver
High Speed Access to Historical Data

Table / Data Accelerator Accelerator Archive
Creation Load / Update /IU Only Only

DB2 DB2 Accelerator
Table A Table A Table A

Accelerator
Table A

Backup Backup

24

Storage options to match data needs
Optimized in both price and performance for differing workloads

High Performance Storage Saver Database Resident Partitions
Single Disk Store Dual Disk Store
• Only stored on Accelerator storage (Less • Stored on both DB2 and Accelerator
Cost) storage
• Optimized performance for • Mixed query workload with transactions,
deep analytics, multifaceted, reporting single record queries and record updates
and complex queries with deep analytics, multifaceted,
• Only full table update or full partition reporting and complex queries.
update from backup • Full table, full partition update, Incremental
• Same high speed query access update from DB2 data
transparently through DB2 • Same high speed query access
transparently through DB2

Cost The right mix of cost and functionality Functionality
25

The zEnterprise Hybrid Solution
Mixed Workloads for Next Generation Business Analytics
Operational Analytic Mixed Workload
Applications Applications Applications

Transaction Processing Data warehousing Operational BI

Shared Everything DB Shared Nothing DB Hybrid DB

High volume business Low volume complex High volume business
transactions and batch queries context switching transactions and batch
reporting running reporting running
concurrently concurrently with complex
queries


Incremental Update

Table or
ELT or ETL Partition Update
OLTP Data DB2 Analytics
Application Warehouse Accelerator

Data Incremental
Replication Update

Synchronizing data to lower data latency
from days to minutes/seconds

Option 1: Full Table Refresh
Changes in data warehouse tables typically
driven by scheduled (nightly or more
frequently) ETL process
Data used for complex reporting based on
consistent and validated content (e.g., weekly
Operational Analytics, Reports, OLAP,
transaction reporting to the central bank)
Multiple sources or complex transformations Continuous
Continuous
Query
prevent propagation of incremental changes Query
Processing
Processing

Full table refresh triggered through DB2 stored
procedure (scheduled, integrated into ETL DB2 z/OS Query Optimizer
DB2 z/OS Query Optimizer
process or through GUI)

DB2 native
DB2 native Accelerator
Accelerator
processing
processing processing
processing
ETL Process

Queries may continue
ETL Process

during full table refresh
Full table refresh
for accelerator
DB2 for z/OS database
Changes / Replacement © 2009 IBM Corporation

Option 2: Table Partition Refresh
Changes in data warehouse table typically driven by “delta” ETL process (considering only
changes in source tables compared to previous runs) or by more frequent changes to most
recent data
Optimization of Option 1 when target data warehouse table is partitioned and most recent
updates are only applied to the latest partition

Table partition refresh triggered through DB2
stored procedure (scheduled, integrated into
Continuous
Continuous
ETL process or through GUI) Query
Query
Processing
Processing

Maintains snapshot
semantics for consistent
reports
Queries may continue DB2 native
Accelerator
Replication
Replication

processing
processing
during table partition
refresh for accelerator January

February

March
ETL Process
ETL Process

April

May Partition refresh

Changes
DB2 for z/OS database © 2009 IBM Corporation

Option 3: Incremental Update
Changes in data warehouse tables typically
driven by replication or manual updates
– Corrections after a bulk-ETL-load of a data warehouse
table
– Continuously changing data (e.g. trickle-feed updates from
a transactional system to an ODS)

Reporting and analysis based on most recent
data
May be combined with Option 1 & 2 (first table Continuous
Continuous
refresh and then continue with incremental Query
Query
Processing
Processing
updates)

Application
Application

DB2 native
Accelerator
Incremental update can be processing
processing
configured per database
table
Replication
Replication

Incremental Update
Changes
DB2 for z/OS database © 2009 IBM Corporation

Now expandable to 960 cores and 1.28 petabytes
1 10

.......

002 005 010 015 020 030 040 060 060 100

Cabinets 1/4 1/2 1 1 1/2 2 3 4 6 8 10

Processing Units 24 48 96 144 192 288 384 576 768 960

Capacity (TB) 8 16 32 48 64 96 128 192 256 320

Effective
32 64 128 192 256 384 512 768 1024 1280
Capacity (TB)*

PureData System for Analytics

Predictable, Linear Scalability throughout entire family

Capacity = User Data space
Effective Capacity = User Data Space with compression *: 4X compression assumed
Low Latency, High Capacity Update © 2009 IBM Corporation

Connectivity Options DB2 DB2
Multiple DB2 systems can connect to a single Accelerator

A single DB2 system can connect to multiple Accelerators DB2

Multiple DB2 systems can connect to multiple Accelerators DB2 DB2

The same table can be stored in the multiple Accelerators
(except High Performance Storage Saver tables) Full flexibility for DB2 systems:
• residing in the same LPAR
Better utilization of Accelerator resources • residing in different LPARs
• residing in different CECs
Scalability • being independent (non-data sharing)
High availability • belonging to the same data sharing group
• belonging to different data sharing groups

32

Analytics Accelerator Table Definition and Deployment
IBM Data Studio Client DB2 for z/OS DB2 Analytics
Accelerator
Accelerator
Accelerator Administrative Netezza Catalog
Studio Stored Procedures

DB2 Catalog

The tables need to be defined and deployed to the Accelerator before data is loaded and queries sent to it for
processing.
Definition: identifying tables for which queries need to be accelerated
Deployment: making tables known to DB2, i.e. storing table meta data in the DB2 and Netezza catalog.
IBM DB2 Analytics Accelerator Studio guides you through the process of defining and deploying tables, as well as
invoking other administrative tasks.
IBM DB2 Analytics Accelerator Stored Procedures implement and execute various administrative operations such
as table deployment, load and update, and serve as the primary administrative interface to the Accelerator from
the outside world including Accelerator Studio.


Shielding Against Disk Failures

Primary

Mirror

Temp

• All user data and temp space mirrored
• Disk failures transparent to queries and transactions
• Failed drives automatically regenerated
• Bad sectors automatically rewritten or relocated


Shielding Against S-BladeTM Failures

. . . . .
. . . . .
. . . . .

S-Blades

• S-Blade failure is automatically detected


Shielding Against S-BladeTM Failures

. . . . .
. . . . .
. . . . .

S-Blades

• Drives automatically reassigned to active S-Blades within a chassis
• Read-only queries (that have not returned data yet) automatically restarted
• Transactions and loads interrupted
• Loads automatically restarted from last successful checkpoint


Disaster Recovery Option 1 – Table Loaded in One Accelerator (1 of 2)
SYSPLEX
App 1 DSG Member 1 DSG Member 2

Tables Tables App 4
of App 4 of App 5
App 2

Tables Tables Tables App 5
App 3 of App 1 of App 2 of App 3

Short Range Short Range
Long
Switch Range
Switch


Accelerator Instance 1 Accelerator Instance 2

Tables Created
but Not Loaded Tables Tables Tables
of App 1 of App 2 of App 3

Tables Tables Tables Tables Tables
of App 1 of App 2 of App 3 of App 4 of App 5


Disaster Recovery Option 1 – Table Loaded in One Accelerator (2 of 2)
App 1
SYSPLEX
App 2
Tables Tables
of App 4 of App 5
App 2
App 3
Tables Tables Tables
App 4
Long App 5
Switch Range
Switch



of App 1 of App 2 of App 3
Already Created
Tables Tables Tables Must LOAD Tables Tables
of App 1 of App 2 of App 3 of App 4 of App 5


Disaster Recovery Option 2– Table Loaded in Two Accelerators (1 of 2)
SYSPLEX

Tables Tables App 4
of App 4 of App 5
App 2

Tables Tables Tables App 5

Long
Switch Range
Switch



Tables Tables Tables Tables Tables Tables
of App 1 of App 2 of App 3 of App 1 of App 2 of App 3

Tables Tables Tables Tables
of App 4 of App 5 of App 4 of App 5


Disaster Recovery Option 2 – Table Loaded in Two Accelerators (2 of 2)
App 1
SYSPLEX
App 2
Tables Tables
of App 4 of App 5
App 2
App 3
App 4
Long App 5
Switch Range
Switch



Data Already
Available
Tables Tables Tables Tables Tables Tables
of App 1 of App 2 of App 3 of App 1 of App 2 of App 3

Tables Tables Tables Tables
of App 4 of App 5 of App 4 of App 5


Why Both?
Marrying the best of both worlds
IBM IBM
PureData N1001 System z

Focused Appliance Mixed Workload System

Capitalizing on the strengths of both platforms while driving to the most
cost effective, centralized solution - destroying the myth that transaction
and decision systems had to be on separate platforms

Very focused workload Very diverse workload


Tailored to your needs
A Hybrid Solution
IBM
IBM System z with
Netezza IBM DB2 Analytics Accelerator

Focused Appliance Mixed Workload System
• Mixed workload system z with operational
• Appliance with a streamlined transaction systems, data warehouse,
database and HW acceleration for operational data store, and consolidated
performance critical functionality data marts.
• Price/performance leader • Unmatched availability, security and
recoverability
• Speed and ease of deployment and
• Natural extension to System z to enable
administration pervasive analytics across the
• Optimized performance for organization.
deep analytics, multifaceted, reporting • Speed and ease of deployment and
and complex queries administration

True Appliance Flexible Integrated System Custom Solution

Simplicity The right mix of simplicity and flexibility Flexibility


Next Steps
Opportunity

Operational New Workload Data
Reporting Consolidation Operational BI
(Accelerator) or Winback
(ISAS + (ISAS + (ISAS +
OR
Accelerator) Accelerator) Accelerator)
Existing z
Warehouse

Warehouse
Workload Collaboration
Assessment Workshop

Optional POC
43

The Ultimate Consolidation Platform
Data Mart Data Mart Data Mart Data Mart Bringing it all together
• Better Business Response
Data Mart Consolidation • Reduced Costs
System z PR/SM
Recognized leader in mixed • More Available
virtualization and workload isolation
• More Secure
Transaction Systems • Reduced Data Movement
(OLTP)
• Better Governance
• Reduced Data Latency
• Reduced Complexity

Data Warehousing
• Reduced Resources
z/OS: Netezza:
Business Intelligence Recognized leader in mixed Recognized leader in
Predictive Analytics workloads with security, cost-effective high
availability speed deep analytics
and recoverability

Together:
Destroying the myth that transactional and decision support
workloads have to be on separate platforms

Learn More

Visit the Data Warehousing &
Business Analytics Webpage
http://www.ibm.com/software/data/businessintelligence/systemz/


Ed Lynch
System z Data Warehousing & Business Analytics
edlynch@us.ibm.com


A Hybrid Technology Platform for Increasing the Speed of Operational Analytics

More Related Content

What's hot (20)

Similar to A Hybrid Technology Platform for Increasing the Speed of Operational Analytics (20)

More from IBMGovernmentCA (20)

Recently uploaded (20)

A Hybrid Technology Platform for Increasing the Speed of Operational Analytics