DB2 for z/OS Update Data Warehousing On System Z

Data Warehousing on System z:
SQL news for Application Developers &
Meeting the need for real-time data for Business
Analytics

Copenhagen, 15.03.2011

Patric Becker
Data Warehousing on System z CoE

© 2011 IBM Corporation

1

Disclaimer
© Copyright IBM Corporation 2011. All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.

THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES
ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE
INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS” WITHOUT WARRANTY OF
ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT
PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM
SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE
RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS
PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY WARRANTIES OR
REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR ALTERING THE TERMS AND
CONDITIONS OF ANY AGREEMENT OR LICENSE GOVERNING THE USE OF IBM PRODUCTS AND/OR
SOFTWARE.

The information on the new product is intended to outline our general product direction and it should not be
relied on in making a purchasing decision. The information on the new product is for informational purposes
only and may not be incorporated into any contract. The information on the new product is not a
commitment, promise, or legal obligation to deliver any material, code or functionality. The development,
release, and timing of any features or functionality described for our products remains at our sole discretion.

IBM, the IBM logo, ibm.com, DB2, and DB2 for z/OS are trademarks or registered trademarks of International Business Machines Corporation in the
United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a
trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was
published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the
Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml

Other company, product, or service names may be trademarks or service marks of others.

2 © 2011 IBM Corporation

Agenda

Data Warehouse Challenges

Smart Analytics System 9600

Data Warehousing with DB2 9 and 10 for z/OS

Accelerating BI workloads with the Smart Analytics Optimizer

Additional thoughts and Summary


Agenda







Market Dynamics are shifting

Business Analytics is now mission critical
– Need to support broader users
– Users are more intense with increasing data access demands
– Requirements for high scalability, availability & performance

Asked to do more with less (IT & Business)
– Better access to relevant information By 2014, externalizing BI will
increasingly become an
– Need economies of scale expected aspect of most
– Consolidation with reduced complexity companies' relationships with
customers and partners.
Corporate regulatory compliance Externally focused BI programs
will frequently go beyond
– Driving intense scrutiny of data security policies information dissemination by
facilitating a collaborative
Environmental concerns still top of mind decision-making process that
breaks through the firewall to
involve stakeholders from
organizations in a broad
ecosystem
Source : Gartner, Prepare for Customer-Facing
6 Business Intelligence, © 2011 IBM,October 2010
Kurt Schlegel Corporation
6

What matters is changing
Results of New Intelligence Enterprise Survey of nearly 3,000 executives

Historic trend analysis Data
and forecasting visualization

Standardized Simulations and
reporting Scenario development

Analytics applied within
Data Business processes
visualization

Regression analysis,
Analytics applied within discrete choice modeling and
Business processes mathematical optimization

Simulations and Historic trend analysis
Scenario development and forecasting

Clustering and Clustering and Respondents were asked to
segmentation segmentation identify the top three
analytic techniques creating
value for the organization
and predict which three
Regression analysis, Standardized would be creating the most
discrete choice modeling and reporting value in 24 months.
mathematical optimization
Source: MIT Sloan Management Review,10 Data Points: Information and Analytics at Work, N Kruschwitz and R Shockley, Fall 2010


All departments, all users, in all roles across the
organization need access to business insights

How are we Why? What should
doing? we do next?

Real-time or historical; Guided or self-service Foresight using Statistical, and
operational or strategic access and exploration… Predictive Analytics…

Common Business Model

Message Application OLAP Modern and
8 Relational Legacy Sources © 2011 IBM Corporation
Sources Sources Sources
Sources

Information is at the Center…


Today, many business users are not getting to the
information they need, when they need it

60%+ of CEOs need to do a better job capturing and
understanding information rapidly in order to make swift
business decisions

27% of managers time is spend searching for information

50% of the information they obtain has no value to them

Sources: IBM & Industry Studies, Customer Interviews
IBM CIO Survey, June 19, 2007
Accenture survey, January 04, 2007

Key components for Business Analytics success
are being implemented & managed in isolation

Business
Data
Analytics
(Business Intelligence & Warehousing
Predictive Analytics)

Infrastructure


Today’s Traditional Infrastructure – a siloed approach
Division ‘A’ Division ‘B’
Marketing & R&D Users Marketing & R&D Users

DW & BA
DW & BA
Data Warehouse
Administrators & Data Warehouse Administrators &
BA Solution / Data Mart
Report Authors / Data Mart Report Authors
BA Solution

Finance

Executive
Management Operations Sales

Data Warehouse
/ Data Mart
BA Solution

Data Warehouse Data Warehouse DW & BA Administrators &
/ Data Mart / Data Mart Report Authors
BA Solution BA Solution

DW & BA Administrators & DW & BA Administrators &
Report Authors Report Authors

12 12 © 2011 IBM Corporation

Data Warehouses have become isolated
• Information to drive the business is known
but not available to the decision makers
• Information in the DW is limited to a small
number of people in the organization
• Little to no interactivity with other systems

• Not built with the same criteria as the
operational systems
• Difficult to manage and maintain multiple
servers and copies of the data
• Minimal control over who is accessing the
data

“Nearly 70% of data warehouses experience
performance-constrained issues of various types.”

- Gartner 2010 Magic Quadrant

Single version of the truth

• Single version of the truth is an
essential element for Analytics
and Business Intelligence

• Data redundancy elimination

• What deviation can be tolerated
to make business decisions ?

More than 98% human

Source: http://www.pbs.org/wgbh/evolution/library/07/3/l_073_47.html

Agenda







Simplicity, Flexibility, Choice
IBM Data Warehouse & Analytics Solutions
Flexible Custom Software,
Appliances Integrated Systems Hardware & Services

IBM Netezza IBM Smart Analytics System IBM InfoSphere Warehouse

Warehouse Accelerators

Information Management Entry Points
(Information Server, MDM, Streams, etc)

Simplicity The right mix of simplicity and flexibility Flexibility


IBM Smart Analytics System 9600
High Value Data Warehousing

Cognos 8 BI
System z
InfoSphere
Warehouse
Cubing
Services
Data
Warehouse
Powercube
Services

DB2 for z/OS VUE Implementation
ELT
(option for MLC) Services

DB2 Utilities Suite
Operational Source Systems Image Copy, LOAD, UNLOAD,
Structured/ Unstructured Data REORG, etc

17 3/16/2011 IBM Smart Analytics System 9600

High Value, Dynamic Warehousing Solution…
…mix and match
user roles…
…in tested, ready to deploy
configurations at a Bottom
Line Price
Data Capacity Sizing BI Capacity Sizing
Cognos Workload
DB2 Base
Named Max Concurrent
Hardware Amt
Usable
Amt
Users Users

User Data*
Disk* 10,000 100
Software
250TB 100TB

Services DB2 Utilities Suite
125TB 50TB
50
62TB 25TB
30TB 12TB
25
5
10TB 4TB

Choose from Each


Hardware: A single centralized BI environment
Building an end-to-end BI environment on System z

End-user functionality

BI tooling in z-Linux LPAR:
• InfoSphere Warehouse
• Cognos
• Optional:
• DataQuant/QMF
• IBI WebFocus, SAS
• Business Objects

z/OS foundation LPAR for enterprise BI:
• DB2 for z/OS – highly available, secure, query prioritization, concurrent utilities/queries
• z/OS: Highly secure, Efficient, Trusted, Responsive

IBM Smart Analytics System 9600 Software
Deeply Optimized by IBM Experts… Flexible Growth…

Powerful Data Warehouse and BI Software
DB2 for z/OS Value Unit Edition (primary) V9
Option for MLC
DB2 Utilities Suite V9
InfoSphere Warehouse on System z V9.5.2
Cognos 8 BI for Linux on System z V8.4
z/OS Operating System Stack V11

Optional Value Priced Add-ons
Tivoli OMEGAMON for DB2 Performance Expert
DB2 Connect
Tivoli Directory Server
InfoSphere Information Server
InfoSphere Replication Server
Q-Rep, CDC and Event Publisher eligible
InfoSphere Federation Server plus Classic Federation on System z
SPSS
Tivoli ITCAM, ITUAM
Cognos Now! For Linux on System z
Cognos Blueprints for Healthcare, Banking and others…
BI User on-boarding application (as proposed for Smart Analytics Cloud)

20 © 2011 IBM Corporation20

A Broad Range of Analysis, Dashboards & Reporting

Pre-optimized Business Intelligence Software
triples out of the box performance*

* Based on IBM Laboratory Tests. Actual results may vary
depending on specific environment and configuration.


Hardware and Software Services
• Services Components as part of the turn-key solution:
• Project Management
• Installation of Hardware
• Installation of Software Stack
• Testing & Validation of system in a table ready state
• Information Transfer
• Accelerated Value Program Advocate ongoing support

• Duration:
• Overall Software services engagement:
• Average 5 weeks effort without the Cognos configuration
• Cognos Configuration is 2-3 weeks
• Advocate Services for one-year following installation

Timeline example for installation in Texas:
Order to delivery: 4 weeks
Hardware installation / Software installation: 1 day / 2 to 3 weeks
Table ready and Ongoing support – 1 year
22 3/16/2011 IBM Smart Analytics System 9600

IBM Smart Analytics System – 9600 foundation
System z10 - Server Options:
• Choose - dedicated z10 server or bolt-on LPARs to existing server Now on
zEnterprise
• Includes general processor, zIIP and IFL engines to meet capacity requirements
• Supporting memory, channels, OSA card
• Fixed configurations based on amount of data, number of concurrent Cognos users
• Upgradeable as requirements grow
• Lab Services will define TWO LPARs
• z/OS LPAR - for DB2 for z/OS database delivering highly available, secure environment
• z/VM LPAR – multiple Linux virtual images – InfoSphere Warehouse, Cognos

Storage Subsystem:
DS8700 Storage Subsystem
• Included in the 9600 configuration:
• DEFAULT - Installs with fiber channel drives
• UPGRADEABLE with 2 additional disk options:
• Optional:
• Solid State Disk
• OR slower, more economical SATA drives
• mix and match for workload
• Automatic Data Management:
• DB2 Online Reorg automatically moves data to SSD
• EASY TIER automatically aligns data to disk types


IBM System z: System Design Comparison
System I/O Bandwidth
288 GB/Sec* Balanced System
CPU, nWay, Memory,
I/O Bandwidth*
172.8 GB/sec*

96 GB/sec

PCI for
Memory 24 GB/sec
1-way
3 TB**
1.5 TB** 512 GB 256 64 300 450 600 920 1202
GB GB

16-way

32-way
z196

54-way z10 EC

z9 EC
64-way
* Servers exploit a subset of its designed I/O capability zSeries 990
** Up to 1 TB per LPAR
80-way
PCI - Processor Capacity Index zSeries 900
Processors

z196 – The Core

• New 5.2 GHz Quad Core Processor Chip
boosts hardware price/performance

• 110+ new instructions –
improvements for CPU intensive,
Java™, and C++ applications
• Over twice as much on-chip cache
as System z10 to help optimize
data serving environment
• Out-of-order execution sequence
gives significant performance
boost for compute intensive
applications
• Significant improvement for Chip Area – 512.3mm2
floating point workloads
– 23.5mm x 21.8mm
• Performance improvement for systems with
large number of cores – improves MP ratio – 13 layers of metal
• Data compression and cryptographic – 3.5 km wire
processors right on the chip
– 1.4 Billion Transistors

Agenda







Information Management for System z

Smart Analytics Optimizer
OMEGAMON DB2 end–to-end monitoring 2011
DB2 10 for z/OS
DB2 Sort for z/OS
Guardium for System z
SPSS for System z
ISAS 9600 for System z
IMS Tools Solution Packs for z/OS Analytics
IMS 11 for z/OS 2010
InfoSphere Discovery for z/OS
Information-Led
InfoSphere Warehouse for System z Transformation
Cognos 8 BI for System z
InfoSphere MDM Server for System z Business Intelligence
Optim Data Governance for System z 2009
Information Server for System z
Information Agenda
2008
Information On Demand Data Warehousing


DB2 9 for z/OS – Data Warehousing was a major topic…
SHRLEVEL(REFERENCE) for Global query optimization DECIMAL FLOAT
REORG of LOB tablespaces Dynamic index ANDing BIGINT
Online RENAME COLUMN Generalizing sparse index and VARBINARY, BINARY
Online RENAME INDEX in-memory data caching method TRUNCATE TABLE statement
Online CHECK DATA and CHECK Optimization Service Center MERGE statement
LOB Autonomic reoptimization FETCH CONTINUE
Online REBUILD INDEX Logging enhancements ORDER BY and FETCH FIRST n
Online ALTER COLUMN LOBs network flow optimization ROWS in sub-select and full-
DEFAULT Faster operations for variable- select
More online REORG by length rows ORDER OF extension to ORDER
eliminating BUILD2 phase Index on expressions BY
Faster REORG by intra-REORG Universal Tablespaces INTERSECT and EXCEPT Set
parallelism Partition-by-growth tablespaces Operations
Renaming SCHEMA, VCAT, APPEND option at insert RANK, DENSE
OWNER, CREATOR Autonomic index page split Instead of triggers
LOB Locks reduction Different index page sizes Various scalar and built-in
Skipping locked rows option Support for optimistic locking functions
Tape support for BACKUP and Faster and more automatic DB2 Cultural sort
RESTORE SYSTEM utilities restart LOB File Reference support
Recovery of individual RLF improvements for remote XML support in DB2 engine
tablespaces and indexes from application servers such as SAP Enhancements to SQL Stored
volume-level backups Preserving consistency when Procedures
Enhanced STOGROUP definition recovering individual objects to a SELECT FROM
Conditional restart prior point in time UPDATE/DELETE/MERGE
enhancements CLONE Table: fast replacement Enhanced CURRENT SCHEMA
Histogram Statistics collection of one table with another IP V6 support
and exploitation Index compression Trusted Context
WS II OmniFind based text Index key randomization Database ROLEs
search Automatic creation of database
DB2 Trace enhancements objects
WLM-assisted Buffer Pools Temporary space consolidation
management zIIP enabled SQL Procedures
...

DB2 SQL 2009
z z/OS 9
common
luw Linux, Unix & Windows 9.7
Multi-row INSERT, FETCH & multi-row cursor UPDATE, Dynamic Scrollable Cursors, GET
z DIAGNOSTICS, Enhanced UNICODE SQL, join across encoding schemes, IS NOT DISTINCT
FROM, VARBINARY, FETCH CONTINUE, MERGE, SELECT from MERGE
Inner and Outer Joins, Table Expressions, Subqueries, GROUP BY, Complex Correlation, Global
c Temporary Tables, CASE, 100+ Built-in Functions including SQL/XML, Limited Fetch, Insensitive
Scroll Cursors, UNION Everywhere, MIN/MAX Single Index, Self Referencing Updates with
o Subqueries, Sort Avoidance for ORDER BY, and Row Expressions, 2M Statement Length, GROUP
BY Expression, Sequences, Scalar Fullselect, Materialized Query Tables, Common Table
m Expressions, Recursive SQL, CURRENT PACKAGE PATH, VOLATILE Tables, Star Join Sparse
Index, Qualified Column names, Multiple DISTINCT clauses, ON COMMIT DROP, Transparent
m ROWID Column, Call from trigger, statement isolation, FOR READ ONLY KEEP UPDATE LOCKS,
o SET CURRENT SCHEMA, Client special registers, long SQL object names, SELECT from INSERT,
UPDATE or DELETE, INSTEAD OF TRIGGER, Native SQL Procedure Language, BIGINT, file
n reference variables, XML, FETCH FIRST & ORDER BY in subselect & fullselect, caseless
comparisons, INTERSECT, EXCEPT, not logged tables, OmniFind, spatial, range partitions, data
compression, session variables, DECIMAL FLOAT, optimistic locking, ROLE, TRUNCATE, index &
XML compression, created temps
l Updateable UNION in Views, GROUPING SETS, ROLLUP, CUBE, more Built-in Functions, SET
u CURRENT ISOLATION, multi-site join, MERGE, MDC, XQuery, XML enhancements, array data
type, global variables, even more vendor syntax, LOB & temp table compression, inline LOB,
w administrative privileges, implicit casting, date/time changes, currently committed

MERGE
• MERGE instead of UPDATE and INSERT on condition

• Standard approach can be part of an incremental updating strategy while fixing
potential data inconsistencies at the same time

• Simplifies SQL coding


TRUNCATE TABLE

• Delete all rows from a table

• Issues with today’s solutions: LOAD REPLACE, DELETE

• Can be used to remove a not-logged TS from the LPL and to reset
RECP

• Applicable for Simple TS, Segmented TS, Partitioned TS and
Universal TS

• The performance ?

• TRUNCATE TABLE tablename


DB2 10 for z/OS – More Data Warehousing topics
Compress on the fly on INSERT Numerous optimizer Workfile spanned records, PBG
Auto-statistics enhancements, paging through support, and in-memory
result sets enhancements
Access path stability and hints
enhancements Parallel index update at insert More online schema changes for
Faster single row retrievals table spaces, tables and indexes
Access path lock-in and fallback via online REORG
for dynamic SQL Inline LOBs
Online REORG for LOBs
Automatic checkpoint interval LOB streaming between DDF and
rest of DB2 Online add log
Automated installation,
configuration & activation of Faster fetch and insert, lower Automatically delete CF
DB2 supplied stored procedures virtual storage consumption structures before/during first
& UDFs DB2 restart
DEFINE NO for LOBs and XML
Data set FlashCopy in COPY & Allow non-NULL default values
MEMBER CLUSTER for UTS for inline LOBs
inline copy
Query parallelism Loading and unloading tables
Inline image copies for COPY enhancements: lifting
YES indexes with LOBs in stream
restrictions
UNLOAD from FlashCopy Currently committed locking
Dynamic Index ANDing semantics
backup Enhancements
REORG enhancements More granular DBA privileges
Option to avoid index entry
Reduce need for creation for NULL value SQL compatibility
reorganizations for indices Index include columns Additional zIIP eligible workloads
CPU reductions Buffer pool enhancements
Hash access path Increased number of active
Versioned data threads
Reducing latch contention


DB2 SQL 2010
z z/OS 10
common
luw Linux, Unix & Windows 9.8
Multi-row INSERT, FETCH & multi-row cursor UPDATE, Dynamic Scrollable Cursors, GET
z DIAGNOSTICS, Enhanced UNICODE SQL, join across encoding schemes, IS NOT DISTINCT
FROM, VARBINARY, FETCH CONTINUE, MERGE, SELECT from MERGE, data versioning,
access controls
Inner and Outer Joins, Table Expressions, Subqueries, GROUP BY, Complex Correlation, Global
c Temporary Tables, CASE, 100+ Built-in Functions including SQL/XML, Limited Fetch, Insensitive
Scroll Cursors, UNION Everywhere, MIN/MAX Single Index, Self Referencing Updates with
o Subqueries, Sort Avoidance for ORDER BY, and Row Expressions, 2M Statement Length, GROUP
m BY Expression, Sequences, Scalar Fullselect, Materialized Query Tables, Common Table
Expressions, Recursive SQL, CURRENT PACKAGE PATH, VOLATILE Tables, Star Join Sparse
m Index, Qualified Column names, Multiple DISTINCT clauses, ON COMMIT DROP, Transparent
ROWID Column, Call from trigger, statement isolation, FOR READ ONLY KEEP UPDATE LOCKS,
o SET CURRENT SCHEMA, Client special registers, long SQL object names, SELECT from INSERT,
n UPDATE or DELETE, INSTEAD OF TRIGGER, Native SQL Procedure Language, BIGINT, file
reference variables, XML, FETCH FIRST & ORDER BY in subselect & fullselect, caseless
comparisons, INTERSECT, EXCEPT, not logged tables, OmniFind, spatial, range partitions, data
compression, session variables, DECIMAL FLOAT, optimistic locking, ROLE, TRUNCATE, index &
XML compression, created temps, inline LOB, administrative privileges, implicit casting,
l date/time changes, currently committed, moving sum & avg.
Updateable UNION in Views, GROUPING SETS, ROLLUP, CUBE, more Built-in Functions, SET
u CURRENT ISOLATION, multi-site join, MERGE, MDC, XQuery, XML enhancements, array data
w type, global variables, even more vendor syntax, LOB & temp table compression,

Running a Large Number of Active Threads
Today DB2 10

Coupling Technology Coupling Technology
LPAR1 LPAR2 LPAR3 LPAR1 LPAR2 LPAR3
DB2A DB2B DB2C DB2A DB2B DB2C
(500 thds) (500 thds) (500 thds) (2500 thds) (2500 thds) (2500 thds)

DB2D DB2E DB2F
(500 thds) (500 thds) (500 thds)

• More threads per DB2 image
• Data sharing and sysplex allows for • More efficient use of large n-ways
efficient scale-out of DB2 images • Easier growth, lower costs, easier
management
• Sometimes multiple DB2s per • Data sharing and Parallel Sysplex still
LPAR required for very high availability and scale
• Rule of thumb: save ½% CPU for each
member reduced, more on memory

Improved Availability – ALTER + REORG

Pending ALTER, Range-Partitioned
then online REORG UTS PBR
to make changes

Single-Table Classic Partitioned
Simple Table Space Single-Table
Table Space Segmented
Table Space
Page size
DSSIZE
SEGSIZE Partition-By-Growth
Hash
MEMBER CLUSTER UTS PBG

INDEX page size ADD active log
INCLUDE cols BUFFERPOOL PGSTEAL NONE

Compress on INSERT

• Data compression occurs when a dictionary exists
• Prior to DB2 V10
• Dictionary not built on a table space with COMPRESS YES attribute until:
• REORG or
• LOAD utility was executed
• For some customers, REORG or LOAD are not executed frequently
• DB2 V10 NFM allows for build of compression dictionary on:
• INSERT
• MERGE
• LOAD SHRLEVEL CHANGE
• No REORG or LOAD needed to get compressed data


Safe Query Optimization
• Bind/Prepare:
• Optimizer will evaluate the risk associated with each predicate
• For example: WHERE BIRTHDATE < ?
• Could qualify 0-100% of data depending on literal value used
• As part of access path selection
• Compare access paths with close cost and choose lowest risk plan
• Runtime:
• If a RID limit is reached
• Overflow RIDs to workfile and continue processing
• Avoid fallback to table space scan as in V9.
• Work-file usage may increase
• Mitigate by increasing RID pool size (default increased in DB2 10).

Additional Non-Key Columns in an Index
• Index is used to enforce uniqueness constraint on table
• To achieve index only access on columns not part of the unique constraint
during queries, additional indexes are often created for those columns not part
of the unique constraint
• Slower DB2 transaction time
• Increased storage requirements
• Additional Non-key Columns can be defined in an unique index to reduce total
amount of needed indexes
• Improves
• insert performance as less indexes needs to be updated
• space usage
• Can stabilize access path as optimizer has less similar indexes to chose from


Additional Non-Key Columns in an Index

• V9 definition
• CREATE UNIQUE INDEX i1 ON t1(c1,c2,c3)
• CREATE (UNIQUE) INDEX i2 ON t1(c1,c2,c3,c4,c5)
• Possible V10 definition
• CREATE UNIQUE INDEX ON t1(c1,c2,c3) INCLUDE (c4,c5)
• or
• ALTER INDEX i1 ADD INCLUDE (c4,c5) and DROP INDEX i2
• The following restrictions will apply:
• INCLUDE columns are not allowed in non-unique indexes
• Indexes on Expression will not support INCLUDE columns
• Indexes with INCLUDEd columns can not have additional unique columns
ALTER ADDed to the index


Versioned data - Current and History
Current SQL Application

Jul 2008
Aug 2008

Sep 2008
Sep 2008

Current
Current History
History
History
Generation

Transparent/automatic
Access to satisfy ASOF
Auditing SQL Application Queries
Using ASOF

New SQL functions

• Moving Sum, Moving Average
• Enhanced query parallelism technology for improved
performance
• Remove query parallelism restrictions
• In-memory techniques for faster query performance


Moving Sum, Moving Average

• OLAP Specification defined using a window.
• A window may specify 3 optional components

• Partitioning – PARTITION BY clause

• Ordering – ORDER BY clause

• Aggregation group – ROWS/RANGE


Sales_History Table
Territory Month Sales
East 199810 10
East 199811 4
East 199812 10
East 199901 7
East 199902 10
West 199810 8
West 199811 12
West 199812 7
West 199901 11
West 199902 6

Compute the average sales over the current month and the preceding
two months for each territory and month:

SELECT Territory, Month, Sales, AVG(Sales)
OVER(PARTITION BY Territory ORDER BY Month ROWS 2 PRECEDING)
AS Moving_Average
FROM Sales_History;


Territory Month Sales Moving_Average
East 199810 10 10
East 199811 4 7
East 199812 10 8
East 199901 7 7
East 199902 10 9
West 199810 8 8
West 199811 12 10
West 199812 7 9
West 199901 11 10
West 199902 6 8


Compute the average sales over the current month, the preceding
Month, and the following month for each territory and month:

SELECT Territory, Month, Sales, AVG(Sales)
OVER(PARTITION BY Territory
ORDER BY Month
ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING)
AS Moving_Average
FROM Sales_History;


Territory Month Sales Moving_Average
East 199810 10 7
East 199811 4 8
East 199812 10 7
East 199901 7 9
East 199902 10 8
West 199810 8 10
West 199811 12 9
West 199812 7 10
West 199901 11 8
West 199902 6 8


Compute the cumulative sum of sales in each territory, ordered by month:

SELECT Territory, Month, Sales, SUM(Sales)
OVER(PARTITION BY Territory ORDER BY Month ASC
ROWS UNBOUNDED PRECEDING)
AS Cumulative_Sum
FROM Sales_History;


Territory Month Sales Cumulative_Sum
East 199810 10 10
East 199811 4 14
East 199812 10 24
East 199901 7 31
East 199902 10 41
West 199810 8 8
West 199811 12 20
West 199812 7 27
West 199901 11 38
West 199902 6 44


DB2 Workloads that leverage zIIP
Portions of the following DB2 for z/OS workloads will benefit from zIIP:
1. ERP, CRM, Business Intelligence or other enterprise applications
• Via DRDA over a TCP/IP connection
• Workloads using DB2 Connect, T4 JCC Universal driver, Data Server
Client and CLI/ODBC, JDBC, SQLJ, .NET, pureQuery APIs
• Remote native SQL procedures (DB2 9 for z/OS NFM)
• XML parsing processing (DB2 9 for z/OS NFM)
2. Data warehousing / Business Intelligence applications
• CPU intensive parallel queries, including star schema queries
3. DB2 for z/OS utility functions used to maintain index structures and
DFSORT sort for some utilities
• Usage of zIIP is transparent to applications
• No changes to applications


What is the zAAP on zIIP capability?

• A new capability that can enable System z Application Assist Processor
(zAAP) eligible workloads to run on System z Integrated Information
Processors (zIIPs).
• For customers with no zAAPs and zIIPs
• The combined eligible workloads may make the acquisition of a single
zIIP cost effective.
• For customers with only zIIP processors
• Makes Java and z/OS XML System Services -based workloads eligible
to run on existing zIIPs – maximizes zIIP investment.

• Available September 25, 2009 with z/OS V1.11 and z/OS V1.9 and V1.10
(with PTF)
• This new capability is not available for z/OS LPARS if zAAPs are installed
on the server.


How to enable the zAAP on zIIP capability
• The capability ships default enabled with z/OS V1.11.
• Parameter in SYS1.PARMLIB(IEASYSxx) : ZAAPZIIP = YES (default in
z/OS V1.11)
• If you wish to disable the function for any reason, you must IPL with
ZAAPZIIP=NO in the IEASYSxx Parmlib member.

• Also available with z/OS V1.9 and V1.10
• With PTF for APAR OA27495, and
• Enabled with ZAAPZIIP=YES in the IEASYSxx Parmlib (the default is NO)

• This new capability does not remove the requirement to purchase and
maintain one or more general purpose processors for every zIIP
processor on the server.


Agenda







IBM Smart Analytics Optimizer
What is it? How is it different?
The IBM Smart Analytics Optimizer is a Performance: Unprecedented
workload optimized, appliance-like, add-on, that response times to enable 'train of
enables the integration of business insights into thought' analysis frequently blocked by
operational processes to drive winning poor query performance.
strategies. It accelerates select queries, with
unprecedented response times. Integration: Connects to DB2 through
deep integration providing transparency
to all applications.
Self-managed workloads: queries are
executed in the most efficient way
Transparency: applications connected
to DB2, are entirely unaware of the
Simplified administration: appliance-
like hands-free operations, eliminating
many database tuning tasks

Breakthrough Technology Enabling New Opportunities

Extreme performance for complex queries
Game Changing Performance
Rapidly delivers information to
decision makers through
breakthrough technologies providing
dramatic performance improvement.
Technology optimized for fast scans
Highly compressed data
Compressed data operations
Hardware-optimized data analysis
Massively parallel architecture

Enables decision makers to submit
queries they never dared in the past,
that analyze trends, predict
outcomes, and produce better
business results.
One Beta customer asked us to repeat a query under lab conditions
conditions
because he couldn‘t believe the acceleration. A query execution time was
couldn‘
reduced from 13 minutes, 42 seconds to just one second (end to end)!
end)!

Test Results with real Customer Data
• The problem queries were provided by a customer
• Expert database tunning done on all the queries
• Q1 – Q6 even after tuning run for too long and consume lots of resources
• Q7 improved significantly – no IBM Smart Analytics Optimizer offload is needed
• The provided workload was representing the IBM Smart Analytics Optimizer “Sweetspot”
perfectly.
• DB2 measurements were done on a System z9.

• The table shows elapsed and CPU times measured in DB2
(w/o Smart Analytics Optimizer)

56 3/16/2011 © 2011 IBM Corporation

Test Results with real Customer Data

57 3/16/2011 © 2011 IBM Corporation

Access plan comparison

Access plan graph
without IBM Smart
Analytics Optimizer:

Access plan graph with IBM
Smart Analytics Optimizer:


IBM Smart Analytics Optimizer - a virtual DB2 Component
Applications DBA Tools, z/OS Console, ...
Application Interfaces Operation Interfaces
(standard SQL dialects) (e.g. DB2 Commands)

DB2
IBM
Data Buffer Log Smart
... IRLM
Manager Manager Manager Analytics
Optimizer

System
optimized
for OLAP
Superior availability
processing
reliability, security,
workload management ...

DB2 for z/OS Smart Analytics
Optimizer
59
Appliance2011 IBM Corporation
©

IBM Smart Analytics Optimizer with multiple DB2 subsystems

System z196 CEC System z196 CEC

Data Sharing Grouop
LPAR LPAR LPAR LPAR

DB2 DB2 DB2 DB2 DB2

IBM Smart Analytics Optimizer


Routing Queries to the Optimizer
• Routing of Queries to the Smart Analytics Optimizer is
determined by the DB2 for z/OS Optimizer


Query Execution Flow

Worker 1

Worker 2

Worker 3

Worker 4

Worker n


Integration also into DB2 Commands:
DISPLAY THREAD

-DIS THD(*) ACC(*) DETAIL

DSNV401I # DISPLAY THREAD REPORT FOLLOWS -
DSNV402I # ACTIVE THREADS -
NAME ST A REQ ID AUTHID PLAN ASID TOKEN
BATCH AC * 231 BI ADMF001 DSNTEP2 0053 55
V666 ACC=ISAOPT1,ADDR=::FFFF:9.30.30.133..446:1076


Defining tables for acceleration

Define Transform


Comparing the Smart Analytics offerings in System z:
IBM Smart Analytics System 9600 IBM Smart Analytics Optimizer

Uses Services of Accelerator

GA June 2010 GA 4Q 2010

Complete, end-to-end environment for BI workload Workload optimized, appliance-like, add-on to a
Processes ALL queries submitted by end-users Data Warehouse on System z

Software: MUST connect TO a DB2 for z/OS environment that
is running a BI workload
Includes z/OS, DB2 for z/OS, Linux, InfoSphere Will enhance a Smart Analytics System 9600
Warehouse, Cognos, DB2 Connect
Software:
Supports:
Includes software running on optimized hardware
Data movement / ETL and connects to DB2 for z/OS to enable query
End user tools (Cognos) / BI queries/reports routing to the Smart Analytics Optimizer
Data Storage (Data warehouse)
Supports:
Runs in z/OS-DB2 LPAR, Linux for System z LPAR for (Subset of) complex and long-running BI queries
Tooling with access to data in a star or snowflake schema
Is an all purpose environment to deploy any BI Order of magnitude acceleration of a SUBSET of
workload queries that are selected/routed.

For acceleration of multidimensional star schema queries, use IBM Smart Analytics Optimizer as add on option


Agenda







A successful strategy requires the
integration of all key components

Business
Data
Analytics
(Business Intelligence & Warehousing
Predictive Analytics)

Keys to Success!

Infrastructure


ETL Solution Accelerates with Automation & Reuse
Member A Member B
Extract DataStage
OLTP OLTP
Load
Member C Member D ETL Accelerator
DWH DWH
zSeries
pSeries
CEC One CEC Two
CF
CF xSeries

• ETL is done using an ETL Accelerator like DataStage.

• E – the data is extracted from the OLTP information sources

• T – 100s of embedded transforms & custom built transforms are applied

• L – Aggregated and transformed data is loaded into the warehouse tables


Maintenance : Incremental Updating is Essential

Member A Member B
OLTP OLTP DataStage
Update & Insert
Member C Member D ETL Accelerator
Change Capture

DWH DWH
zSeries
MQ pSeries
CEC One CEC Two
CF
CF xSeries
MQ

MQ
z/OS

Data Event Publisher Capture

VSAM IMS

MQ
Distributed InfoSphere Changed Data Capture
Classic Data Event Publisher Capture

CA Software AG Capture
IDMS Adabas DB2 UDB Oracle Additional
LUW Sources


Workload Management

• Traditional workload management approach:

• Screen queries before they start execution

• Time consuming for DBAs.

• Some large queries slip through the crack.

• Running these queries degrade system performance.

• Cancellation of the queries wastes CPU cycles.


Workload Manager
Work is managed according to it’s Business Importance and Goals.
•You can prioritize work based on
• Business requirements
• Resource requirements – by allowing system to automatically lower
priority of queries as they consume system resources: period aging
• Limits the impact of unexpected long running queries

y
Low

rit
er p

io
Priority Period
rior

pr
Priority ity Aging

r
we
Lo
Business
Importance

ts
en
Short

ire e
High

qu urc
m
Medium

Re so
Medium Low
Long

Re
Business Importance


Prevent Query Monopolization with WLM
Avoiding Monopolization.
More sophisticated
WLM policy put in Allow long -running deep
place analytics to consume
spare cycles while
Query Monopoliztion often continuing to provide
witnessed w/un -sophisticated consistent service to
resource manager smaller consumers
600.0

500.0
100 = 1 processor engine

400.0
CPU

300.0

200.0

100.0

0.0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53
Time

When you have unexpected spikes in workload, will
you be able to handle it?
… Or do you have to micro-manage the system ? Business Analyst Large Business Analyst Medium
Business Analyst Small Opertnl BI
Data-Mining_Q


IBM zEnterprise 196: The heart of the new machine
The industry’s fastest and most scalable enterprise system Infrastructure

Dramatic improvement over IBM System z10™:

5.2 GHz superscalar
For Linux For z/OS
processor
Up to 96 Cores, 1 to 80
Up to
60%
Up to
40%
configurable for client use
Improvement in Improvement in
Up to 3 TB RAIM memory performance performance
Over 100 new instructions
1.5MB L2 Cache per core,
24MB L3 Cache per
for
35% with
60%
processor chip Less cost More capacity

Cryptographic enhancements
With no increase in energy consumption
Optional water cooling
And even better performance with new software

73
© 2010 IBM Corporation

IBM Smart Analytics Optimizer Data
Capitalizing on the best of relational and columnar databases
Warehousing
What is it? How is it different
The IBM Smart Analytics Optimizer is a • Performance: Unprecedented
workload optimized, appliance-like add-on that response times to enable 'train of
enables the integration of business insights into thought' analyses frequently blocked by
operational processes to drive winning poor query performance.
strategies. It accelerates select queries, with
• Integration: Connects to DB2 through
unprecedented response times. deep integration, providing transparency
to all applications.
• Self-managed workloads: queries are
executed in the most efficient way
• Transparency: applications connected
to DB2 are entirely unaware of the
accelerator
• Simplified administration: appliance-
like hands-free operations, eliminating
many database tuning tasks

Breakthrough Technology Enabling New Opportunities

IBM InfoSphere Warehouse Data
• Adds core DW and analytics capability to DB2 for z/OS Warehousing
• Advanced physical database modeling and design
• In-database data movement and manipulation capabilities
• Multidimensional reporting and analysis of data with Cubing Services

IBM DB2 Value Unit Edition
• Offers the same robust DB2 for z/OS data server at a one-time charge price
• Available for eligible net new applications workloads System z Other
MDX
Linux on System z
IBM InfoSphere Information Server Warehouse
DB2 z/OS
• Profiles, cleanses and integrates MS Excel
information from heterogeneous SQW Cubing
sources to drive greater business Services
insight faster, at lower cost SQW

Operational Source Systems

75
3/16/2011 IW Design Studio
75 75 75 3/16/2011 © 2011 IBM Corporation

IBM Cognos Business Intelligence
Business
for Linux on System z Analytics

• Full range of BI capabilities
• Query, reporting, analysis,
dashboarding, realtime monitoring

• Delivers information where,
when and how it is needed
• Self-service reporting and analysis
• Automated delivery of information in context
• Author once, consume anywhere

• Purpose-built SOA platform
that fits client environments and scales easily

76

Business
• IBM SPSS Analytics
• Full breadth of predictive analytics
• Data collection, statistics, data mining,
predictive modeling, deployment services…

• Putting prediction in hands of the
business
• Decision Management
• Driving better business outcomes
• Attract and retain more profitable customers
• Detect and prevent fraud
• Improve resource allocation

77

Some key Redbooks

• Enterprise Data Warehousing with DB2 9 for z/OS
• http://www.redbooks.ibm.com/abstracts/sg247637.html

• Co-Locating Transactional and Data Warehouse Workloads on System z
• http:// www.redbooks.ibm.com/abstracts/sg247726.html

• DB2 for z/OS: Data Sharing in a Nutshell

• System Programmer’s Guide To: Workload Manager

• Application Design for High Performance and Availability

• Workload Management for DB2 Data Warehouse, REDP-3927
• http:// www.redbooks.ibm.com/abstracts/redp3927.html

• DB2 9 for z/OS Performance Topics

IBM Professional Services
Accelerated Success
• New “how-to” books deliver expertise
• BI Strategy Book
• BI on Cloud
• BI Redbook
• Workshops to help shape strategy
• Champion workshops
• Business Analytics experience
• Services & Training
• Proven practice workshops, learning assessment
and user adoption services
• Broader portfolio of self-paced training options
• Growth in communities
• Innovation Center
• DeveloperWorks, C^3 Blog
• Twitter, facebook and linked-in


Miami-Dade County
Selects IBM System z platform to expand their IBM Cognos
8 BI enterprise infrastructure

…We are now able to expand the usage of our Business
Intelligence reporting. By the end of 2010, we will have users from
over 42 County departments with over 1500 users creating and
consuming reports with stable environments on System z.

—Jaci Newmark, Project Lead, Enterprise Business Intelligence Architecture,
Miami-Dade County

11 days to go from distributed to System z deployment model
Consolidated multiple BI deployments onto a single platform
Consolidate multiple, disparate data sources onto a single platform
Ensured 99.999% availability & complete disaster recovery plan


University of North Carolina Health Care
Deploys a hybrid data warehouse solution combining
the strengths of InfoSphere and DB2 software on
System z and System p platforms

With the deployment of the Carolina Data Warehouse for
Health, we have been able to increase the timeliness of
the information available to our researchers, staff and
physicians," "Because the system can also support
general queries that relate to the diagnosis and treatment
of a wide array of patients, we are now able to make more
intelligent decisions leading to improved patient care."

Donald Spencer, MD, MBA,
Associate Director of Medical Informatics, UNC Health Care


Numius’ Client Success Story
Uses DB2 for z/OS and Cognos 8 BI for Linux on System z

Produced 400 reports in the same time as 1 report in the old environment
400 reports ran in 45 minutes as opposed to 1 report in 46 minutes
Easily supported 130 concurrent users, as opposed to 8 on the
source environment
No reports timed out, not one user was rejected. Even when the system
slowed down, it remained stable
The client would not need to redesign his application to achieve his
objective of reaching out to a large community
This solution helps improve waste control in Belgium by informing
producers, consumers and authorities faster and more thoroughly
Jo Coutuer, Senior Business Intelligence Architect, Numius


IBM Cognos BI Total Cost of Ownership Study
Explores the TCO of choosing an x86 based infrastructure vs. System z
for a Cognos 8 BI deployment using proven IBM TCO measurement
methodology

Average savings over 5
years of with a System z
deployment: 36%
Reduction in high availability
costs with System z: 50%
System administration
savings alone pay for System z
investment.


Blue Insight,
The IBM internal Private Analytics Cloud

Our commitment to informed decision making led us to
consider private cloud delivery of Cognos via System z,
which is the enabling foundation that makes possible
+$25M savings over 5 years.

-IBM CIO Office

Consolidated 115 multi-product, departmental BI deployments to 1 Cognos 8 BI on System z
Support for our global workforce (2009: 72K, 2010: 130K, 2011: 200K)
Realizing value from +60 data sources across IBM
Projected $25M in savings (60% Consolidation, 35% Standardization, 5% Automation


Industry Analysts Agree
“In short, I believe that mainframes are the most modern
platforms available in the commercial marketplace today.”
Source: Clabby Analytics, Migrate From Mainframe? To What?, Joe Clabby, June 24, 2010

“Companies that buy into outdated hype about its complexity fail to
realize the potential performance gains associated with
mainframe use.”
Source: Aberdeen Group, The Fable of Mainframe Complexity, Max Gladstone,
May 5, 2010

“Clients wishing to evolve their legacy application portfolios into
more-modern technologies and architectures can do so on
the IBM mainframe.”
Source: Gartner, Mainframe Modernization: When the Platform Is the Solution,
Dale Vecchio, 8 January 2010

“The mainframe has long been recognized as a platform with an enviable reputation for
reliability, security, and efficiency, Ovum considers that IBM by exploiting these characteristics
has produced the next generation of data centre and cloud centre management platforms.”
Source: Roy Illsley, Principal Analyst, Ovum


Typical Utilization for Servers
Windows: 5-10% Unix: 10-20% System z: 85-100%

System z can help reduce your floor space
up to 75%-85% in the data center

For additional information please visit
www.ibm.com/software/data/businessintelligence/systemz/

System z can lower your total cost of ownership, requiring as little as 30%
of the power of a distributed server farm running equivalent workloads

The cost of storage is typically three times more in
distributed environments


IBM Product Portfolio
Business Analytics and Data Warehousing on System z
Business Intelligence Data Integration and Movement
• InfoSphere Information Server
Cognos 10 Business Intelligence • InfoSphere Change Data Capture
Predictive Analytics • InfoSphere Replication
SPSS Statistics 19 • InfoSphere Federation
• Global Name Recognition
SPSS Modeler
SPSS Collaboration and Deployment Master Data Management
Services • InfoSphere Master Data Management Server
Data Warehousing
DB2 for z/OS VUE (Value Unit Edition) InfoSphere Industry Models
• Banking, Insurance, Retail. Telco, Heath
InfoSphere Warehouse Payor, Heath Provider, Financial Markets
Flexible Deployment Options
*Solution Edition for Data Warehousing (pricing
option) • IBM Smart Analytics System 9600
• IBM Smart Analytics Cloud
• IBM Services


DB2 for z/OS Update Data Warehousing On System Z

More Related Content

What's hot (20)

Similar to DB2 for z/OS Update Data Warehousing On System Z (20)

More from Surekha Parekh (13)

DB2 for z/OS Update Data Warehousing On System Z