IBM Operations Analytics For z Systems V2.2 - Client Long Pres

© 2015 IBM Corporation
z Systems Software
IBM Operations Analytics for z Systems
Transforming Data into Insights
The Next Generation of IT Service
Management

z Systems Software
Note to the presenter …
This deck contains SEVERAL slides. It is intended
to be modified by YOU to include the slides that
YOU need for YOUR specific customer presentation.
If you are just looking for the high level overview of
IOAz (‘short’ deck) OR the 1-pager slide, please retrieve
those from the sales kit:
http://w3.ibm.com/software/spcn/content/F080197Y39289C99.html
2

z Systems Software
Agenda
 Why IT Analytics?
 Overview of IBM Operations Analytics for z Systems
– Functional capability
– What’s New in 2015
– Architecture
– Out-of-the-box Value
– Customize to meet your needs
– Integration with Service Management tooling
 Additional Detail
– Bring Your own Data – Example using HMC log
– IOAz V2.2 Details
– CICS insights
– Network insights
– Security insights
– Log Forwarder improvements
3

z Systems Software
Solution Branding
4
This solution was previously branded as IBM SmartCloud Analytics - Log Analysis.
The support to search and analyze z/OS logs was initially provided in March, 2014
under the following product names:
IBM SmartCloud Analytics - Log Analysis z/OS - Insight Packs – SYSLOG V1.1
IBM SmartCloud Analytics - Log Analysis z/OS - Insight Packs - IBM WebSphere® Application Server V1.1
Subsequent releases were named with the SmartCloud brand until April 2015, when
Version 2 of the product was rebranded to
Initial release under the new name:
IBM Operations Analytics for z Systems v2.1 (GA on April 24, 2015)
Current release:
IBM Operations Analytics for z Systems v2.2 (GA on October 16, 2015)
Note that the distributed version of the product is now named
IBM Operations Analytics – Log Analysis

z Systems Software
5
Systems scaling model and security to manage and optimize both
Systems of
Record
Social, Mobile, Analytics
Smarter Infrastructure
 Business Transactions
 Quality of Service
 Command & Control
 Facts and data “source of truth”
 z/OS Systems
 Mobile and Social
 Dynamic
 Interactions and
Collaboration
 Insight, trends, analytics
Systems of
Engagement
Rapid growth of data from latest technologies can be
supported seamlessly on z Systems

z Systems Software
Analytics for System z addresses rapid growth of data
and next generation technology
6
 Much greater amount of critical IT operational data
(SMF, log, journal) than distributed-only environments.
– Focus on problem determination and time to resolution
while placing premium on availability of services and
applications.
– 100x to 1000x explosion in data flooding existing tools.
– New runtimes, programming languages needing
complex instrumentation.
 By 2016, 40% of Global 2000 enterprises will have
IT operations analytics architecture in place, up from
< 1% in 2014, looking to integrate across their
enterprise to reduce outages (Gartner).
 90% of the Fortune 1000 companies are running z
and have ‘Systems of Record’ dependencies for
transactional processing and data serving applications.

z Systems Software
Is managing IT
today like sipping
from a fire hose?
New Technologies like cloud, mobile and big data already
challenging current Enterprise tools
7
 Too long to isolate, diagnose problems in
applications and infrastructure.
– Complex application workloads span multiple platforms
– Increasing amounts of IT data: Performance metrics,
events, infrastructure logs, application logs,
configuration files, traces
 Existing IT tools need additional data analysis
capabilities to manage of Systems of Engagement
– 100x to 1000x explosion in data flooding existing tools.
– New runtimes, programming languages needing
complex instrumentation.
 Reactive analytics misses critical information
leading to outages
– Need to move to a more proactive model
– Analysing ALL information better for predicting
problems.

z Systems Software
Predict:
Pro-Active Outage Avoidance
Predict problems before they occur
Search & Analyze:
Quickly search and analyze large volumes of data from a single search bar
Perform log and performance analysis while searching
Correlate messages from multiple logs for end-to-end problem diagnosis
Optimize:
Improve performance across IT Infrastructure
IBM is focused on managing end-to-end analytics for
improved performance and workload management
8
IBM Analytics solutions for z Systems
Predict
IBM zAware
Proactive Outage Avoidance
Search &
Analyze
IBM Operations
Analytics for z Systems
Faster Problem Resolution
Optimize
IBM Capacity
Management Analytics
(CMA)
Optimized Performance

z Systems Software
9
 Analyze various types of data (logs, metrics, events, trouble tickets) from
multiple sources (mainframe and distributed)
 Locate problems from system, configuration, software logs and performance
metrics using rapid index search and pattern analysis
 Isolate issues across various domains including OS, Middleware, applications, etc.
 Leverage Expert Advice via links to support documentation and operations notes
to resolve problems quickly
 Visualize search results with analytic tools to rapidly determine root cause
 Out-of-the-box analysis and insights for z/OS, WebSphere, DB2, CICS, IMS, MQ,
Network, Security as well as distributed systems
 Enable early error detection and broaden
scope of automation with event notifications
 Fully customizable to meet your needs
SEARCH
ANALYZE Launch to Support Doc
RESOLVE
INTEGRATE
Accelerate problem isolation and identification … Reduce mean time to repair
• Network insights
• Security insights
• Event notification
• Hadoop support
• Analysis of performance metrics (SMF real time Data Provider)
• Integration with existing Service Management tooling (Automation,
Monitoring, Event and Incident Management)
• Role-based access control
• Multi-time zone support
in 2015

z Systems Software
Applications Search
Mainframe
z/OS
z/Linux
z/OSLog
Forwarder
Log
FileAgent
or
Logstash
SMF
Real-time
Data
Provider
WAS SYSOUT
z/OS Syslog
CICS MSGUSR
Annotators
CICS EYULOG
…
Operations Analytics Server
WAS SYSPRINT
WAS SYSOUT
DB2
DB2 App
Syslog
Web Access Log
 The IBM Operations Analytics server is installed on z System
(or x System) running Linux (64 bit)
 z/OS Insight Packs are installed on the IBM Operations Analytics server
 z/OS Log Forwarder / SMF Data Provider installed on each z/OS LPAR
where you want to provide Search and Analysis
WAS SYSPRINT
Indexers
Current/
Archive
Tier
Hadoop
Tier
USS Log Files
Other Logs
10
Alert Actions
Index
Log
E-mail
Script
SNMP
EIF
Alerts
Insight Pack
(z/OS)
Distributed
Systems Insight
Packs
SMF Data
Generic
Receiver
Other Logs
NetView
Message
Gatherer
IBM Operations Analytics
Architecture and Flows
Joblogs
NetView Netlog
……

z Systems Software
Search specific
logs or ALL logs
Timeframe
Enter search string
Save My Search
Simple search interface EASY to customize
11
Quick Searches, Analysis,
Annotations, Patterns, Expert
Advice, Dashboards will
populate the Navigation tree
Log data is analysed and insights are surfaced as you search
Find problems you didn’t know existed

z Systems Software
Easy to use – Quick Search
Domain-specific ‘Quick Searches’
available out-of-the-box
or create and save your own
 Provided with every z/OS Insight Pack
 Provided by subject matter experts,
support teams and customers
 Immediate value out of the box
 Easy to modify or create and save your
own
12

z Systems Software
Dashboards, Information Links and Expert Advice
Visualize the data with Dashboards
Quick links to additional information and
support documents.
 Provided with every Insight Pack
 Expert Advice to access white papers,
tech notes, APARs, etc. for faster
problem resolution
 Dashboard views created by subject
matter experts, support teams and
customers
 Immediate value out of the box
 Easy to modify or create and save your
own
13

z Systems Software
Quickly and easily access IBM Support Portal based
Expert Advice from Log Analysis
Launch to Tech Note
All IBM support site documents that
reference messages from search results
Search for expert advice with the click of a button
14

z Systems Software
Analyze logs as you Search
Insights are surfaced automatically
as you search. Patterns are
surfaced based on the log type.
 Provided with every Insight Pack
 Logs are analysed automatically
 Log data is categorized by hostname,
data source, message type, message
source, etc.
 Patterns/Insights are surfaced to help
you focus on the source of the
problem.
For example, log analysis automatically
surfaces java exceptions in application
logs.
 Perform searches and analyse multiple
logs, organized per the needs of your
enterprise.
 Create your own Insight Pack for any
text logs with time stamps
15

z Systems Software
Sample dashboard
View your log and metric data however you like
16

z Systems Software
Out-of-the-box dashboards (Example – Display message counts and java exceptions)
OR
Build Your Own Dashboard with the click of the mouse
Sample dashboard
View your log and metric data however you like

z Systems SoftwareVisualizing the Data
Search and Analyze SMF Data (New in 2015)
Analyze your SMF data AND your log data for a
complete view of the enterprise.
18
CPU utilization, Working Set Size,
Paging & IO Rates

z Systems Software
Perform simple free-form searches using the standard set of search keywords and operators
Build complex queries with range searches and DateMath functions
To learn more, consult Online Help available from the Learn More → Search Bar → Search
query syntax menu:
BYOD – Bring your own Data – The z/OS Log Forwarder can be configured to forward your
text logs to enable Search, Analysis, Dashboards and Expert advice.
BYOIP – Build your own Insight Pack
BYOV – Build your own Views (Graphs, Charts and Dashboards)
Out-of-the-Box capabilities provide immediate value.
Additionally, IOA can easily be tailored to your specific needs.
Create your own – Queries, Dashboards, Feeds
19

z Systems Software
Customer Experiences
Large Insurance Company (29 hours down to 29 minutes)
Experienced an application outage that resulted in the team working around the clock for 29 hours.
Multiple customers and IBM support staff poured through logs and traces to determine the root
cause of the issue. After the issue was resolved, the logs were captured and sent to IBM lab for
analysis using IBM Operations Analytics for z Systems. Within minutes, the IBM team was able to
focus in on the root cause of the problem and to find the relevant PTF to resolve the issue through
the integrated expert advice.
State Agency (up and running in 2.5 hours)
Were able to download, install, configure and use IBM Operations Analytics for z Systems to
search their logs in 2.5 hours.
Numerous Customers (improve visibility and find problems you weren’t aware of)
Errors lurking in logs that are never examined because they don’t necessarily cause SLA or
performance problems. For example, IBM Operations Analytics for z Systems found
Over 4,000 invalid login attempts in a three day period that had otherwise gone unnoticed.
MQ channel errors causing MQ errors in logs from distributed systems – not being monitored
SQL errors in multiple logs
20

z Systems Software
New capabilities in 4Q, 2015
21
General capabilities (delivered via IBM Operations Analytics – Log Analysis and
included with IOAz)
Additional real-time alerting actions: SNMP Traps, EIF Events
Role-based access control
Support for multiple time zones and time intervals
Service Desk Extension: Incident and service request analytics
z/OS capabilities (included in the z/OS Insight Pack)
Additional CICS insights from SMF 110 and EYULOG
Additional network insights from NetView netlog
Security insights
Pattern-based configuration for z/OS Log Forwarder job log data gatherer
Additional out-of-the-box searches for DB2 and MQ
Translation of z/OS Insight Packs (English + 10 languages) and documentation

z Systems Software
Alerting actions: SNMP Traps, EIF Events
22
 Benefit:
– Utilize your existing event management tooling to track, highlight, enrich,
correlate and act upon conditions that are identified in their operational data
by IBM Operations Analytics for z Systems through the use of SNMP Traps,
Informs or EIF events.
– Broaden your scope of automation. Use NetView or other automation tools
to take automatic action on any messages or other operational data as long
as that data is consumed by IBM Operations Analytics for z Systems. This
expands your current automation capabilities to automate on ANY data
source that is fed into IBM Operations Analytics.
IOA now enables you to generate SNMP Traps and EIF Events.
This is in addition to existing notifications (text, email, etc.)

z Systems Software
23
Role-based access control and audit
 Benefit:
– Role-based access control and auditing capabilities enable customers to
maintain compliance with their data segregation and access control
requirements. It is of special interest for service provider environments in
which segregation of data is of particular importance.

z Systems Software
Support for multiple time zones and time intervals
24
 Benefit:
– All users connected to a single IOA Log Analytics server, regardless of their
location, are able to view search results and graphs in their local time zone or
in a different time zone of their choice. This new capability is particularly
helpful for teams that are distributed across multiple time zones.
– Applications can now specify more than a single occurrence of a relative time
interval. Instead of specifying “Last Day”, applications can specify “Last 3
Days” for example.

z Systems Software
Integration with Service Management Solutions
25
IOAz integrates with Monitoring, Automation and Event Management
Automation
 NetView / SA (or other Automation tooling)
– Receive and enrich, action or forward Events from ANY log source (not just Syslog)
Event Management
 Netcool Operations Insights (NOI)
– Launch to IOAz to analyze logs and metrics (IOA is included with NOI)
– Search and analyze Events
– Receive, correlate, enrich and action Events from IOAz (NOI or other Event Management
System)
Incident Management
 IBM Service Desk (or other incident management / trouble ticketing solutions)
– Generate Events to create Trouble Tickets
– Analyze Trouble Tickets
Monitoring
 OMEGAMON
– Launch in context to IBM Operations Analytics from OMEGAMON and ITM workspaces
– OMEGAMON Insight Pack to analyze ITM logs (RKLVLOG)
 Service Management Unite (included with Performance Management and Service Management
Suites)
– Launch in context to analyze logs and SMF data in context of performance problem diagnosis

z Systems Software
Using IOAz
to broaden the scope of
Event Management and Automation
26

z Systems Software
Event processing
 Generate Events from ANY log message(s) or other data in IOA
 Notifications can be in the form of:
– Text message
– Email
– SNMP Trap
– EIF Event
Enhance your Visibility & Automation Capabilities
27
Increase scope of log monitoring
and automation
Improve event correlation
IOAz can generate notifications for messages
from any log in your enterprise.
Be Proactive!

z Systems Software
28
Getting the most out of IOA notification capabilities
IBM Operations Analytics provides the ability to generate events based on
messages, combination of messages over time, number of occurrences, etc.
Notifications can be generated from any data source:
– Messages from Mainframe and Distributed Logs
– SMF data
– Events
– Other
Examples include:
– Send an email or text message whenever a specific message(s) is written
to a log … For example, message IRRB069I (RACF is being shut down)
– Generate a SNMP Trap or EIF event when there are more then 500 failed
logon attempts in a 30 minute period

z Systems Software
29
Notifications
Index alert action (ie send events back into IOA so they can be searched)
You can use the index alert action template to index any triggered alerts.
Email / Text alert action
You can use the email template to send an email when a condition is met. Emails can
easily be sent as text messages by most carriers
EIF alert action
You can use the EIF template to send an EIF formatted event when a condition is met.
SNMP Trap alert action
You can use the SNMP Trap template to send an SNMP Trap when a condition is met.
Script alert action
You can use the Script template to execute a custom script when a condition is met.
Write to Log alert action
You can use the Write to Log template to write an entry to a log file of your choice
when a condition is met.
Send notifications in many forms …

z Systems Software
Sending Events to any Event Receiver
Data Source 2
Alert Actions
IOA Server
Ingestion
Pipeline
Alert
Runtime
Index
Log
E-mail
Script
SNMP
EIF
Event
Management
System
(NOI, OMNIbus
or other Event
Management tool)
Event Receiver
Automation
(NetView/SA
or other
automation tool)
Data Source N
Data Source 1
30
IOA can generate standard SNMP
Traps and/or EIF Events that can be
received and processed by ANY
Event Receiver.
Event Receiver
Any Event
Processor
Event Receiver

z Systems Software
31
Event Configuration is Simple
From IOAz
Specify the message or messages to trigger the Notification
Choose the event criteria (message IDs, number of occurrences, time period, etc.)
Specify address of Event Receiver hostname/port or email address
From your automation tool
Create an automation statement(s) to:
– Enrich the Event
– Forward the Event
– Automate to correct the problem
– Other
From your Event Management tool
 Enrich the Event
 Correlate with other Events and Log Messages
 Automate to correct the problem
 Create Trouble Tickets
 Other
Increase the scope of automation to
include ANY log message
Most z/OS automation tools are limited to
z/OS Syslog and Console messages
Correlate z/OS Events with Events from
distributed systems to resolve problems
end-to-end

z Systems Software
Event driven automation scenarios
Optimized Performance
32
There are many scenarios where events can drive automation. Prior to IOAz, these
scenarios were limited to events being driven from Syslog, because most z/OS
automation tools only monitor the z/OS Syslog.
Since IOAz has access to many more logs than Syslog, we now have the ability to
drive automation from messages coming from other logs and even other platforms.
We have included just a few examples in the subsequent slides. The possibilities are
endless.
Benefit:
• The subject matter expert can now access messages from ANY log in the enterprise
• Events coming from IOA can be consumed by ANY Event receiver to automate, enrich,
correlate or forward Events or generate trouble tickets
• Events can be generated in SNMP or EIF format. As a result, the events can be consumed
by any Event Receiver (Event Management or Automation tool).
• Since IBM Operation Analytics for z systems can generate events from ANY message it
consumes and NetView can act as an event receiver, NetView can now automate on ANY
log message (not just messages from Syslog). This scenario will work with any automation
tool that can drive automation from events.
• This feature will enable customers the ability to ‘TAKE ACTION’ on any messages being
consumed by IBM Operation Analytics for z Systems.

z Systems Software
33
Sample scenario for MQ
WebSphere MQ channel stopped abnormally
– MQ server runs on Windows with a MQ channel defined to MQ running on z/OS.
– MQ server detects that the MQ channel to z/OS is not active and writes error
messages to the Windows MQ AMQError log. Subsequent MQ communications fail.
Without IBM Operations Analytics for z Systems:
The ‘Channel down’ message is never proactively observed and the support team(s)
struggle for hours to debug the problem and finally re-initiate the Channel.
With IBM Operations Analytics for z Systems:
IBM Operations Analytics for z Systems detects the problem through MQ error messages
written to the Windows MQ AMQError log.
IBM Operations Analytics for z Systems generates an SNMP Trap (or EIF event) and
forwards it to NetView (or other automation solution).
Automation is driven from this event and resolves the problem by issuing a command to
restart the MQ channel.
Outage avoided with IOAz!

z Systems Software
34
Sample scenario for DB2 DDF
DB2 DDF applications timed out
– DB2 runs on z/OS; IBM Operations Analytics for z Systems collects DB2MSTR address
space log.
– The customer applies bulk maintenance for z/OS and DB2 over the weekend. After
application of maintenance, DB2 DDF applications experience time-outs.
Without IBM Operations Analytics for z Systems:
Because maintenance occurs on a Saturday, operators do not catch the resulting problem
until later. The DBA is notified on Saturday evening, a PMR is opened against IBM DB2,
and diagnostics are started with the DB2 and TCP/IP L2 teams. By Monday morning, none
of the agents can run transactions. DB2 and z/OS maintenance have to be backed out.
IBM Operation Analytics is able to detect the time-out problem immediately after the
maintenance is applied. Operators are notified immediately and are able to determine the
root cause of the issue. End users do not experience downtime when they come into work
on Monday morning.
Issue reported immediately with IOAz. Maintenance backed out.
Problem avoided!

z Systems Software
Search and analyze events, logs and metrics using IOA and Netcool
Operations Insight.
 Easily identify ‘related’ Events that may be candidates for suppression
 Identify “difficult to spot” seasonal events that often result in regular periodic
problems
 Easily identify which events occur in clusters
 Leverage visualizations that help you quickly isolate more sever and significant
problems.
Also provides opportunities
for event reduction thus
improving operational
efficiency.
Log Analysis and Event Management in Netcool Operations Insight
(IOA is included in the box with NOI)
35

z Systems Software
Log Analysis – Streamline Incident Management
Incident Management
The traditional incident management process usually begins with one or more trouble tickets being opened for
an incident (for example, slow response time for a specific application).
The first step is to engage the application support team and associated Subject Matter Experts for each
of the application components (WebSphere, CICS, DB2, etc.). Each SME examine data from their specific
subsystem and we usually experience a phenomenon commonly referred to as ‘ticket hopping’. During the
ticket hopping phase, the trouble ticket will be reassigned multiple times before it lands on the correct
SME’s lap. Over the lifetime of the incident, there is very little collaboration with respect to data and there’s
usually a fair amount of ‘finger pointing’.
In the post mortem session, we usually conclude that the ‘time to resolution’ is very high and so is the
number of people involved in the process of diagnosing the problem.
IBM Operation Analytics will provide a unified view of the data, enabling the application support team to
quickly focus on the problem component. The ability to search and analyze the data helps to quickly
identify the problem area and the expert advice feature assists in finding the solution or workaround.
If an SME is needed for a specific component, you can transfer the ticket to that SME with the data that was
surfaced by IOAz. Post mortem reveals that time to resolution is significantly decreased by as much as
50% with less involvement by the SME community.
To be more proactive and improve mean time to recovery even more, the team can incorporate the use of
IOA notifications to immediately notify (Text, Email, SNMP Trap or EIF Event) that a problem is occurring.
Early detection will significantly decrease time to resolution and automation can be triggered to resolve
the issue before the problem affects the end user.

z Systems Software
Integration with Performance Monitoring
The One Two – Punch: Combine two very powerful tools to ensure performance and
high availability of your enterprise.
Perform log analysis in context of OMEGAMON workspaces – This approach
enables OMEGAMON users to perform in-context log analysis while doing problem
determination
– From your OMEGAMON workspace, use the IOA search bar to search logs (using
LPAR or Sysplex as the default context)
– Easy to implement - Configure TEP to display the IOA search bar
Launch IOA from OMEGAMON
performance monitoring
workspaces to search logs in
context
OMEGAMON + IBM Operations Analytics – Launch in Context from TEP
37

z Systems Software
Search and Analyze Operational Data in Context
38
Select a row first. In this example, a row
specifies a Queue Manager.
Select a row first. In this example, a row
specifies a Queue Manager.
Specify a search string and timeframe
to analyze operational data from the
appropriate system(s)
Specify a search string and timeframe
to analyze operational data from the
appropriate system(s)

z Systems Software
Analysis of Operational Data
39
 IBM Operations Analytics analyzes log,
metric and event data and surfaces
insights
 Built on industry expertise
 Expert Advice for faster time to resolution
 Expand analysis to include additional
data sources (from mainframe and
distributed systems)
Launch into IBM Operations Analytics to
analyze logs and other operational data
to gain additional perspective and insights
and help diagnose root cause.
Launch into IBM Operations Analytics to
analyze logs and other operational data
to gain additional perspective and insights
and help diagnose root cause.

z Systems Software
Integration with existing Service Management solutions
(in a nutshell)
40
• Search and analyze logs, metrics,
events and incident reports
• Launch from ITM, OMEGAMON,
Service Management Unite & NOI
Surface
anomalies
POWerful tools integrate to
ensure performance and high
availability of your enterprise.
IBM zAware
Service Management Unite
Automation &
Problem
Determination
NetView
Performance
Monitoring
ITM/OMEGAMON
Event Management
OMNIbus/NOI
Incident
Management
Control Desk
Alert,
enrich,
correlate
and
automate
IBM Operations Analytics

z Systems Software
Send us your logs!
 Request a product demo using logs from your own test, development
or production environments
– IBM will load your logs into an IBM Operations Analytics server, then demo
the results back to you
• A secure, dedicated drop box will be assigned to you
• You will be sent detail upload instructions via email
• Any file uploaded will be automatically moved to a dedicated IBM
Operations Analytics environment within 24 hours
• All log data will be purged from the IBM Operations Analytics
environment within 48 hours after the demo event
To request your hosted demo, visit:
http://services-useast.skytap.com:18280/WebDemo/
41
A guided demo is provided online at:
http://ibm.biz/ioazlivedemo
Or Take IOAz for a Test Drive

z Systems Software
Announcing the IBM Operations Analytics for z Systems Early Access and Beta Program!
In 2015, we built on the strong foundation established over recent months as we develop and implement
our product roadmap.
We are looking for customers and business partners worldwide who would like to help influence our
roadmap and test new capabilities. The program is open-ended; interested participants may join at any time
and stay on as long as they wish. That said, it is our desire to establish a set of “customer sponsor”
relationships that will become instrumental in shaping the future of our offering.
To see the full program announcement, and to learn how to sign up, please visit us in our developerWorks
community at:
https://ibm.biz/BdEkZV
IOA for z Systems Early Access and Beta Program
42

z Systems Software
Additional IBM Operations Analytics Reference Material
 Analytics Overview Video
– https://www.youtube.com/watch?v=OQJapWiQECs
 IOA for z Systems videos:
– Overview: https://goo.gl/XsmGtq
– Domain Insights: https://goo.gl/ktXYpL
– Installation and Configuration: https://goo.gl/3QLm2N
 IOA for z Systems Documentation
– Knowledge Center:
• http://www.ibm.com/support/knowledgecenter/SS55JD
 IOA – Log Analysis (server) Documentation
– Service Management Connect
• http://www.ibm.com/developerworks/servicemanagement/ioa/log/index.html
– Knowledge Center
• http://www.ibm.com/support/knowledgecenter/SSPFMY
43

z Systems Software
44

z Systems Software
More Detail and drill down to the next level
The remainder of the deck includes details for the following:
Bring your own Data
Feeding additional logs into IOA
Options
Example using the Hardware Management Console log
IOAz 2.2
Enhanced CICS insights
Enhanced Network insights
Security insights
Log Forwarder improvements
45

z Systems Software
Bring Your Own Data
46

z Systems Software
In addition to the out-of-the-box Insights for z/OS, DB2, IMS, CICS, MQ, Network
and Security, IOAz can also be customized to meet the needs of your enterprise.
BYOD – Bring your own data
– Enable Search and surface Insights for any text log (messages must have
time stamps)
– Create your own Saved Searches
– Create your own dashboards, graphs and charts
– Gain access to any log data in the enterprise to debug end-to-end
applications, generate notifications and surface events
Enhance your Visibility – Avoid ‘Blind Spots’
47
Increase scope of log monitoring
and automation
Remove ‘blind spots’

z Systems Software
 IOAz provides a number of defined data sources to help you ingest
your data (out-of-the-box Insight Pack)
– z/OS – Syslog (including CICS, IMS, DB2, MQ, Security, Network, etc.), CICS
MSGUSR and EYULOG, WebSphere sysout and sysprint, USS syslogd, SMF,
NetView
– Distributed Systems – MQ, DB2, Javacore, WebSphere, Service Desk, Microsoft
SQL, Active Directory, and many more. See the following URL for the latest list of
Insight Packs available for IOA-LA: https://developer.ibm.com/itoa/resources/
 But what if you want to add your own custom data types ? (BYOD)
• Custom Application Logs
• Generated Report Files
• Statistical Records
• Others ?
• Job Log data
48

z Systems Software
Example: HMC Log as a data source
Customers often ask us if IOAz can consume logs from applications that they
have written or even just some other log in the enterprise that is not currently
supported by an existing Insight Pack.
The answer to this question is ‘YES!’. As long as it is a text log and the messages
have a time stamp, they can be consumed by the IOA server.
Anyone can write an Insight Pack … And there are several options, depending on
what you want to do with the data … In all cases, it is Quick and Easy.
We have received several requests to provide an Insight Pack for the HMC.
Retrieve the HMC log
We are using the HMC Log Tool (HLT) from the TechDocs library
– http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS5300
– This tool allows you to retrieve log information using the zEnterprise Web
Services APIs and generates reports on the desired information.
– Data is written to a file on USS (IOAz can consume USS files)
Feed the log data into IOAz for analysis and audit purposes.
– Multiple options here depending on what you want to do with the data. All are
simple.
49

z Systems Software
What do you want to do with the data?
 Search with Default Annotations (takes 10 minutes to configure)
– Configure the z/OS Log Forwarder (data source type = ‘other’) to send the log data to IOAz.
The IOAz generic receiver will index and annotate the data to make it available for Search.
– You can create and save quick searches, graphs and dashboards.
 Annotate and analyze a CSV style log file (takes 20 minutes)
– The DSV toolkit can be used to process any log file that follows a “Delimiter Separated
Value” format. Simply edit a text file describing the DSV layout.
– Each separated value column becomes an annotated field in IOA
 Annotate and analyze any text or log file (1-2 hours)
– Allows for complete control on how files are read into IOA and parsed. This approach
allows for more sophisticated pattern searching and correlation.
– Can be written in Java, AQL, Python, etc.
50
IOAz can consume and analyze any text log. There are 3 options to enable
this depending on what you want to do with the log data

z Systems Software
Example: HMC Insight Pack
02/04/2015 14:54:13.300: 2007: User SooAcsadmin has acknowledged viewing l
02/04/2015 15:31:59.860: 1100: The system clock has changed.
02/05/2015 13:57:42.790: 734: Remote support call generated on USYS is bei
02/05/2015 13:57:45.900: 674: Remote support call generated on USYS comple
02/12/2015 14:12:20.230: 722: An upgrade to EC level N98841 was performed.
Above is an excerpt from an HMC log. The log contains time stamps, event
IDs and text.
In the following slides, we will show the difference between generic
processing (Search with Default Annotations) and processing using the HMC
Insight pack.
The HMC API was used to retrieve the log data and the logs were stored in
the USS file system.
The z/OS Log Forwarder was configured to send the log to IOAz.
51

z Systems Software
HMC Insight Pack – Search (using Generic Receiver)
52
Generic receiver discovered
patterns in the logs.
Search capability generates
helpful search results.
Graphical view of
message volumes.
It take approximately 10 minutes to configure
the log forwarder to send a log to IOA.
The generic receiver enables search and
analysis, graphical views, save searches,
expert advice, etc.
But in this case, it discovers too many generic
patterns. We decided to write a quick insight
pack to surface better insights.

z Systems Software
Creating a Java based Insight Pack
 Eclipse plugin available to assist in creating
insight packs.
 The plugin generates the required insight pack
structure and supporting files.
 All that is required is to write 2 pieces of java
code and define the record structure of the
resulting lines for IOA
– Splitter.java – Receives blocks of log data and breaks it up into distinct lines of
data.
– Annotator.java – Receives each line of data and identifies the individual fields to
be annotated for IOA.
 Finally define what the name of the data source will
be and map the splitter and annotator code using
the supplied plugin.
 Run an ant build using the provided ant build sample
53
It takes an approximately 1-2
hours to write an Insight
Pack.
Time is significantly reduced
after writing your first one.

z Systems Software
Example: HMC Insight Pack in Java
02/04/2015 14:54:13.300: 2007: User SooAcsadmin has acknowledged viewing l
02/12/2015 14:12:20.230: 722: An upgrade to EC level N98841 was performed.
54
The 734 is the Event ID
which would be useful to
annotate on
The 734 is the Event ID
which would be useful to
annotate on
Annotated fields show
up in the Search
Patterns
Annotated fields show
up in the Search
Patterns
Fields are assigned to a
key-value pair and can
show up in the grid view
Fields are assigned to a
key-value pair and can
show up in the grid view
An Insight Pack can surface important
messages, text strings, text patterns, etc.
This is just a simple example.

z Systems Software
Create custom quick searches
Once the data is in IOA, it’s easy to create quick searches by
simply running a search and saving it.
55
Save button lets you add
any search to the saved
search menu
Save button lets you add
any search to the saved
search menu

z Systems Software
Creating custom graphs – Remote Support Calls
To create a graph of data, simply run a query, select
column(s) to graph and then click the graph button and save it
56
The graph button allows
you to graph 1 or more
data columns.
The graph button allows
you to graph 1 or more
data columns.

z Systems Software
Creating custom dashboards
Multiple graphs can be saved to single dashboard
57
Distribution of Event IDs over time
Critical data backups
Login failures
Remote Support Calls

z Systems Software
Reference Material – Building Insight Packs
 IBM Operations Analytics – Log Analysis resources web site
– https://developer.ibm.com/itoa/resources/
 An example of creating a Java based insight pack
– https://developer.ibm.com/itoa/docs/log-analysis/insight-pack-creating
58

z Systems Software
IOAz V2.2
More detail
59

z Systems Software
Key enhancement: Enhanced CICS insights
60

z Systems Software
Enhanced CICS Insights
61
IOAz consumes, analyses and provides insights using the following CICS data:
CICS message data gathered from the following logs
– SYSLOG
– MSGUSR
– EYULOG
SMF 30 – CICS Jobs
– CPU utilization
– IO Rate
– Paging Rate
– Working Set
CICS SMF 110 data
– CICS Monitoring Exceptions
– CICS Global Transaction statistics (per APPLID)
In IOAz V2.2, the subject matter expert and/or application developer can
gain insight and quickly isolate and diagnose CICS problems from a
combination of CICS performance metrics and log data.
CICS Monitoring Exceptions and Global Transaction Statistics are now
available to aid in diagnosis.
IOA notifications enable proactive monitoring for early detection and
automation.

z Systems Software
Enhanced CICS Insights
62
New Quick Searches for CICS
IOAz V2.2 includes the following ‘Quick Searches’ (in addition to the current set of predefined
searches for SMF30)
Display all CICS Exceptions
Display CICS Wait on Storage Exceptions
Display CICS Policy Exception
Display CICS Transaction Interval Summary
Display CICS End of Day Summary- last week
Display CICS Transactions summaries where Tasks = MAXTASKS.
New CICS Dashboards
CICS Regions Dashboard
– Interactive dashboard that integrates SMF and log data to diagnose:
• Wait on Storage events per region over last day
• Exceptions by Resource ID over last day
• Short on Storage per region over last day
• Tasks at Maximum Threshold over last day
• Storage Violations per region over last day
CICS Regions Transaction Dashboard
– Static dashboard showing:
• Transactions – top 5 regions over last day
• Transactions – Max and Average per region over last day

z Systems Software
CICS Demo Scenario
63
Eric the application developer is investigating an issue with his application running
slowly.
Prior to IOAz, Eric would have had to browse through each log looking for
information on the state of CICS in general and his application specifically. This
might involve the SYSLOG for the system and MSGUSR and EYULOG for each
CICS region.
If his application spanned multiple systems, he would have to log on to each
system first and again manually browse each SYSYLOG and MSGUSR and
EYULOG for each CICS region.
He would be required to log on to yet a separate application to view SMF data.
Once he has identified all the data sources, he will then need to manually correlate
the data across the multiple sources to diagnose the problem.

z Systems Software
CICS Demo Scenario, continued
64
With IOAz, Eric can logon to a single user interface and access all of the relevant logs in
the enterprise and also reference SMF data to determine if an associated performance
problem or CICS monitoring exception has occurred.
First he checks the CICS Jobs dash board and does see a spike in CPU, I/O Rate and
Working Set size.
He looks at the Transaction dashboards and sees a corresponding spike in the
transactions.
He then looks at the Regions dashboard and does see some exceptions related to
storage issues.
Double clicking on the exceptions (insights automatically surfaced on the left side of the
user interface) he can see most are related to the progname ICC$HEL
Then he checks the integrated IOA log search. This allows him to quickly search the
SYSLOG, MSGUSR, EYULOG and SMF. In here he can search specifically for his
application name to see where problems might lie, in this case ICC$HEL
To be more proactive, Eric can configure a notification when a message, combination of
messages or additional data indicates a problem. Notifications can be in the form of a
text message, email, SNMP Trap or EIF Event and can be used to automatically resolve
an issue, gather additional data or generate a trouble ticket.

z Systems Software
Key enhancement: Enhanced network insights
65

z Systems Software
Enhanced Network Insights
66
Network insights were first introduced in IOAz V2.1 to enable the search
and analysis of z/OS network data provided by VTAM, TCPIP and syslogd
message logs.
In IOAz V2.2, you can now ingest, search and analyze NetView message
data; specifically the NetView messages that are written to the NetView
netlog.
Customers can quickly and easily identify issues that are surfaced only
through NetView netlog without actually going to NetView or searching the
NetView Canzlog. In addition to network messages, the NetView Netlog
contains other messages; for example, messages related to NetView
automation failures.
Since IOAz can consume logs from multiple systems, you can perform a
single search across all logs to determine if a problem is occurring on more
than one system.
IOA notifications enable proactive monitoring for early detection of
network or automation-related issues.

z Systems Software
Enhanced Network Insights
67
New Quick Searches for NetView Netlog
IOAz V2.2 includes the following ‘Quick Searches’ for NetView Netlog (in addition to the current set of
predefined searches for Network insights provided in IOAz V2.1)
All NetView for z/OS Messages
All NetView for z/OS Action, Decision or Error messages
NetView for z/OS messages that indicate automation table violations
NetView for z/OS messages that indicate command authorization table violations
NetView for z/OS messages that indicate resource limits or storage thresholds
NetView for z/OS messages that indicate insufficient access authority or security environment violations
New NetView Netlog Dashboards
NetView Message Counts - Top 5 per hour over Last Day
NetView Message Type Counts - Top 5 per hour over Last Day
Total NetView Message Counts per hour over Last Day
NetView Messages by Hostname - Top 5 per hour over Last Day
NetView Message Types by Hostname - Top 5 over Last Day
Total NetView Messages by Hostname per hour over Last Day

z Systems Software
Insights from NetView netlog
68
User Scenario 1
Alice as the Subject Matter Novice is notified of a problem with the NetView automation table. A new
automation statement has been added to the automation table to perform an automation action on an
autotask when a specific message is processed. Even though the message is processed, the automation
action did not occur.
Instead of signing on to NetView, Alice uses IBM Operations Analytics for z Systems and issues a Saved
Search for NetView Automation to look for possible errors.
The search results find a NetView message: DWO032E AUTOMATION ACTION action COULD NOT BE
ROUTED TO TASK(S) task.
Alice determines from this message that the problem is not with the actual automation table statement but
instead is with the autotask where the automation action is to be performed. Next she can search for the
specific autotask in the NetView messages to ensure the autotask is configured properly and started.
Alice can search and analyze ALL of the NetView logs at one time by issuing a single search command
from IOAz.
Alice can create automation to check for these messages in the future and then issue the command to
restart the autotask if it is not already started.

z Systems Software
69
User Scenario 2
Jim as the Subject Matter Expert needs to determine if there were any attempts by NetView operators to
issue unauthorized commands over the last week. Even though a NetView operator is allowed to sign on to
NetView, an operator can be assigned access to a specific set of commands.
Instead of going to each NetView domain and looking for related security messages in the NetView
Canzlog, Jim uses IBM Operations Analytics for z Systems and issues a Saved Search for NetView
Security to look for unauthorized command attempts.
Jim can search and analyze ALL of the NetView logs at one time by issuing a single search command
from IOAz.
The search results show Jim that there were multiple attempts to issue unauthorized commands on over
the last week.
– BNH232E 'userid' IS NOT AUTHORIZED TO ISSUE COMMAND 'command’
– BNH233E THE COMMAND 'command' IS PROTECTED BY COMMAND IDENTIFIER
'commandid' IN 'auth_method’
– DSI213I ACCESS TO 'object' IS NOT AUTHORIZED

z Systems Software
70
Search resultsSearch results
No typing necessary. Just click
on one of the IBM-provided
quick searches or a saved
search
No typing necessary. Just click
on one of the IBM-provided
quick searches or a saved
search
Analysis reveals patterns
which can be combined to
build more complex searches;
simply by selecting them.
Analysis reveals patterns
which can be combined to
build more complex searches;
simply by selecting them.
Search logs from
multiple systems or
choose to see results
from specific systems
Search logs from
multiple systems or
choose to see results
from specific systems

z Systems Software
Key enhancement: Security insights
71

z Systems Software
Security Insights
72
IOAz V2.2 introduces security insights so the subject matter expert,
security administrator and/or application developer can quickly identify
RACF failures and security issues. RACF failures often contribute to
failed applications (for example, failure in read/write operations due to
missing or incorrect RACF authorization).
RACF security issues such as invalid authority and a significant number of
invalid logon attempts can be a sign of a security breach.
IOA notifications enable proactive monitoring for early detection of RACF
security issues.

z Systems Software
Security Insights
73
New Quick Searches for Security-related issues
Display all RACF Messages
Display all RACF Action Error or Warning messages
Display all Insufficient Access errors
– User attempted to access something but does not have the proper access authority
Display all Insufficient Authority errors
– User attempted to execute a program but does not have the proper authority to do so
Display all Invalid Password messages
– Any variation of the ICH/IRR messages dealing with invalid passwords.
New Security Dashboards
 Security Message Counts - Top 5 per hour over Last Day
 Security Message Type Counts - Top 5 per hour over Last Day
 Total Security Message Counts per hour over Last Day
 Security Messages by Hostname - Top 5 per hour over Last Day
 Security Message Types by Hostname - Top 5 over Last Day
 Total Security Messages by Hostname per hour over Last Day

z Systems Software
Key enhancement: Pattern-based z/OS Log Forwarder
data source configuration
74

z Systems Software
Log collection configuration for complex
environments ... made easy
75
Log Forwarder configuration has been enhanced to save time, be more
dynamic and less error-prone.
The System programmer is responsible for configuration of the z/OS
Log Forwarder(s). Depending on the number of log files being sent to
the IOA server, the Log Forwarder configuration can be time-
consuming. Today, it is a manual task and it can be error-prone.
Log Forwarder configuration has been enhanced in IOA V2.2 to
support a wildcard and discovery feature to save time and
decrease the possibility of configuration errors.
Significant reduction in time to configure log collection and analysis
from large environments

z Systems Software
76
Zach the System Programmer has a large number of WebSphere Application
servers, or a large number of CICS regions, and he wants to collect data from all of
them.
Prior to IOAz, Zach would have had to configure the z/OS Log Forwarder for each
and every job name.
–This can be a time-consuming process, even when using the supplied Configuration
Assistant. If you have 50 job names to configure, and each takes just two minutes, it
will take an hour and 40 minutes to configure all 50.
–This can be error-prone given the manual nature of the task – mistyping a job name
means that job logs are not being ingested.
–If a new WebSphere Application Server or CICS region is added after the
z/OS Log Forwarder is started, the z/OS Log Forwarder must be restarted to pick up
the new job log gatherer definition.

z Systems Software
77
With IOAz, Log collection configuration is much easier. Zack can now create a single
job log gatherer definition with one or more wildcard characters in the Job Name field.
This definition serves as a template for all jobs that match the job name pattern.
A handful of definitions can now cover many job names. Instead of an hour and 40
minutes to create definitions for 50 job names, it may take a minute or two.
Valid wildcard characters are:
– * which represents any sequence of zero or more characters
– ? which represents any single character
The z/OS Log Forwarder will start a data gatherer internally for each job name on
JES spool that matches the wildcard job name value.
The z/OS Log Forwarder continues searching for job names that match the pattern
even after initialization has been completed. If a new job name appears it will attempt
to start a job log gatherer for it.

IBM Operations Analytics For z Systems V2.2 - Client Long Pres

More Related Content

What's hot (20)

Similar to IBM Operations Analytics For z Systems V2.2 - Client Long Pres (20)

More from IBM z Systems Software - IT Service Management (20)

Recently uploaded (20)

IBM Operations Analytics For z Systems V2.2 - Client Long Pres

Editor's Notes