Large scale implementation of ibm tivoli composite application manager for web sphere and response time tracking redp4162

Front cover

Large-Scale Implementation of
IBM Tivoli Composite Application
Manager for WebSphere and
Response Time Tracking
Planning for performance of
management infrastructure

Implementing with multiple
servers

Performing mass update
of agents

Budi Darmawan
Aleem Subhedar
Celena Tan
Howard Anglin
Huang Chuan
Rohit Dhall

ibm.com/redbooks Redpaper

International Technical Support Organization

Large-Scale Implementation of IBM Tivoli
Composite Application Manager for WebSphere
and Response Time Tracking

December 2007

Note: Before using this information and the product it supports, read the information in
“Notices” on page vii.

Second Edition (December 2007)

This edition applies to Version V6.0 of ITCAM for Response Time Tracking (product number
5698-A75) and Version 6.0 of ITCAM for WebSphere (product number 5698-A71).
© Copyright International Business Machines Corporation 2006, 2007. All rights reserved.
Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP
Schedule Contract with IBM Corp.

Contents

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
The team that wrote this Redpaper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Become a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Chapter 1. Overview of IBM Tivoli Composite Application Manager
implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Application management with IBM Tivoli. . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 IBM Tivoli systems management portfolio . . . . . . . . . . . . . . . . . . . . . 3
1.1.2 IBM Tivoli Composite Application Manager solution . . . . . . . . . . . . . . 5
1.2 Scope of and concerns relating to large-scale implementation . . . . . . . . . . 6
1.2.1 Defining large-scale implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.2 Concerns and considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 Overview of IBM Tivoli Composite Application Manager. . . . . . . . . . . . . . . 8
1.3.1 Understanding ITCAM for WebSphere . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3.2 Understanding IBM Tivoli Composite Application Manager
for Response Time Tracking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.4 Document organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Chapter 2. Planning for ITCAM for WebSphere . . . . . . . . . . . . . . . . . . . . . 17
2.1 Planning considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2 Product architecture of ITCAM for WebSphere . . . . . . . . . . . . . . . . . . . . . 18
2.3 Deciding on the size of the servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3.1 Sizing parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3.2 Sizing estimation for ITCAM for WebSphere managing server. . . . . 22
2.3.3 Data collector overhead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.4 Implementation options for ITCAM for WebSphere. . . . . . . . . . . . . . . . . . 25
2.4.1 Designing the managing server . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.4.2 Deploying a large number of data collectors . . . . . . . . . . . . . . . . . . . 28
2.5 Communication and security considerations . . . . . . . . . . . . . . . . . . . . . . . 29
2.5.1 Communication security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.5.2 Firewall and port consideration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.6 Reliability and high availability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.6.1 Failover and fault tolerance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.6.2 Disaster recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

© Copyright IBM Corp. 2006, 2007. All rights reserved. iii

Chapter 3. Installing ITCAM for WebSphere . . . . . . . . . . . . . . . . . . . . . . . . 33
3.1 Installing ITCAM for WebSphere managing server . . . . . . . . . . . . . . . . . . 34
3.1.1 Installation configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.1.2 Database installation and configuration . . . . . . . . . . . . . . . . . . . . . . 35
3.1.3 WebSphere Application Server considerations . . . . . . . . . . . . . . . . . 41
3.1.4 Configuring the split server of the managing server . . . . . . . . . . . . . 42
3.1.5 Installation and setup of split server . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.1.6 Verifying the installed components in a split environment . . . . . . . . 47
3.1.7 Adding additional publish servers and archive agents . . . . . . . . . . . 48
3.2 Deploying data collectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.2.1 Setting up the silent installation process . . . . . . . . . . . . . . . . . . . . . . 50
3.2.2 Installing the data collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.2.3 Installing and configuring the data collector together . . . . . . . . . . . . 54
3.2.4 Configuring data collectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.2.5 Automatically discovering the installation parameters . . . . . . . . . . . 61
3.3 Configuring and setting up SSL communication . . . . . . . . . . . . . . . . . . . . 64
3.3.1 Managing server Secure Socket Layer setup . . . . . . . . . . . . . . . . . . 64
3.3.2 Data collector Secure Socket Layer setup . . . . . . . . . . . . . . . . . . . . 65
3.3.3 Working with custom certificates . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Chapter 4. Maintenance of ITCAM for WebSphere . . . . . . . . . . . . . . . . . . . 75
4.1 Operating ITCAM for WebSphere . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.2 Performance and availability of the managing server . . . . . . . . . . . . . . . . 76
4.2.1 Performance of the WebSphere Application Server . . . . . . . . . . . . . 76
4.2.2 Database maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.2.3 Data trimming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4.3 Backup and recovery configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.3.1 ITCAM for WebSphere backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.3.2 WebSphere configuration backup . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.3.3 Database backup and restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4.4 Log files and configuration files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.4.1 Managing log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.4.2 Managing the configuration files . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.5 Performing product maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.5.1 Getting software updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.5.2 Updating ITCAM for WebSphere managing server. . . . . . . . . . . . . . 87
4.5.3 Updating ITCAM for WebSphere data collectors . . . . . . . . . . . . . . . 90

Chapter 5. Planning for ITCAM for Response Time Tracking . . . . . . . . . . 93
5.1 Planning considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
5.2 Product architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
5.3 Sizing the servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.3.1 Sizing parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

iv Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

5.3.2 Sizing estimation for ITCAM for Response Time Tracking . . . . . . . . 97
5.4 Deployment of ITCAM for Response Time Tracking . . . . . . . . . . . . . . . . . 98
5.4.1 Designing the management server . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5.4.2 Deploying the management agent . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.5 Communication and security considerations . . . . . . . . . . . . . . . . . . . . . . 100
5.5.1 Communication security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.5.2 Firewall and port considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.6 Reliability and high availability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.6.1 Failover and fault tolerance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.6.2 Disaster recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

Chapter 6. Installing ITCAM for Response Time Tracking. . . . . . . . . . . . 105
6.1 Clustering the management server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
6.1.1 Preparing the operating system . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
6.1.2 Installing the database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.1.3 Installing WebSphere Application Server . . . . . . . . . . . . . . . . . . . . 109
6.1.4 Installing WebSphere Load Balancer . . . . . . . . . . . . . . . . . . . . . . . 118
6.1.5 Installing the management server . . . . . . . . . . . . . . . . . . . . . . . . . . 124
6.1.6 Checking the configuration of the RTT cluster application . . . . . . . 128
6.2 Deploying the management resources . . . . . . . . . . . . . . . . . . . . . . . . . . 129
6.2.1 Silent installation of the management agent . . . . . . . . . . . . . . . . . . 129
6.2.2 Command-line interface for management components . . . . . . . . . 130
6.2.3 Defining management resources . . . . . . . . . . . . . . . . . . . . . . . . . . 131
6.3 Setting up Secure Sockets Layer certificates . . . . . . . . . . . . . . . . . . . . . 133
6.3.1 Secure Sockets Layer for ITCAM for Response Time Tracking . . . 133
6.3.2 Working with custom certificates . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

Chapter 7. Maintenance of ITCAM for Response Time Tracking . . . . . . 137
7.1 Operational issues pertaining to a large environment . . . . . . . . . . . . . . . 138
7.2 Performance and availability of management server . . . . . . . . . . . . . . . 138
7.2.1 Performance of WebSphere Application Server . . . . . . . . . . . . . . . 138
7.2.2 Database maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.3 ITCAM for Response Time Tracking files . . . . . . . . . . . . . . . . . . . . . . . . 140
7.3.1 Backup and recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
7.3.2 Managing the log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7.4 Performing product maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
7.4.1 Getting software updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
7.4.2 Updating ITCAM for Response Time Tracking management server142
7.4.3 Updating ITCAM for Response Time Tracking management agents143

Abbreviations and acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

Contents v

Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

vi Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area.
Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM
product, program, or service may be used. Any functionally equivalent product, program, or service that
does not infringe any IBM intellectual property right may be used instead. However, it is the user's
responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document.
The furnishing of this document does not give you any license to these patents. You can send license
inquiries, in writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions
are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES
THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT,
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer
of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may
make improvements and/or changes in the product(s) and/or the program(s) described in this publication at
any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any
manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the
materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without
incurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm
the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on
the capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to the names and addresses used by an actual business
enterprise is entirely coincidental.

COPYRIGHT LICENSE:
This information contains sample application programs in source language, which illustrates programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the
sample programs are written. These examples have not been thoroughly tested under all conditions. IBM,
therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy,
modify, and distribute these sample programs in any form without payment to IBM for the purposes of
developing, using, marketing, or distributing application programs conforming to IBM application
programming interfaces.

© Copyright IBM Corp. 2006, 2007. All rights reserved. vii

Trademarks
The following terms are trademarks of the International Business Machines Corporation in the United States,
other countries, or both:

Redbooks (logo) ® ETEWatch® OMEGAMON®
pSeries® IBM® OS/400®
z/OS® IMS™ Rational®
AIX® Lotus Notes® Redbooks®
CICS® Lotus® Tivoli®
Database 2™ Monitoring On Demand® WebSphere®
DB2 Universal Database™ MVS™ Workplace™
DB2® Notes®
ETE™ Operating System/400®

The following terms are trademarks of other companies:

Oracle, JD Edwards, PeopleSoft, Siebel, and TopLink are registered trademarks of Oracle Corporation
and/or its affiliates.

Snapshot, and the Network Appliance logo are trademarks or registered trademarks of Network Appliance,
Inc. in the U.S. and other countries.

ITIL is a registered trademark, and a registered community trademark of the Office of Government
Commerce, and is registered in the U.S. Patent and Trademark Office.

Enterprise JavaBeans, EJB, Java, JavaBeans, JDBC, JMX, JNI, JRE, JVM, J2EE, Solaris, and all
Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or
both.

Microsoft, Outlook, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United
States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

viii Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

Preface

This IBM® Redpaper discusses large-scale implementation of IBM Tivoli®
Composite Application Manager for WebSphere® and IBM Tivoli Composite
Application Manager for Response Time Tracking. Large-scale implementation is
typically characterized by the number of monitoring agents deployed and the
number of transactions load-managed. A typical large-scale implementation of a
monitoring product contains the following challenges:
Keeping up the performance of the monitoring tools to accommodate the
processing load from the agents.
Automation of installation, update, and maintenance of monitoring agents
based on silent installation and automated update.
Specific day-to-day maintenance actions to ensure performance and
availability of the monitoring solution.

This IBM Redpaper addresses these issues with regard to the implementation of
ITCAM for WebSphere and ITCAM for Response Time Tracking on distributed
platforms. The discussion is divided into planning issues, implementation guides,
and maintenance considerations.

The team that wrote this Redpaper
This Redpaper was produced by a team of specialists from around the world
working at the IBM International Technical Support Organization (ITSO), Austin
Center.

Budi Darmawan is a Consulting IT Specialist at the IBM ITSO, Austin Center.
He writes extensively and teaches IBM classes worldwide on all areas of Tivoli
systems management products. Before joining the ITSO in 1999, Budi worked as
Solution Architect and Implementer in Integrated Technical Services, IBM
Indonesia. His current interests include availability management, z/OS® systems
management, and Java™ programming.

Aleem Subhedar is a Staff Software Engineer with India Software Labs in Pune,
India. He has seven years of experience in AIX® and Middleware System
Administration. He holds a degree in Chemistry from Pune University. His areas
of expertise include AIX, pSeries®, and related system technologies. He is an
IBM Certified System Expert. His areas of interest include pSeries virtualization
and high availability.

© Copyright IBM Corp. 2006, 2007. All rights reserved. ix

Celena Tan is a Managing Consultant with IBM Software Group Services in
Australia. She has 14 years of experience in the IT field. She holds a Masters of
Technology from National University of Singapore and a Bachelor of Electrical
Engineering (Hons) from the University of Tasmania. Her areas of expertise
include ITCAM family products and rational testing, and change and
configuration management products.

Howard Anglin is a Deployment Expert for ITCAM for WebSphere, Response
Time Tracking, IBM Tivoli Monitoring in the United States. He has worked with
various large customers, and in his role as an IT Specialist he has resolved
deployment, integration, and performance issues. He has nine years of
experience in the software test and development field with emphasis on the
WebSphere Application Server. He holds a Bachelor of Science in Electrical
Engineering from Manhattan College, Riverdale, New York. Howard began his
career at IBM in the pSeries Hardware Group as a Test Engineer developing
automation solutions for the production line. He then transferred to the software
group.

Huang Chuan is a Senior Test Lead of IBM China CSDL lab. He has five years
of experience in software developing and over six years of experience in
software product testing. He has led the ITCAM for Response Time Tracking test
project for several releases. He holds a degree in Computer Science from the
University of Electronic Science and Technology of China.

Rohit Dhall is an IT Architect with GBS, IBM India. He has 10 years of IT
experience in technologies like client-server computing, Web-based
transactional systems, data warehousing, and data mining. His major expertise is
in designing, implementing, and tuning large-scale Internet banking, eMortgage,
and anti-money laundering solutions for the banking and financial sector. He is
EXIN ITIL® certified and also holds certification in Java and EJB™ from
Brainbench. His current interests include SOA and IBM Virtualization offerings.

Thanks to the following people for their contributions to this project:

Donna Martin, Noel Lewis, Tony Williams, Marco De Gregorio, Sushanto Pandit
IBM Software Group, Tivoli Software

John Horton
Author of the first edition of Large-Scale Implementation of IBM Tivoli Composite
Application Manager for WebSphere and Response Time TrackingLarge-Scale
Implementation of IBM Tivoli Composite Application Manager, REDP-4162

Julie Czubik
International Technical Support Organization, Poughkeepsie Center

x Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

Become a published author
Join us for a two-week to six-week residency program! Help write an IBM
Redbook dealing with specific products or solutions, while getting hands-on
experience with leading-edge technologies. You will team with IBM technical
professionals, Business Partners, or customers.

Your efforts will help increase product acceptance and customer satisfaction. As
a bonus, you will develop a network of contacts in IBM development labs, and
increase your productivity and marketability.

Find out more about the residency program, browse the residency index, and
apply online at:
ibm.com/redbooks/residencies.html

Comments welcome
Your comments are important to us!

We want our papers to be as helpful as possible. Send us your comments about
this Redpaper or other Redbooks® in one of the following ways:
Use the online Contact us review book form found at:
ibm.com/redbooks
Send your comments in an e-mail to:
redbook@us.ibm.com
Mail your comments to:
IBM Corporation, International Technical Support Organization
Dept. HYTD Mail Station P099
2455 South Road
Poughkeepsie, NY 12601-5400

Preface xi

xii Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

1

Chapter 1. Overview of IBM Tivoli
Composite Application
Manager implementation
This chapter provides an overview of the large-scale implementation issues for
IBM Tivoli Composite Application Manager. This chapter covers the following
topics:
1.1, “Application management with IBM Tivoli” on page 2
1.2, “Scope of and concerns relating to large-scale implementation” on
page 6
1.3, “Overview of IBM Tivoli Composite Application Manager” on page 8
1.4, “Document organization” on page 15

© Copyright IBM Corp. 2006, 2007. All rights reserved. 1

1.1 Application management with IBM Tivoli
Computer-based applications are the lifeblood of modern enterprises. Most
business processes are driven by the so-called computer application that
promotes productivity, automates processing, and minimizes human errors.
These applications enable business persons to focus on what must be done,
instead of how to do it. However, as business processes rely more on these
applications, the applications become critical to the business. The applications
must be available for the execution of the business processes.

Most applications evolved from centralized applications typically managed by the
information technology (IT) department or mainframe-based applications, where
all the application layers are maintained from the central mainframe. Today,
applications tend to have multiple layers, often distributed across different
servers, different platforms, and different components. These applications are
called composite applications. This complicates the management of applications
on matters such as operational settings, problem determination, and
performance management.

Applications as a business-critical entity must be available with adequate
response time for users to perform their tasks. With application components
spread throughout the enterprise, problem determination and performance
management are typically complicated. There is no clear path for finding which
component faces the problem. Is it the database? A network problem? The
application server experiencing a bottleneck? A user machine stall? Sometimes,
these components even belong to different organizations.

Figure 1-1 shows a typical composite application. This is used by multiple users
through the Internet and intranet. It consists of multiple application layers, each
with its own abstraction level. Some of the applications have the original back
end in the mainframe transactions.

Figure 1-1 Composite application

2 Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

Composite applications are regarded as the ultimate application management
challenge, as they span different application servers that communicate with each
other. This architecture enables modular application development, where
changes in a layer may not affect other layers, but introduces the complexity of
multiple components.

This paper demonstrates how to implement the IBM Tivoli Composite Application
Manager family of products in a large-scale environment. This chapter introduces
IBM Tivoli product portfolio and how IBM Tivoli Composite Application Manager
product fits.

1.1.1 IBM Tivoli systems management portfolio
IBM Tivoli product solutions are aligned towards an overall IBM IT Service
Management approach. Figure 1-2 shows the IBM IT Service Management
portfolio structure.

IT CRM & Service Service Information Business
Business Delivery Deployment Management Resilience
Management & Support

IT Process
Management Products

IT Service Change and Configuration
Management Platform Management Database

IT Operational
Management Products

Best Practices Business Server, Network
Storage Security
Application & Device
Management Management

Figure 1-2 IBM IT Service Management

This approach provides Information Technology Infrastructure Library-aligned
automation work flows. Future offerings will provide an open standard-based and
configuration management database-based solution, as well as a workflow
engine.

Chapter 1. Overview of IBM Tivoli Composite Application Manager implementation 3

The operational management pillar shown in Figure 1-2 on page 3 is divided into
software families. The availability solution addressed in business application
management and server, network, and device management can be viewed as an
integrated offering, as shown in Figure 1-3.

Business Service Management

Orchestration and Provisioning

Security Event Correlation and Automation Storage

Composite Application Management

Resource Monitoring

Figure 1-3 IBM Tivoli software portfolio

As shown in Figure 1-3, the Tivoli software portfolio is divided into the following
components:
Resource monitoring
Measures and manages IT resource performance, including servers,
databases, and middleware.
Composite application management
Monitors and manages an application and its components, and understands
applications from the availability standpoint.
Event correlation and automation
Correlates and automates events or faults that are generated by resource
monitoring, application monitoring, or both to provide a concise root-cause
analysis of failure in the environment.
Orchestration and provisioning
Provides the ability to deploy or redeploy servers or components as
requested on demand to fulfill processing requirements, if the necessity
arises as indicated by the correlation engine.


Business service management
Provides a high-level view of business status as reflected by its underlying
monitoring components. The view is either in real time or based on a
service-level agreement.

1.1.2 IBM Tivoli Composite Application Manager solution
The IBM Tivoli Composite Application Manager family resides in the application
management pillar of the Tivoli software portfolio. The current application
management portfolio consists of the following products:
ITCAM for Response Time Tracking V6.1
ITCAM for Response Time V6.2
IBM Tivoli Composite Application Manager for service-oriented architecture
(SOA) V6.1
ITCAM for WebSphere V6.1
ITCAM for J2EE™ V6.1
ITCAM for Web Resources V6.2
ITCAM for CICS Transactions V6.1
ITCAM for IMS Transactions V6.1
IBM Tivoli OMEGAMON® XE for Messaging V6.0

Figure 1-4 shows the scope of composite application management.

Response Time WebSphere CICS/IMS
Tracking performance transaction

Web Services calls WBI messaging

Figure 1-4 Composite application management


Manage the overall composite application from the following sides:
Get the user side of response time and availability with ITCAM for Response
Time Tracking.
Get IBM WebSphere middleware performance and analyze in-depth resource
usage through ITCAM for WebSphere.
Manage messaging from IBM WebSphere Business Integration MQ Series
using IBM Tivoli OMEGAMON XE for IBM WebSphere Business Integration.
For more details, refer to Implementing IBM Tivoli OMEGAMON XE for
WebSphere Business Integration V1.1, SG24-6768.
Manage message flow in an SOA environment and collect metrics for Web
service calls using IBM Tivoli Composite Application Manager for
service-oriented architecture (SOA).
Provide the integration view with a mainframe-based, back-end application
such as Information Management System (IMS™) or Customer Information
Control System (CICS®) using ITCAM for IMS Transactions or ITCAM for
CICS Transactions.

1.2 Scope of and concerns relating to large-scale
implementation
This paper discusses large-scale implementation of IBM Tivoli Composite
Application Manager. It specifically provides information about the
implementation of ITCAM for WebSphere and ITCAM for Response Time
Tracking in large-scale environments. The discussion is about large-scale
implementation in distributed and mainframe environments, and includes the
following topics:
1.2.1, “Defining large-scale implementation” on page 6
1.2.2, “Concerns and considerations” on page 7

1.2.1 Defining large-scale implementation
There are several indications relating to large-scale implementation. These
indications are based on the following factors:
The number of application servers to be monitored
Each application server must have an agent installed to be monitored and
managed. With the number of application servers ranging from hundreds to
thousands, additional care must be taken to manage the deployment,
maintenance, and processing of the managing server.


The transaction rates on application servers
The transaction rates contribute to the overhead of the monitoring system. A
balance of data collection and system health must be achieved. A large
number of transactions potentially require larger management server
processing.
The number of network sites
The number of network sites typically corresponds to the potential bottlenecks
between the sites. The bottlenecks may be from production data, monitoring
data, or a security requirement such as a firewall.
The requirement for high availability or fail over
This additional requirement, although not directly related to the scale, is
typically a must for a large-scale implementation.
The existence of multiple managed spaces that a site must handle
Managed space is defined as a group of environments with a single
management database and a set of management server processes. Different
managed spaces are usually used to separate the production and
development environments. They are also used to prepare and test the
changes to the management environment.

1.2.2 Concerns and considerations
Following is a list of concerns and considerations that are specific to a
large-scale environment:
Server size
As this is a large-scale implementation, sizing the servers to manage the
environments is critical. The placement, configuration, and specification of a
single server or multiple servers must be predetermined in order to avoid
bottlenecks in processing. This sizing must also take into consideration
special processing requirements such as debugging and troubleshooting and
data collection and recovery.
Deploying agents
The number of agents that must be deployed are enormous and prohibitive to
being performed manually. Automated efforts must be included in the ability
to deploy and implement the agents automatically with minimal manual
intervention. This must cover initial deployment, fix pack implementation, and
maintenance action.


Security
This includes confidentiality support and firewall support.
– Confidentiality support secures information transfer between the agents
and the servers.
– Firewall support allows the sites to be secured, with management action
still flowing through in order to effectively manage the environment.
Reliability
Fail over and fault tolerance are critical to maintain while monitoring
business-critical applications. The reliability factor must be promptly
addressed and ensured.
Maintenance
Changes do happen, as with deployment. These changes must be applied to
both the servers and the agents. Special consideration must be provided for a
large-scale implementation with changes on both the servers and the agents.
While server consideration applies to preserving, monitoring, and data
collection with minimal downtime, agent consideration relates to automating
the deployment process with minimal manual intervention and outage.

This paper deals with and addresses these concerns for ITCAM for Response
Time Tracking and ITCAM for WebSphere implementations.

1.3 Overview of IBM Tivoli Composite Application
Manager
This section explains the following topics:
1.3.1, “Understanding ITCAM for WebSphere” on page 8
1.3.2, “Understanding IBM Tivoli Composite Application Manager for
Response Time Tracking” on page 11

1.3.1 Understanding ITCAM for WebSphere
This section provides an overview of ITCAM for WebSphere. The discussion
includes the following topics:
“Features and functions” on page 9
“Components” on page 9
“Platforms supported” on page 10


For more information about ITCAM for WebSphere, visit the following Web site:
http://www.ibm.com/software/tivoli/products/composite-application-mgr-
websphere/

Features and functions
ITCAM for WebSphere helps increase the performance and availability of
business-critical applications by providing facilities for real-time problem
detection, analysis, and repair. Correlation spanning Java 2 Platform, Enterprise
Edition (J2EE), Customer Information Control System, and Information
Management System, and diagnostics at the method level pinpoint code
problems to help resolve problems quickly and reduce support and operations
costs.

Today’s business processes often depend on a number of complex applications.
Although most businesses have traditional monitoring tools to manage individual
resources at a high level, many lack an integrated solution to automatically
monitor, analyze, and resolve problems at the service, transaction, application,
and resource levels. As a result, operations and development may take a long
time to identify, isolate, and fix composite application problems.

ITCAM for WebSphere is an application management tool that helps maintain
the availability and performance of on demand applications. It helps you to
quickly pinpoint, in real time, the source of bottlenecks in application code, server
resources, and external system dependencies. This product also provides
detailed reports that you can use to enhance the performance of your
applications. ITCAM for WebSphere provides in-depth, WebSphere-based
application performance analysis and a tracing facility.

ITCAM for WebSphere enables multiple levels of analysis to get a complete view
of the application, depending on the requirement. From production-level
monitoring to detailed heap and method debugging, it digs into Structured Query
Language (SQL) performance analysis without the need for database monitors. It
provides SQL information and information about calls that were made through
Java Database Connectivity (JDBC™). ITCAM for WebSphere provides a
composite status correlation for transactions that use Customer Information
Control System and Information Management System as the back end.

Components
ITCAM for WebSphere contains the following components:
Managing server
This acts as the central component that manages and administers the data
collectors. It stores that data in a relational database repository. A Web-based


application is provided to show monitoring results. This interface is also called
the visualization engine.
Data collector
This runs on the application server and collects performance information for
the managing server.
Tivoli Enterprise Monitoring Agent
This collects information that shows the status of the WebSphere Application
Server and sends it to the Tivoli Enterprise Monitoring Server for display on
the Tivoli Enterprise Portal. The Tivoli Enterprise Monitoring Agent is installed
on individual machines where data collectors reside. This component is
moved to IBM Tivoli Composite Application Manager for Web Resources in
Version 6.2.

Platforms supported
For a complete platform coverage list, refer to the following Web site:
http://publib.boulder.ibm.com/tividd/td/ITCAMWAS/prereq60/en_US/HTML/itc
am6.html

Table 1-1 provides an overview of the platforms supported for ITCAM for
WebSphere V6.

Table 1-1 Platforms supported for ITCAM for WebSphere
Component Software

Managing server operating IBM AIX V5.2 and V5.3
system Solaris™ 8 and Solaris 9 (SPARC)
Hewlett-Packard UNIX® (HP-UX) 11i 1
Windows® 200 Server or Advanced Server with
Service Pack 4 (SP4)
Windows 2003 Server Standard Edition/Enterprise Edition
(SE/EE)
Red Hat Enterprise Linux® (RHEL) 3.0 and 4.0
SUSE Linux Enterprise Server (SLES) 8 and 9

Managing server database IBM DB2® V8.1 Fix Pack 6 (FP6) or IBM DB2 V8.2
Oracle® 8i SE R3 8.1.7, Oracle 9i SE R2 9.2, Oracle 10g

Managing server WebSphere WebSphere Application Server V5.1.x or WebSphere Application
Server V6.x


Component Software

Data collector platform AIX V5.2 and V5.3
Solaris 8 and 9 SPARC
HP-UX 11i 1
Windows 200 Server or Advanced Server with SP4
Windows 2003 Server SE/EE
RHEL 3.0 and 4.0
SLES 8 and 9
Red Flag Advanced Server (RFAS) 4.0 and 4.1(xLinux)
IBM Operating System/400® (OS/400®) V5.2 and V5.3
IBM z/OS V1.4, V1.5, or V1.6

Customer Information Control V2.2, V2.3, and V3.1
System

Information Management V7.1, V8.1, and V9.1
System

1.3.2 Understanding IBM Tivoli Composite Application Manager
for Response Time Tracking
This section provides an overview of ITCAM for Response Time Tracking. It
discusses the following topics:
“Features and functions” on page 9
“Components” on page 12
“Platforms supported” on page 14

For more information about ITCAM for Response Time Tracking, visit the
following Web site:
http://www.ibm.com/software/tivoli/products/composite-application-mgr-rtt/

Features and functions
ITCAM for Response Time Tracking proactively recognizes, isolates, and
resolves transaction performance problems by using robotic and real-time
techniques. It is an end-to-end transaction management solution that monitors
user response time and helps you to visualize the transaction’s path through your
application systems, including the response time contributions of each step.
ITCAM for Response Time Tracking uses Application Response Measurement
(ARM) technology to track the response time of a distributed application.


Today’s business processes often depend on composite applications that span
Web servers, J2EE application servers, integration middleware, and mainframe
systems. Although most businesses have traditional monitoring tools to manage
individual resources, many lack an integrated solution to automatically monitor,
analyze, and resolve user response time problems. As a result, it may take a
long time to identify, isolate, and fix distributed transaction performance
problems.

ITCAM for Response Time Tracking enables you to follow the path of a user
transaction end-to-end across your business infrastructure. You can drill down to
each step the transaction takes as it travels across multiple systems, and
measure how each component of a transaction contributes to the overall
response time. The entire transaction analysis process is transparent to
customers and application developers. It collects transaction performance
through robot and browser simulation, in-depth J2EE server instrumentation, and
feedback from Customer Information Control System and Information
Management System.

ITCAM for Response Time Tracking feeds the Tivoli Enterprise Monitoring
Server to provide a comprehensive performance management solution on Tivoli
Enterprise Portal. This enables the development of custom monitoring
workspaces for managing enterprise applications.

Components
ITCAM for Response Time Tracking consists of the following components:
Management server
This acts as the central point of contact for ITCAM for Response Time
Tracking. It consists of a WebSphere-based J2EE application that performs
the management and administrative functions. The management server
stores data in a central database repository.
Store and Forward Agent
This relays traffic to and from the management agents. Typically, the Store
and Forward agent is used in a firewall environment. It consolidates the port
requirements for the connectivity.
Management agent
This performs the monitoring function. Typically, it investigates the
performance of the distributed application, depending on the management
components deployed on it. The components that you can deploy are:
– Generic Windows workstation
This allows deployment of IBM Rational® Robot to measure transaction
performance.


– Client Application Tracker
This uses IBM ETEWatch® scripts to collect performance information.
Default monitoring is available for measuring IBM Lotus® Notes® and
Microsoft® Outlook® performance.
– Synthetic Transaction Investigator (STI)
This performs Web-based transactions and measures the resulting
response time.
– Quality of Service monitoring agent
This collects information about user performance by acting as reverse
proxy between the user and the Web server.
– JavaTM 2 Platform, Enterprise Edition (J2EE) monitoring agent
This instruments and collects performance information about J2EE-based
application servers such as WebSphere or WebLogic.
– Web Response Monitor component
– Rational Performance Tester
– Tomcat and JBoss monitoring component
– Generic Application Response Measurement (ARM) agent
This collects ARM events from a custom-instrumented application.
Tivoli Enterprise Monitoring Agent for Tivoli Enterprise Monitoring Server
This feeds data from the ITCAM for Response Time Tracking server to
display on the Tivoli Enterprise Portal.


Platforms supported
For a complete platform coverage list, visit the following Web site:
http://publib.boulder.ibm.com/tividd/td/ITCAMRTT/prereq60/en_US/HTML/
Version60.html

Table 1-2 provides an overview of the platforms supported for ITCAM for
Response Time Tracking V6.0.

Table 1-2 Platforms supported for ITCAM for Response Time Tracking
Component Software level

Management server operating Microsoft Windows 2000 Server with SP4
system Windows 2000 Advanced with SP4
Windows 2003 Server SE or EE
IBM AIX V5.2 or V5.3
Solaris 9 or 10
HP-UX 11i 1
RHEL 3.0 or 4.0
SLES 8 or 9

Management server database Oracle 9i SE 9.2
IBM DB2 V8.1 ESE with FP3+ (required for WebSphere
Application Server V5.1.x)
IBM DB2 V8.1 ESE with FP6a+ (required for WebSphere
Application Server V6.x)
IBM DB2 V8.2

Management server WebSphere WebSphere Application Server V5.1.x or later versions
WebSphere Application Server V6.0.1.x or later versions

Management agent platform Windows 2000 Professional, Server or Advanced Server with
SP4
Windows 2003 Server SE or EE
Windows XP Professional with SP1
IBM AIX V5.2 or V5.3
Solaris 9 or 10
HP-UX 11i
RHEL 3.0 or 4.0
SLES 8 or 9
RFAS 4.0 or 4.1 (xLinux)
z/OS V1.4, V1.5, or V1.6
OS/400 V5.2 or V5.3


1.4 Document organization
This paper discusses the following topics:
Before the implementation
Chapter 2, “Planning for ITCAM for WebSphere” on page 17, and Chapter 5,
“Planning for ITCAM for Response Time Tracking” on page 93, discuss the
planning and sizing considerations.
The implementation
Chapter 3, “Installing ITCAM for WebSphere” on page 33, and Chapter 6,
“Installing ITCAM for Response Time Tracking” on page 105, discuss
additional steps that are required, such as reliability and automation
considerations.
After the implementation
Chapter 4, “Maintenance of ITCAM for WebSphere” on page 75, and
Chapter 7, “Maintenance of ITCAM for Response Time Tracking” on
page 137, discuss maintenance considerations and operational concerns
relating to a large-scale implementation.


2

Chapter 2. Planning for ITCAM for
WebSphere
This chapter provides information about areas that must be considered during
the planning phase of IBM Tivoli Composite Application Manager implementation
in a large environment. This chapter discusses the following topics:
2.1, “Planning considerations” on page 18
2.2, “Product architecture of ITCAM for WebSphere” on page 18
2.3, “Deciding on the size of the servers” on page 21
2.4, “Implementation options for ITCAM for WebSphere” on page 25
2.5, “Communication and security considerations” on page 29
2.6, “Reliability and high availability” on page 32


2.1 Planning considerations
This section discusses the following aspects pertaining to large-scale
implementations (see also 1.2.2, “Concerns and considerations” on page 7):
Understanding the product architecture
This allows you to make the correct decisions. Section 2.2, “Product
architecture of ITCAM for WebSphere” on page 18, describes the architecture
for ITCAM for WebSphere.
Sizing the servers
This is important to correctly acquire adequate servers and choose a sound
software configuration option. Section 2.3, “Deciding on the size of the
servers” on page 21, describes one approach.
Understanding the servers’ configuration options and agent deployment
This is discussed for ITCAM for WebSphere in 2.4, “Implementation options
for ITCAM for WebSphere” on page 25.
Planning for communication security
This is a mandatory step for an enterprise with business-critical and sensitive
information in a transaction environment. Section 2.5, “Communication and
security considerations” on page 29, discusses confidentiality and firewall
requirements.
Discussing reliability, failover, and disaster recovery issues
These are the other mandatory aspects pertaining to a critical business
process on a large enterprise. Section 2.6, “Reliability and high availability” on
page 32 discusses this.

2.2 Product architecture of ITCAM for WebSphere
This section discusses the product architecture of ITCAM for WebSphere. This
understanding is critical to plan and decide about the server configuration and
other implementation issues. See also IBM Tivoli Composite Application
Manager V6.1 Family Installation, Configuration, and Basic Usage, SG24-7151.

ITCAM for WebSphere V6.0 evolved from WebSphere Studio Application
Monitor (WSAM) and IBM Tivoli OMEGAMON XE for WebSphere. ITCAM for
WebSphere observes and reports on the health of Java 2 Platform, Enterprise
Edition-based applications. It tracks the progress of applications as they traverse
through Java 2 Platform, Enterprise Edition (J2EE) application servers,


middleware adapters and transports, and database calls, to back-end systems
such as Customer Information Control System (CICS) or Information
Management System (IMS) to extract business data or to invoke mainframe
business processes.

The tracking of applications produces request traces, where the events in a
request’s life are recorded and stored in a monitoring repository database.
ITCAM for WebSphere captures the CPU and the elapsed internal times when
events are called and exited, measuring as far down as the CPU consumed and
the elapsed internal times charged to individual methods in J2EE classes. The
methods or events taking the most time are marked as an application’s parts that
deserve attention for runtime improvement studies and code optimizations.

ITCAM for WebSphere does not require modification of any J2EE or mainframe
application code. Java Virtual Machine Tool Interface (JVMTI) interfaces and
primitives, along with WebSphere Performance Management Interface (PMI) and
z/OS System Measurement Facility (SMF) 120 records, are ITCAM for
WebSphere’s principal data sources. The monitoring data is collected and
analyzed to offer a wealth of information about the health of J2EE applications
and their servers.

Many system-level performance metrics are collected and reported about J2EE
application servers. The status of the servers and their resources, particularly at
vital checkpoints such as CPU utilization, memory usage, and the status of
internal components such as database connection pools, Java Virtual Machine
(JVM™) thread pools, Enterprise JavaBeans™ (EJB) usage, and request
processing statistics, are very important in locating real-time problems with J2EE
applications. ITCAM for WebSphere brings attention to these critical indicators
with real-time, graphical displays of their values and their trends over a span of
time.

ITCAM for WebSphere is a distributed performance monitoring application for
application servers. Its components are connected through IP network
communication. The central component of ITCAM for WebSphere, the managing
server, is its heart and brain. It collects and displays various performance
information from application servers.

The application servers run a component of ITCAM for WebSphere called data
collector, which is a collecting agent that runs in the application server and
sends monitoring information to the management server. These data collectors
operate independently of each other.

Chapter 2. Planning for ITCAM for WebSphere 19

Figure 2-1 shows the overall architecture of ITCAM for WebSphere.

Browser interface
ITCAM
for WebSphere
Managing Server

I

Web Server Tivoli Enterprise
Monitoring Server
and
Application servers with Tivoli Enterprise
ITCAM for WebSphere Portal Server
Data collectors

Figure 2-1 ITCAM for WebSphere architecture

The application monitor comprises the following main parts:
Managing server
A managing server comprises several Java-based components that provide
the environment to collect and present management data.
Data collector agent
A data collector agent runs on each monitored application server, whether
J2EE, Customer Information Control System (CICS), or Information
Management System (IMS), and communicates essential operational data to
the managing server. Unique sampling algorithms maintain low CPU and
network overhead, while providing application-specific performance
information.


2.3 Deciding on the size of the servers
The scale of the implementation must decide the size of the servers to be used.
Sizing determines the hardware configuration and implementation consideration
of the servers. This section discusses the following topics:
2.3.1, “Sizing parameters” on page 21
2.3.2, “Sizing estimation for ITCAM for WebSphere managing server” on
page 22

2.3.1 Sizing parameters
The following parameters must be considered before deciding on the size of the
servers:
The number of data collectors for ITCAM for WebSphere
This value assumes that the application servers run a similar load profile. If
the application servers have several load profiles, consider them in different
groups.
The transaction rate for application servers
The number of transactions executed for each minute, when multiplied with
the number of data collectors or monitoring agents, gives the total amount of
transaction information captured for a given period.
The complexity of a transaction
It is not easy to understand the complexity of a transaction. This requires a
more subjective approach than transaction rate counting, which can be
retrieved from the transaction data or the application log. The relative
complexity of transactions is determined by the number of method calls per
transaction. Typically, the number of methods a complex transaction invokes
is around four to six times that of a simple transaction.
There are some product-specific parameters that affect sizing considerations.
These parameters are built to filter out unimportant or insignificant information
from the data that is collected. These parameters are:
– Data collection filter
– Sampling rate
– Monitoring level
– Listening policy mask
– Instrumentation level


2.3.2 Sizing estimation for ITCAM for WebSphere managing server
Specific to ITCAM for WebSphere, consider the following parameters for sizing:
Communication bandwidth
Memory size
Processing requirement
Database size

Important: Sizing estimation for ITCAM for WebSphere managing server
must be estimated for a worst-case scenario, that is, in the state that level 3
monitoring is run for the highest number of data collectors concurrently.

Several communication traffic flows exist between the managing server and the
data collector. The communication traffic flows are:
Initial communication with the kernel to collect configuration information
This only happens in the initial connection when the data collector is started.
This configuration information consists of sending the configuration and
managing server Java archives.
Management information to modify data collection level, sampling interval, or
logging level from the kernel
This happens by request or when scheduled by Monitoring On Demand®.
The size of this communication is small and negligible.
Visualization engine requests for current active transactions
The impact of these requests depends on the following factors:
– The transaction rate and the average transaction response time that make
up the average number of in-flight transactions
– The number of concurrent Web console users who may request the
in-flight transaction information
Transaction information is streamed to the publish server as it happens
This is the largest contributor to network load. It uses up the largest amount of
network bandwidth. The formula is as follows:
– Monitoring in level 1: transaction rate x 353
– Monitoring in level 3: (transaction rate x 353) + (transaction rate x method
call x 172)


As an illustration, we use a sample environment with a transaction rate of
3,000 requests per minute on level 1 and 300 requests per minute on level 3
monitoring. The average method calls is 500 methods per requests. The
transaction bandwidth required is:
– Level 1 transaction load: (3000 transaction / 60 seconds) x 353 = 17,650
bytes/sec
– Level 3 transaction load: (30 transaction / 60 seconds) x 353 + (30 / 60) x
5,000 x 172 = 430,176 bytes/sec
As shown in this example, the majority of network usage is spent on level 3
analysis. In a real production environment, for the majority of time, ITCAM for
WebSphere runs on level 1. Therefore, the communication requirement is
low. However, prepare an installation to occasionally increase monitoring in
level 3 for problem determination purposes.

Memory size
Memory requirement is typically important for the following components:
Kernel
The memory size of the kernel is directly related to the number of data
collectors. The typical size of 64 MB in the setenv.sh may have to be
increased for more than 50 data collectors.
Publish server
The memory size is related to the number of transactions the publish server
has to process, with some consideration to the transaction complexity factor,
that is, the number of methods invoked. The publish server’s memory must be
adequate to handle the data size between garbage collector intervals. For
garbage collection per minute, you must accommodate a minute’s worth of
data. In the example provided in “Communication bandwidth” on page 22, the
total size of publish server memory for processing the load must be around
4.3 x 60 x (1.5) = 387 MB. Note that the base publish server was already
using around 100 MB of storage.
Archive agent
This requires memory as a subset to the publish server and is masked by the
sampling percentage from the publish server. The archive agent uses more
memory than the sampling rate percentage, as it performs Java Database
Connectivity (JDBC) database calls.


Visualization engine memory size
This depends on the number of users who are connected and the activities
that they perform. Users are categorized into the following groups:
– Users monitoring the availability screens
– Users collecting performance reports
– Users monitoring in-flight threads
Modify the visualization engine’s memory size by using the WebSphere
Application Server administration console.

Memory sizes for ITCAM for WebSphere components are defined in the
setenv.sh file that is sourced by all overseer components.

Processing requirement
The processor requirement for ITCAM for WebSphere is directly related to the
transaction rate. The largest processor usage is for the following components:
Publish server: to process transaction data
Database engine: for interface to the database
Archive agent: to perform SQL calls
WebSphere Application Server: to process user requests

Database size
The typical database size requirement depends on:
The number of application server statistics
The transaction volume to be stored
The complexity of transaction
The duration to keep the data

Database table information that increases in size during ITCAM for WebSphere
execution is:
requests: number of requests x 353 bytes
methods: number of methods x # requests in L3 x 172 bytes
pmidata: number of data collectors x (3600/polling interval) x 73 bytes
serverstats: number of data collectors x (3600/polling interval) x 107 bytes
volumestats: number of data collectors x (3600/polling interval) x 74 bytes
memorydata: number of data collectors x (3600/polling interval) x 115 bytes
gcdata: number of data collectors x (3600/garbage collection interval) x104
bytes


2.3.3 Data collector overhead
Monitoring with ITCAM for WebSphere has overhead related to data collectors
running on a production WebSphere Application Server. The overhead is
minimal for data collectors running on level 1 monitoring. This is typically around
a 2–3% increase of CPU time with no notable memory or disk input/output (I/O)
requirement.

When the monitoring level is increased, the processing overhead of ITCAM for
WebSphere data collectors also increases. This increase is due to the fact that
ITCAM for WebSphere collects more data from more sources. A typical level 2
monitoring generates around a 10% increase in processing usage, while a level
3 monitoring generates around 25–30% overhead.

This means that level 2 or level 3 monitoring must be used sparingly in your
production environment. To change the monitoring level for purposes of problem
determination, schedule it to start and then step back to level 1 automatically in
order to reduce the impact on users.

2.4 Implementation options for ITCAM for WebSphere
Depending on the size of your implementation, there are some considerations for
implementing ITCAM for WebSphere. This section discusses the following
topics:
2.4.1, “Designing the managing server” on page 25
2.4.2, “Deploying a large number of data collectors” on page 28

2.4.1 Designing the managing server
The ITCAM for WebSphere managing server consists of the following products:
IBM DB2 Universal Database™ Enterprise Server or Oracle database server
WebSphere Application Server
ITCAM for WebSphere managing server


Figure 2-2 shows the conceptual relationship between the components.

Snapshot traffic
Publish traffic
Global Publish
Server (SAM)

Publish Server (PS)
Kernel (KL) Visualization Engine
Message Dispatcher Provide services on: Provide services on:
(MD) - Lookup -Administration
- Registration -Availability
- Recovery -Problem Determination
Archive Agent (AA) - Configuration -Performance Management

Polling Agent (PA)

OCTIGATE
database

Figure 2-2 ITCAM for WebSphere components

The following ITCAM for WebSphere components are displayed in Figure 2-2:
Kernels
These control the managing server. There are always two copies of kernels
running on an ITCAM for WebSphere managing server for redundancy and
failover. The kernels register components as they join the managing server,
periodically renew connections and registrations with components and data
collectors, and collect server and component availability information.
Publish servers
These receive application and system event data from the data collectors,
gather and compute request-level information about performance metrics
such as response times, and implement the trap monitoring and alerts
features.
Archive agents
These receive monitoring data from the publish servers and store the
monitoring data in ITCAM for WebSphere’s repository.
Global publishing server
This collects information from the publish servers and correlates all parts and
pieces of multi-server requests, such as requests from J2EE servers to
execute Customer Information Control System (CICS) or Information
Management System (IMS) programs.


Message dispatcher
This is a conduit for messages from ITCAM for WebSphere using e-mail and
Simple Network Management Protocol (SNMP) facilities.
Polling agent
This collects data from Web servers for Apache 2.0 and later versions.
Visualization engine
This is a Web-based graphical user interface (GUI) with access to graphics,
ITCAM for WebSphere performance reports, real-time views of different slices
of monitoring data, ITCAM for WebSphere internal commands, and
event-driven functions. The visualization engine runs on a J2EE server such
as WebSphere Application Server.

Although ITCAM for WebSphere provides the facility to install all the components
in a single wizard, which is called embedded installation, individually installing
each component allows more flexibility in terms of verifying each component and
configuring them to suit your requirements. The considerations that you must
keep in mind when installing the components are:
Database
You can install the database locally on the managing server or on a separate
database server. ITCAM for WebSphere provides database configuration
scripts to assist with the configuration of a remote database.
Utilizing a remote database, regardless of whether it is a DB2 Universal
Database or an Oracle database, relieves the processing load on the
managing server. An environment with hundreds of data collectors generates
a large amount of data flowing into the database. This amount increases
considerably if the data collectors are set to run monitoring at level 2 or level
3, even for a short period of time.
A remote database allows database query processing and recording to be
processed using dedicated hardware, instead of sharing with the main
managing server that is already busy with processing the transaction
information.
The visualization engine of the managing server acts as the administration
console for ITCAM for WebSphere. The visualization engine is deployed on a
WebSphere Application Server JVM that resides in a standalone application
server or an application server that is a part of a network deployment
environment.
We recommend that you install the visualization engine on a separate
application server JVM that is not monitored by ITCAM for WebSphere data
collectors, especially in a network deployment environment. This reduces any


possible conflicts that may arise with respect to ITCAM for WebSphere. In
addition, if an issue does arise, problem determination will be somewhat
easier due to the separation.
ITCAM for WebSphere components
Configure the managing server to handle large amounts of data by adding
additional components, such as the publish servers and the archive agents.
When adding the publish servers and the archive agents, the distribution of
data is handled by the managing server. The amount of data being written to
the database is handled more efficiently as well.
Another major consideration for the managing server is the split server
installation. This option provides the managing server with the overseer
processes that exist on separate machines, including the kernel, which
provides load balancing and failover capabilities.
There are benefits to this type of configuration when there are hundreds of
data collectors providing data to the managing server. This type of setup not
only allows the managing server to handle more memory and disk space
usage, but also provides a failover capability. For more information about split
server installation, refer to 3.1, “Installing ITCAM for WebSphere managing
server” on page 34.

2.4.2 Deploying a large number of data collectors
Installation of a small amount of ITCAM for WebSphere data collectors is
performed by using the graphical-based installation and configuration wizard
provided by the product. When presented with the task of deploying hundreds of
data collectors into an environment, the graphical interface is no longer a good
option. This non-interactive automated installation method is commonly known
as silent installation.

The use of silent installation provides a means to deploy a larger number of data
collectors in a more efficient manner. When performing the silent installation,
information about the WebSphere environment must be known ahead of time. A
response file will be used during the installation, and if incorrect information is
used, may result in a failed install.

When performing the silent installation of data collectors, the WebSphere
Application Server version must be taken into account, as V6 introduced the
usage of profiles. The response files for silent installation are different for various
versions of the WebSphere Application Server. In some cases, when two
versions of WebSphere Application Server are present, it is better to have two
separate master response files.


Installing the Tivoli Enterprise Monitoring Agent (TEMA) is also an option of the
silent installation. Although Tivoli Enterprise Monitoring Agent can be installed
using silent installation, more configuration must be performed to connect to IBM
Tivoli Monitoring V6.1 Tivoli Enterprise Monitoring Server.

Mass automated installation is also possible by using a software distribution or
provisioning solution such as the IBM Tivoli Configuration Manager.

There is an additional consideration for deploying data collectors on a machine
that has multiple application servers installed. Consider installing a separate data
collector directory set for each application server, because applying a fix pack for
data collectors requires you to stop the application server. You have a more
flexible scheduling option with separate data collector installation for each
application server.

2.5 Communication and security considerations
Communication and security issues are vital to the inter-networked world that we
live in. Applications and their management infrastructure must be secured in
order to protect resources from unauthorized sources. This section discusses the
following planning considerations:
2.5.1, “Communication security” on page 29
2.5.2, “Firewall and port consideration” on page 30

2.5.1 Communication security
Communication security relates to the confidentiality of the information
transmitted over a network. Management information that is used by IBM Tivoli
Composite Application Manager products may contain details about application
processing internals. This requires the content of the management information to
be secured from being accessed by unauthorized sources.

WebSphere security
WebSphere security plays a significant role in a large-scale implementation. In
some cases, WebSphere security is not enabled during the test phase of an
implementation, but in a production environment. This requires certain additional
considerations. The WebSphere user must have the appropriate permissions to,
for instance, issue a wsadmin command.

The configuration of data collectors involves the use of Java Command
Language (JACL) scripts, and can fail when there is a permission problem.


If any of the application servers on which the data collector is installed has
WebSphere security enabled on it, the entire ITCAM for WebSphere
environment must have it enabled as well. This includes WebSphere security
being enabled on the ITCAM for WebSphere managing server.

Secure Sockets Layer communication
Secure communication between the managing server and the data collector is a
viable option if there is a requirement for data to be encrypted during
transmission. Using Secure Sockets Layer (SSL) provides secure data
transmission from the data collector to the managing server and must appease
corporate security requirements, if necessary. Additional configuration must take
place on the managing server and the data collector when enabling SSL. A
certificate key generator is included with the product. This key generator
provides the facility to use custom-generated keys.

A best practice is to complete the default installation of the managing server and
the data collector and then enable SSL for both. This isolates problems (that is,
whether the problem is caused by the basic installation or the SSL configuration).

2.5.2 Firewall and port consideration
Firewall and port issues arise when the data collectors are on a different site,
location, or subnet from the managing server. Problems such as name resolution
occur if the Domain Name System (DNS) is not set up correctly on either the
managing server or the data collectors. Routing problems occur if the Internet
Protocol (IP) addresses used belong to different subnets. The entire network
environment must be looked into in order to determine where a firewall, router, or
bridge may exist.


Figure 2-3 shows the communication port requirement for ITCAM for
WebSphere.

DC
Command Agent

KL1

DC
Event Agent
DC
Command Agent KL2

Port
Consolidator

DC PS1
Event Agent
DC
Command Agent

PS2

DC
Event Agent

Figure 2-3 Communication port requirements

The managing server requires open ports for each kernel and publish server.
The data collector requires open ports for the command agent and the event
agent. The port consolidator requires a port to communicate to the managing
server. Use a single port consolidator to consolidate communication from
multiple data collectors.

A port consolidator is useful to limit the number of ports required for
communication between the data collector and the managing server. Port
consolidation is a viable option if there is a limit to the number of ports that can
be opened on the firewall. Additional configuration must be carried out on the
data collector, including the configuration of the data collector to go through the
port consolidator, and starting the port consolidator process.


2.6 Reliability and high availability
This section discusses reliability issues that relate to failover and disaster
recovery.

2.6.1 Failover and fault tolerance
Split server configuration for ITCAM for WebSphere or the clustering server for
ITCAM for Response Time Tracking consists of having two or more management
servers running on separate physical machines. Hardware or software errors do
occur on a machine and cause the server to cease functioning. Using the
separate server configuration, the secondary server can handle the entire load
until the failing machine is recovered.

The switchover to the secondary managing server is not automatic. Manual
intervention must take place for the failover to be successful. There are specific
ITCAM for WebSphere components that can only be run on one managing
server. They must therefore be started on a secondary server, such as the global
publish server or the message dispatcher, if the primary server goes down.

2.6.2 Disaster recovery
There are three areas where a backup is necessary for disaster recovery with
respect to the IBM Tivoli Composite Application Manager server:
Database
Perform database backup regularly to collect the most up-to-date information.
Use the database utility function to perform the backup function.
WebSphere Application Server configuration for the server and agent or data
collectors
Perform a backup for these for them to be restored in a disaster recovery
scenario.
IBM Tivoli Composite Application Manager servers
These servers must be physically considered for recovery in the disaster
recovery site.


3

Chapter 3. Installing ITCAM for
WebSphere
This chapter discusses the installation and deployment procedure for ITCAM for
WebSphere. This chapter provides information about the following topics:
3.1, “Installing ITCAM for WebSphere managing server” on page 34
3.2, “Deploying data collectors” on page 50
3.3, “Configuring and setting up SSL communication” on page 64


3.1 Installing ITCAM for WebSphere managing server
This section discusses some of the considerations for installing the ITCAM for
WebSphere managing server and provides information pertaining to the
following topics:
3.1.1, “Installation configuration” on page 34
3.1.2, “Database installation and configuration” on page 35
3.1.3, “WebSphere Application Server considerations” on page 41
3.1.4, “Configuring the split server of the managing server” on page 42
3.1.5, “Installation and setup of split server” on page 43
3.1.7, “Adding additional publish servers and archive agents” on page 48

3.1.1 Installation configuration
The installation configuration for IBM Tivoli Composite Application Manager for
WebSphere managing server in a large-scale environment is shown in
Figure 3-1.

OCTIGATE

Managing Server machines Database Server

Figure 3-1 Installation configuration for ITCAM for WebSphere managing server

Depending on the scale of your implementation, choose to implement one or
more of the following structures:
Remote database, so that the database processing overhead does not hinder
the processing of the managing server, although the database server must be
connected through a local network to the managing server to minimize the
networking overhead for storing the data over the network.


The managing server split-server configuration facilitates the load to be
spread into several machines, besides allowing fault tolerance of the
managing server components.

The configuration that is discussed here covers the database, application server,
and the managing server overseer components.

3.1.2 Database installation and configuration
As mentioned in 2.4, “Implementation options for ITCAM for WebSphere” on
page 25, utilizing a remote database for the managing server helps reduce
memory consumption and processing load in a large-scale implementation. This
section discusses how to deploy the database on a remote server using IBM
Database 2™ (DB2) Universal Database (UDB), Enterprise Server Edition.

Instructions about how to set up and configure a remote database are available
in Chapter 13 of IBM Tivoli Composite Application Manager for WebSphere
Installation and Customization Guide, GC32-9506.

Implementing ITCAM for WebSphere with a remote database involves the
following steps:
1. Installing the DB2 UDB, Enterprise Server Edition on the database server
machine.
2. Installing DB2 Client Application Enabler on the managing server machines
and visualization engine machines. These clients are necessary for
communicating with the remote database server.
3. Cataloging database information about the DB2 clients.
4. Defining the OCTIGATE database structure on the remote database server
machine.

The software installation is not discussed here, as this must be performed in
accordance with the database documentation.

DB2 Universal Database server considerations
The DB2 UDB installation includes defining the distribution files and file systems
for the database itself. In a large environment where performance is critical,
follow these recommendations:
Store the database files in a separate physical disk rather than the operating
system paging and DB2 UDB system files. This allows you to minimize disk
contention for accessing the database files.

Chapter 3. Installing ITCAM for WebSphere 35

Allocate adequate memory for DB2 UDB application pools to ensure
performance. The most important pool is the DB2 bufferpool, which is used to
buffer the database data for quick and efficient access.
Use a faster disk for database storage, possibly separating the file systems
for recovery log and data.

Database installation scripts
ITCAM for WebSphere provides remote database creation scripts in the tar file
supplied in the managing installation image:
<ms-installation-image-directory>/base/scripts/db2-remote-scripts.tar

The scripts in the tar file are aimed at a UNIX-based environment installation or a
Windows-based environment with Microsoft Windows Services for UNIX (SFU)
installed.

On untarring the db2-remote-scripts.tar file, you will be presented with various
installation scripts in the bin directory that is created. The primary executable is
db2install. Depending on the platform, different syntax exist for executing the
db2install command. The following example uses amuser as the owner ID and
admin as the default user from the visualization engine.
For Windows, the syntax is:
db2install.bat jdbc_user jdbc_password adminVEuser instance_user
instance_password
For example:
db2install.bat amuser ampasswd admin db2inst1 dbpasswd
For UNIX, the syntax is:
db2install.sh db2username db_user absolute_path_of_install_directory
adminVEUser
For example:
./db2install.sh db2inst1 amuser /tmp/db2/ admin

The script uses the su command several times to switch the user as the DB2
instance owner. We recommend that you log in as the root user, so that you are
not prompted for the DB2 user password multiple times. If you log in as the root
user, add the root user to the DB2 administrator group, typically called db2grp1.

You must also define the installation owner user in the operating system of the
database machine. This user is typically called amuser and requires
administrative access.


Cataloging the database on the DB2 Universal Database client
The DB2 client must be able to connect to the actual DB2 OCTIGATE database.
This connection is defined using the following definitions:
Node definition
This indicates where the DB2 UDB server resides.
Database definition
This indicates the database selection in the node.

Start the DB2 UDB command-line interface using the db2cmd command in a
Windows-based platform or by sourcing the db2profile on a UNIX-based
platform.

To catalog the node, issue the following command:
db2 catalog tcpip node <ipnode> remote <ipaddr> server <ipport>

Following is a description of the parameters in this command:
ipnode: the database node name.
ipaddr: the TCP/IP host name or address the client addresses the server by.
ipport: the port number on which the DB2 server listens. The default port
number is 50000.

To catalog the database, issue the following command:
db2 catalog database OCTIGATE at node <ipnode>

In this command, ipnode is the database node name as defined by the catalog
tcpip node command.


The actual OCTIGATE database must exist to connect to the database. The
connection is performed using the db2 connect command to OCTIGATE.
Figure 3-2 shows a successful connection.

# . /home/db2inst1/db2profile
# db2 connect to OCTIGATE user db2inst1 using db2passw

Database Connection Information

Database server = DB2/AIX 8.2.0
SQL authorization ID = DB2INST1
Local database alias = OCTIGATE

Figure 3-2 Successful connection to OCTIGATE

We recommend that you create the database first and then catalog it. This is to
help you to test the connection to the database from the DB2 client immediately.
Although you can catalog the database without testing the connection, testing it
ensures connectivity.

Changing database name from OCTIGATE

Note: Although we do not recommend changing the database name of the
ITCAM for WebSphere managing server from OCTIGATE, some installations
may have strict naming conventions for the database name.

To change the database name OCTIGATE to another name, there are some
modifications required on:
Installation and configuration scripts
Runtime database connection

Example 3-1 illustrates the changes required in db2configuration.sh to modify the
database name from OCTIGATE to CAMFWAS.

Example 3-1 Modification to db2configuration.sh
# Drop the octigate database first
echo "$DBUSER> db2 drop database camfwas"
db2 drop database camfwas
#Fix SQL1005N problem.
#One of the causes is that octigate database may be created by other DB2
instance
#so it still exists in system catalog.
echo "$DBUSER> db2 catalog database camfwas"


db2 catalog database camfwas
echo "$DBUSER> db2 drop database camfwas"
db2 drop database camfwas
echo "$DBUSER> db2 uncatalog database camfwas"
db2 uncatalog database camfwas
# Create the octigate database
echo "$DBUSER> db2 create database camfwas using codeset UTF-8 territory US"
db2 create database camfwas using codeset UTF-8 territory US
status="$?"
echo "$DBUSER> db2empfa camfwas"
db2empfa camfwas
# Configure DB2 and the Octigate Database
echo "$DBUSER> db2 -tf $2"
db2 -tf $2
# echo "Change IBMDEFAULTBP size to -1 so BUFFPAGE setting will be used"
db2 connect to camfwas
...

Example 3-2 illustrates the changes required in db2createschema.sh to modify
the database name to CAMFWAS.

Example 3-2 Modification to db2createschema.sh
# Connect to DB2
echo "$DBUSER> db2 connect to camfwas"
echo "$DBUSER> db2 -tf $DBSCRIPT"
su - $DBUSER -c "db2 connect to camfwas; db2 -tf $DBSCRIPT; db2 terminate"
#Change the database table to update the user "admin"
echo "$DBUSER> db2 connect to camfwas"
status="$?"
echo "$DBUSER> db2 update users set username = '$ADMINUSER', ext_user =
'$ADMINUSER' where username = 'admin' and ext_user = 'admin'"
su - $DBUSER -c "db2 connect to camfwas; db2 "update users set username =
'$ADMINUSER', ext_user = '$ADMINUSER' where username = 'admin' and ext_user =
'admin'" ; db2 terminate"

Example 3-3 illustrates a successful database connection and creation. Start the
DB2 UDB command-line interface using the db2cmd command. Then connect to
the database as the schema user (for example, amuser) and list the table.

Example 3-3 Checking database
db2 => connect to camfwas user amuser using ampasswd
Database Connection Information
Database server = DB2/LINUX 8.2.4
SQL authorization ID = AMUSER
Local database alias = CAMFWAS
db2 => list tables


Table/View Schema Type Creation time
------------------------------- --------------- ----- --------------
CONFIG AMUSER T 2007-06-07-11.25.24.284983
CTG_SR_GATEWAY AMUSER T 2007-06-07-11.25.25.150959
...
2007-06-07-11.25.22.927237
VOLUMESTAT AMUSER T 2007-06-07-11.25.23.838550
WBI_BO_VIEW AMUSER V 2007-06-07-11.25.25.555333
WBI_REQUEST_OVERVIEW AMUSER T 2007-06-07-11.25.25.435186
WEBSERVERCHARTDATA AMUSER T 2007-06-07-11.25.24.772717

65 record(s) selected.

Two further modifications are required in the managing server to reference the
database:
$ITCAM_HOME/bin/setenv.sh
To change the database name from OCTIGATE to CAMFWAS, change the
definition of JDBC_DRIVER_URL to point to CAMFWAS. For example:
JDBC_DRIVER_URL=jdbc:db2://lima.itsc.austin.ibm.com:50000/camfwas
Set the JDBC properties in the WebSphere Application Server where
visualization engine is running:
a. Log on to the WebSphere Application Server administrative console.
b. Go to Resources → JDBC → Data Sources.
c. Select ITCAMDataSource.


d. Modify the database name in the DB2 Universal Database Property to
reflect the name of the ITCAM Database. For example, Figure 3-3
illustrates the change to the database CAMFWAS.

Figure 3-3 Configuration for ITCAMDataSource

3.1.3 WebSphere Application Server considerations
ITCAM for WebSphere requires the WebSphere Application Server V5 or V6 to
run the visualization engine. The WebSphere Application Server runs the
visualization engine Web application that acts as the ITCAM for WebSphere
Web console. The WebSphere Application Server can be a standalone server or
a part of a network deployment environment.

Considerations for operating the Web console
In case of a visualization engine error, minimize the load of the non-managing
server application and ensure access to the WebSphere administration console.
This is not a problem for a network deployment environment because the
administration console runs independently of the application server. For a


standalone server with WebSphere Application Server V5, do not use the default
server1, as it runs the administration console application. The reasons why you
must not do this are:
There will be some overhead from the administration console.
If ITCAM for WebSphere stops functioning, you are left without an
administration console and have to update the eXtensible Markup Language
(XML) files manually.

3.1.4 Configuring the split server of the managing server
The split server configuration of the managing server consists of running two
kernels on separate physical machines. Implementing the ITCAM for
WebSphere managing server in this manner provides failover capabilities for
reporting data collectors.

Figure 3-4 shows the split-server configuration in a two-system environment.
Note that this figure shows only the managing server components.

Kernel 1 Kernel 2

Publish Server 1 Publish Server 3
Publish Server 2 Publish Server 4

Archive Agent 1 Archive Agent 3
Archive Agent 2 Archive Agent 4

Visualization Engine
Message Dispatcher
Global Publish Server

Managing server - 1 Managing server - 2

Figure 3-4 Split server configuration

Note: Currently, the alerts and traps setting requires the publish server to
contact a local message dispatcher process. We recommend that you have a
message dispatcher on server1.


As Figure 3-4 on page 42 shows, there are two managing servers:
Managing server 1
This acts as the main server. In a split server configuration for ITCAM for
WebSphere, the following components can reside only on one of the
managing servers, particular the primary server:
– Visualization engine
– Message dispatcher
– Global publish server
Managing server 2
This acts as a backup server and offloads the processing of the primary
server.

3.1.5 Installation and setup of split server
Both of the managing servers have the following requirements:
For a Windows-based server, install Microsoft Services for UNIX.
Access to the OCTIGATE database, either locally or using the database client
software. Our example sets the DB2 client similarly for both of the managing
server machines.
Java environment, which is typically acquired from the WebSphere
Application Server installation on the machine. Even though you do not install
WebSphere Application Server, you still have to adjust JAVA_HOME to a
suitable Java Runtime Environment (JRE™). The WebSphere’s j2ee.jar must
also be present. Copy $WAS_HOME/java and $WAS_HOME/lib/j2ee.jar from
the primary managing server to the secondary server.

For ease of maintenance, we recommend that both servers run the same level of
software. One approach is to tar the primary managing server directory and
restore it on the secondary managing server. Alternatively, you can also install
the managing server software on both the machines.


Perform the following changes to the configuration files on one or both the
servers:
$ITCAM_HOME/bin/setenv.sh
This file determines the configuration information for the managing server.
a. To allow a split server, change the definition for KERNEL_HOST01 and
KERNEL_HOST02 to point to each managing server, assuming that you
use the same port definitions for both the servers. Each kernel requires
three ports for operation. Perform this activity on both the servers, as
shown in Example 3-4.

Example 3-4 Modification of setenv.sh
KERNEL_HOST01=srv152.itsc.austin.ibm.com
PORT_KERNEL_CODEBASE01=9122
PORT_KERNEL_RFS01=9120
PORT_KERNEL_RMI01=9178
KERNEL_HOST02=srv153.itsc.austin.ibm.com
PORT_KERNEL_CODEBASE02=9123
PORT_KERNEL_RFS02=9121
PORT_KERNEL_RMI02=9119

b. Edit the $ITCAM/etc/ms.properties file and set the kernel.hosts property,
as shown in Example 3-5.

Example 3-5 Example of ms.properties
kernel.hosts=srv152.itsc.austin.ibm.com:9122:9120:9178:9178,srv15
3.itsc.austin.ibm.com:9123:9121:9119

c. Adjust the setting for JAVA_HOME accordingly. This must refer to the
same Java Virtual Machine (JVM) environment that is similar to the
WebSphere Application Server. To check the Java version, use the java
-version command to make sure that you are using the same JVM as
your WebSphere Application Server.


$ITCAM_HOME/etc/ve.properties
This is the configuration for the visualization engine. Reflect the correct kernel
codebase and rfs addresses that are stored in the value of kernel.codebase
and kernel.rfs.address, as shown in Example 3-6. Perform this activity for
both the managing servers.

Example 3-6 Modification of ve.properties
kernel.codebase=http://srv152.itsc.austin.ibm.com:9122/kernel.core.j
ar
http://srv153.itsc.austin.ibm.com:9123/kernel.core.jar
kernel.rfs.address=srv152.itsc.austin.ibm.com:9120
srv153.itsc.austin.ibm.com:9121

For all the components that have been started in Managing server 2, edit the
corresponding control files for these components. Make sure that the entries
pointing to kernel codebase and ports like RMI and RFS are pointing to the
correct location. We edited the aactl.sh and psctl.sh files. These were the
components running on managing server 2, as shown in Figure 3-4 on
page 42.

Example 3-7 Modification of aactl.sh
CODEBASE_ADDRESS02=$KERNEL_HOST02:$PORT_KERNEL_CODEBASE02
KERNEL_CODEBASE=”http://$CODEBASE_ADDRESS02/kernel.core.jar”
RFS_ADDRESS=”KERNEL_HOST02:$PORT_KERNEL_RFS02”
RMI_CODEBASE=”http://$CODEBASE_ADDRESS02/archiveagent-intf.jar
http://$CODEBASE_ADDRESS02/archiveagent.jar”


$ITCAM_HOME/bin/am-start.sh
This is the startup script for ITCAM for WebSphere. Modify this script to
indicate which components must be started on which machine. Refer to
Figure 3-4 on page 42 to comment out the appropriate component. The
secondary managing server must suppress the kernel 1 and kernel 1 watch
dog. The primary managing server must suppress the kernel 2 and kernel 2
watch dog. Similarly, you have to add appropriate commands to start the
individual components on each server. Our sample am-start.sh for managing
server 1 is shown in Example 3-8. You must also shorten the wait time for the
watch dog, as there is only one kernel to start in each machine.

Example 3-8 Sample content of am-start.sh for managing server 1
${CYANEA_HOME}/bin/amctl.sh wd1 start
${CYANEA_HOME}/bin/amctl.sh aa1 start
${CYANEA_HOME}/bin/amctl.sh aa2 start
${CYANEA_HOME}/bin/amctl.sh ps1 start
${CYANEA_HOME}/bin/amctl.sh ps2 start
${CYANEA_HOME}/bin/amctl.sh md start
${CYANEA_HOME}/bin/amctl.sh sam1 start

$ITCAM_HOME/bin/am-stop.sh
Perform the similar modification in the am-stop.sh script as well, in order to
eliminate errors when stopping the components.

Note: For am-start.sh and am-stop.sh, start the primary managing server
first. Use an automation mechanism to coordinate this process. One such
automation mechanism is the use of a remote execution shell.


3.1.6 Verifying the installed components in a split environment
Start ITCAM for WebSphere on both the managing servers using the
am-start.sh command. Once all the components are started on both the
machines, verify the split server configuration using the Web console. Select
Administration → Managing server → Self Diagnosis. In the component
window, the running components and their Internet Protocol (IP) addresses are
displayed, as shown in Figure 3-5.

Figure 3-5 Self-diagnostic window that shows the managing server components

Note: The polling agent has been removed from this version of ITCAM for
WebSphere. You cannot add or configure the polling agent, even though an
empty node for the polling agent is displayed in the application monitor
component tree in the Web console.


3.1.7 Adding additional publish servers and archive agents
The basic configuration has only two publish servers and two archive agents. To
further distribute the load, add publish servers or archive agents. Typically, in a
split-server environment (shown in Figure 3-4 on page 42), two publish servers
and two archive agents per machine are defined.

Restriction: You can add only publish servers and archive agents. There are
only two instances of kernels and a single instance of global publish server,
message dispatcher.

The instructions for adding additional publish servers and archive agents are
derived from IBM Tivoli Composite Application Manager for WebSphere
Installation and Customization Guide, GC32-9506. Defining new components
includes the following modifications:
1. Run the following command from the $ITCAM_HOME/bin/ folder:
./add-ps.sh publish_server_port_number
This adds a new publish server. The output of the command is shown in
Example 3-9

Example 3-9 Output of executing add-ps.sh
[root@srv153 bin]# ./add-ps.sh 9105
Current number of ps = 2
New number of ps = 3
Creating a new /opt/IBM/itcam/WebSphere/MS/etc/ps3.properties, using
ps1.properties as a template
testing UUID=f0039d70-0f1b-dc01-c4dc-000c293bdac6
Use UUID=f0039d70-0f1b-dc01-c4dc-000c293bdac6
Creating a new /opt/IBM/itcam/WebSphere/MS/etc/log-ps3.properties by
copying log-ps1.properties
Done modifying setenv.sh
Done modifying am-start.sh
Done modifying am-stop.sh
Done modifying am-forcestop.sh

2. Run the following command from the $ITCAM_HOME/bin/ folder:
./add-aa.sh archive_agent_port_number


This adds a new archive agent. The output of the command is shown in
Example 3-10.

Example 3-10 Output of executing add-aa.sh
[root@srv153 bin]# ./add-as.sh 9031
Current number of AA = 2
New number of AA = 3
Creating a new /opt/IBM/itcam/WebSphere/MS/etc/aa3.properties, using
aa1.properties as a template
testing UUID=d078edc1-0e1b-dc01-2da7-000c293bdac6
Use UUID=d078edc1-0e1b-dc01-2da7-000c293bdac6
Creating a new /opt/IBM/itcam/WebSphere/MS/etc/log-aa3.properties by
copying log-aa1.properties
Done modifying setenv.sh
Done modifying am-start.sh
Done modifying am-stop.sh
Done modifying am-forcestop.sh

3. Modify the am-start.sh and am-stop.sh scripts to add the new components for
both servers:
a. For am-start.sh, if you have added two more archive agent aa3 and aa4,
respectively, and published server ps3 and ps4, the following entries must
be made (Example 3-11).

Example 3-11 Addition to am-start.sh
MS_home/bin/amctl.sh ps3 start
MS_home/bin/amctl.sh ps4 start
MS_home/bin/amctl.sh aa3 start
MS_home/bin/amctl.sh aa4 start

b. For am-stop.sh, use the commands shown in Example 3-12.

Example 3-12 Addition to am-stop.sh
MS_home/bin/amctl.sh ps3 stop
MS_home/bin/amctl.sh ps4 stop
MS_home/bin/amctl.sh aa3 stop
MS_home/bin/amctl.sh aa4 stop


3.2 Deploying data collectors
This section lists the configuration options for installing the data collector. You
modify the configuration options either in the response file or at the command
line to run the silent installation.

You have the option to only install the data collector, both install and configure the
data collector, or only configure the data collector using this procedure. If you
only want to configure the data collector, perform this the configuration after the
data collector has been installed.

Data collector deployment for a large installation requires a custom-automated
silent installation and configuration of ITCAM for WebSphere data collectors on
the target application servers.

This section discusses the following topics:
3.2.1, “Setting up the silent installation process” on page 50
3.2.2, “Installing the data collector” on page 53
3.2.3, “Installing and configuring the data collector together” on page 54
3.2.4, “Configuring data collectors” on page 56
3.2.5, “Automatically discovering the installation parameters” on page 61

3.2.1 Setting up the silent installation process
The silent installation process is supported by the install-DC command, with the
following syntax:
install-DC -silent -options inputfile

The input file can either be modified from a supplied sample or generated using
the installation wizard. In Version 6.1, the installation wizard and the
configuration tool wizard both have the ability to generate the silent installation
option file. Table 3-1 lists the available sample option files for the wizard. These
options files are supplied in the silent sub-directory of the installation image.

Table 3-1 Options files list
Application server type Sample option file name

WebSphere Application Server V6 DC6.opt

WebSphere Application Server V5 DC.opt

WebSphere Application Server DC_was.opt

WebSphere Portal Server DC_portal.opt


Application server type Sample option file name

WebSphere Process Server DC_ps.opt

WebSphere Enterprise Service Bus DC_esb.opt DC_esb.opt

Workplace™ Collaboration Service Mail Server DC_wcsmail.opt

Option for upgrading WebSphere V5 or V6 data collector migrate_DC.opt
V6.0 into V6.1

The install settings in the option files are commented using leading number
characters (###). If you enable an option, you must supply a value. A blank or a
null value is not acceptable. The silent installation option requires advance
knowledge of the target machine, including the target WebSphere environment.
A prefilled response file may not be efficient in a large environment. An external
mechanism is required to collect the information and populate the silent
installation control files beforehand. Some of this information requires the use of
naming conventions. See 3.2.5, “Automatically discovering the installation
parameters” on page 61.

Table 3-2 lists the parameters for installation.

Table 3-2 Installation parameters
Option string Description

-V disableOSPrereqChecking=true | false Whether to check the operating system level before the
installation.

-V LICENSE_REJECT_BUTTON=true | false Must be false.

-P installLocation=value Specifies the path of the installation directory for the data
collector.

-V LOG_DIR=value This specifies the writable directory to which the
installation and configuration programs will write log
files.

-W LogSetting.consoleOut=true | false Specifies whether to display messages issued by the
installation and configuration program on the console.

-V DC_WD_JAVAHOME=value Path of the Java home directory.

-V DC_CC_TEMA=true | false Whether to install Tivoli Enterprise Monitoring Agent
collector interface.

-V LAUNCH_CONFIG=true | false Specifies whether to launch or defer the data collector
Configuration Tool after installation.
-V DEFER_CONFIG=true | false


-V GC_LOG_PATH=value If Tivoli Enterprise Monitoring Agent is true, specifies the
path for the garbage collector log.

-W LogSetting.logLevel=level Level: FATAL, ERROR, WARNING, INFO,
DEBUG_MIN, DEBUG_MID, DEBUG_MAX, ALL.

Table 3-3 lists the parameters for configuration.

Table 3-3 Configuration parameters
Option String Description

-V LOG_DIR=value Writable directory to which the installation and
configuration programs write log files.

-V LAUNCH_CONFIG=true | false Specifies whether to launch or defer the data collector
configuration tool after installation.
-V DEFER_CONFIG=true | false

-V DC_CCUC_CONFIG=true | false Specifies whether to configure or unconfigure a data
collector.
-V DC_CCUC_UNCONFIG=true | false

-V DC_RECONFIG_ALLOW=true | false Indicates whether reconfiguring the data collector is
allowed.

-V DC_MSKS_SERVERNAME=value Fully qualified host name of the managing server.

-V DC_MSKS_CODEBASEPORT=value Remote file system port for the managing server.

-V DC_MS_AMHOME=value Installation directory of the managing server.

-V DC_CC_ITCAMFWAS=true | false Whether the application monitor interface support will be
installed.

-V DC_CAS_WAS=true | false Specifies which type of data collector to configure,
WebSphere Application Server, Enterprise Service Bus,
-V DC_CAS_ESB=true | false Process Server, or WebSphere Portal Server.
-V DC_CAS_PS=true | false

-V DC_CAS_WPS=true |false

-V APP_SERVER_NAMES=value Path to the application server.

-V WS_NODE_NAME=value Application server node name.

-V DC_WD_JAVAHOME=value Location of the Java home directory.

-V DC_WD_PROFILENAME=value Applies only to Version 6 application servers.

-V DC_WD_WASBASEDIR=value Location of the application server base directory.

-V DC_WD_WASVER=value Version of the application server.


-V DC_ASL_HOSTNAME=value Administrative application host name where the data
collector will be installed.

-V DC_ASL_SOAPPORT=value SOAP interface port for the administrative application.

-V WAS_BASEDIR=value Only for ESB, portal server, and process server,
indicates path and version of the base WebSphere
-V WAS_BASE_VERSION=value Application Server.

-V DC_CC_TEMA=true | false Whether to run secondary collector for Tivoli Enterprise
Monitoring Agent.

-V GC_LOG_PATH=value If Tivoli Enterprise Monitoring Agent is true, specifies
path for the Garbage Collector log.

-V TEMA_OFFLINE_ALLOW=true | false Allows Tivoli Enterprise Monitoring Agent to be off line.

-V FIREWALL_ENABLED=true | false Whether the data collector machine is behind a firewall.

-V PROBE_RMI_PORT=value If the firewall is enabled, define RMI port for probe agent.

-V PROBE_CONTROLLER_RMI_PORT=value If the firewall is enabled, define RMI port for controller.

-W LogSetting.logLevel=level Level: FATAL, ERROR, WARNING, INFO,
DEBUG_MIN, DEBUG_MID, DEBUG_MAX, ALL.

3.2.2 Installing the data collector
A sample option file for our WebSphere Application Server V6.1 is shown in
Example 3-13. This installation is on the Linux-based server.

Example 3-13 Sample option file for WebSphere Application Server V6.1
# Log Parameters
-V LOG_DIR="/var/ibm/tivoli/common"
-V disableOSPrereqChecking="true"
-W LogSetting.logLevel="ALL"
-W LogSetting.consoleOut="false"
-P installLocation="/opt/IBM/itcam/WebSphere/DC"
-V DC_WD_JAVAHOME=/opt/IBM/WebSphere/AppServer/java
-V LICENSE_ACCEPT_BUTTON="true"
-V LICENSE_REJECT_BUTTON="false"
-V LAUNCH_CONFIG="false"
-V DEFER_CONFIG="true"
-V DC_CC_TEMA="true"
-V GC_LOG_PATH="/var/ibm/tivoli/common/CYN/logs/gc.log"


Run the following silent installation command to silently install the data collector:
./setup_DC_<platform>.bin -silent -options <PATH>/DC6.opt

The option file in Example 3-13 on page 53 does not launch the configuration
tool. You can check the installation result in the
/var/ibm/tivoli/common/CYN/logs/trace-install.log file. When the installation is
successful, the end of trace-install.log contains the message in Example 3-14.

Example 3-14 Sample trace-install.log for a successful data collector installation
<LogText><![CDATA[Exit, return value = The InstallShield Wizard has
successfully installed ITCAM for WebSphere Data Collector 6.1.
Choose Finish to exit the wizard.]]></LogText>

3.2.3 Installing and configuring the data collector together
A sample option file for our WebSphere Application Server V6.1 is shown in
Example 3-15. This installation is on the Linux-based network deployment server
called srv107 with the managing server on srv152. Notice that the option -V
LAUNCH_CONFIG="true" since we are both installing and configuring the data
collector.

Example 3-15 Sample option file for WebSphere Application Server V6.1
# Install Parameters
-V LOG_DIR="/var/ibm/tivoli/common"
-V disableOSPrereqChecking="true"
-W LogSetting.logLevel="ALL"
-P installLocation="/opt/IBM/itcam/WebSphere/DC"
-V DC_WD_JAVAHOME=/opt/IBM/WebSphere/AppServer/java
-V LICENSE_ACCEPT_BUTTON="true"
-V LICENSE_REJECT_BUTTON="false"
-V LAUNCH_CONFIG="true"
-V DEFER_CONFIG="true"
-V GC_LOG_PATH="/var/ibm/tivoli/common/CYN/logs/gc.log"

# Configuration Parameters

-V DC_CCUC_CONFIG="true"
-V DC_CCUC_UNCONFIG="false"
-V DC_RECONFIG_ALLOW="true"
-V DC_OFFLINE_ALLOW="false"
-V DC_MSKS_SERVERNAME="srv152.itsc.austin.ibm.com"
-V DC_MSKS_CODEBASEPORT="9122"


-V DC_MS_AMHOME="/opt/IBM/itcam/WebSphere/MS"
-V DC_CC_ITCAMFWAS="true"
-V DC_CAS_WAS="true"
-V DC_CAS_ESB="false"
-V DC_CAS_PS="false"
-V DC_CAS_WPS="false"
-V DC_CAS_WCSMAIL="false"
-V
APP_SERVER_NAMES="cells/srv107Node01Cell/nodes/srv107Node02/servers/ser
ver1"
-V WS_NODE_NAME="cells/srv107Node01Cell/nodes/srv107Node02"
-V DC_WD_PROFILEHOME="/opt/IBM/WebSphere/AppServer/profiles/ProcSrv01"
-V DC_WD_PROFILENAME="ProcSrv01"
-V DC_WD_WASBASEDIR="/opt/IBM/WebSphere/AppServer"
-V DC_WD_WASVER="60"
-V DC_ASL_HOSTNAME="srv107.itsc.austin.ibm.com"
-V DC_ASL_SOAPPORT="8880"
-V DC_ASL_USERNAME="NULL"
-V DC_ASL_PASSWD="NULL"
-V WAS_BASEDIR="/opt/IBM/WebSphere/AppServer"
##
# TEMA Parameters
##
-V GC_LOG_PATH="/var/ibm/tivoli/common/CYN/logs/GC"
-V TEMA_OFFLINE_ALLOW="false"
##
# DC Host Parameters
##
-V AM_SOCKET_BINDIP="srv107.itsc.austin.ibm.com"
-V FIREWALL_ENABLED="false"

For additional flexibility and the reuse of response files, you can manually type
some configuration options on the command line, instead of including them in the
response file. This is useful for providing:
Unique options: to reuse the response file in multiple installations and
configurations, but some of the options are unique to each.
Password protection: to safeguard the password by manually entering it
during each installation and configuration. If you record the password in the
option (.opt) file, the password is unencrypted and visible to anyone who
opens the file.


The following command sets administrative server passwords on the command
line and relies on a response file to provide all other configuration details:
./setup_DC_lin.bin -silent -V DC_ASL_PASSWD="tivoli" -options
/<PATH>/DC6.opt

Note: Configuration options specified in the response file take precedence
over those entered in the command line.

3.2.4 Configuring data collectors
As a machine may have more than one application server, we can now configure
additional application servers on that machine. This is typically performed using
the config_DC utility that is installed in the ITCAM_HOME/config_DC directory.
You may also need to run the configuration if you defer launching the
configuration tool, as in 3.2.2, “Installing the data collector” on page 53.

There are two ways of configuring the data collectors without interaction:
Using the config_dc -silent utility and modifying the dcInpouts.txt
The drawback of this is that all the error messages require a graphical user
interface (Windows or X11) to be available and responded upon.
Using the Java Command Language (JACL) script configDataCollector.jacl

Running config_dc
Example 3-16 shows the options file that allows multiple data collector
configurations to be performed in a single run.

Example 3-16 Sample properties for configuring two servers in the same node

-V DC_CC_TEMA="false"


-V
APP_SERVER_NAMES="cells/khartoumCell01/nodes/khartoumNode01/servers/Cli
entSvc"
-V WS_NODE_NAME="cells/khartoumCell01/nodes/khartoumNode01"
-V DC_WD_PROFILEHOME="/opt/IBM/WebSphere/AppServer/profiles/AppSrv01"
-V DC_WD_PROFILENAME="AppSrv01"
-V DC_WD_WASVER="61"
-V DC_ASL_HOSTNAME="khartoum.itsc.austin.ibm.com"
##
# TEMA Parameters
##
-V GC_LOG_PATH="NULL"
##
##
-V AM_SOCKET_BINDIP="khartoum.itsc.austin.ibm.com"
##
##
-V
APP_SERVER_NAMES="cells/khartoumCell01/nodes/khartoumNode01/servers/Ser
verSvc"


-V WS_NODE_NAME="cells/khartoumCell01/nodes/khartoumNode01"
-V DC_WD_PROFILEHOME="/opt/IBM/WebSphere/AppServer/profiles/AppSrv01"
-V DC_WD_PROFILENAME="AppSrv01"
-V DC_WD_WASVER="NULL"
-V DC_ASL_HOSTNAME="khartoum.itsc.austin.ibm.com"
##
# TEMA Parameters
##
-V GC_LOG_PATH="NULL"
##
##
-V AM_SOCKET_BINDIP="khartoum.itsc.austin.ibm.com"

Run the following silent configuration command:

/opt/IBM/itcam/WebSphere/DC/config_dc/config_dc.sh -silent -options
itcam_dc_config.opt

The file dcInputs.txt gets created during the install process of the data collector
and is later used by the data collecter configuration wizard, along with an options
file (.opt).

During the configuration process the dcInputs.txt file (see Example 3-17) gets
further updated and is used by the remainder of the configuration process.

Example 3-17 Sample dcInputs.txt file
TEMP_DIR=/tmp/ibm_am_installer_dc/
TOOLKITHOME=/opt/IBM/itcam/WebSphere/DC/toolkit
JAVA_HOME=/opt/IBM/itcam/WebSphere/DC/_jvm/jre
AM_HOME_CONFIG_DC=/opt/IBM/itcam/WebSphere/DC/config_dc
OS=linux
FILESEP=/
DCHOME=/opt/IBM/itcam/WebSphere/DC/itcamdc
ITCAM61HOME=/opt/IBM/itcam/WebSphere/DC
UNCONFIGURE_SUCCESSFUL=true


DMGR_RMI=
TIVOLI_COMMON_DIR=/var/ibm/tivoli/common
OS_VERSION=2.6.9-42.ELsmp
CONNECTION=SOAP
JDK_VERSION=
WS_RMI_PORT=
CONFIGURE=true
JDK_TYPE=
COBRAND=ITCAM
REPLACE_JAVA_SECURITY_POLICY=true
LOG_COMMON_DIR=/var/ibm/tivoli/common
OS_NAME=RedHatEnterpriseLinux
AM_HOME=/opt/IBM/itcam/WebSphere/DC/itcamdc
LICENSE=true

Note: The <PATH>/DC/dc_config/dcInputs.txt file must exist in order for the
data collector configuration to succeed.

Example 3-18 shows a successful silent configuration entry in trace-install.log.

Example 3-18 Sample trace-install.log for a successful data collector configuration
<LogText><![CDATA[DATA: Configure Finish: Successfully
configured the application server
cells/khartoumCell01/nodes/khartoumNode01/servers/ClientSvc
cells/khartoumCell01/nodes/khartoumNode01/servers/ServerSvc.]]>
</LogText>

Running configDataCollector.jacl
Use a JACL script called configDataCollector.jacl that exists in the config_DC
directory. The input file for this script is found in the temporary directory that is
created from the installation.

To install the data collector into a secondary application server in the node
laredoNode02 with profile vbdpro01, we use the parameter file shown in
Example 3-19.

Example 3-19 Sample properties for configuring data collector
server=server1(cells/laredoNode02Cell/nodes/laredoNode02/servers/server
1|server.xml#Server_1144783086875)
server.genericJvmArguments=-Xbootclasspath/p:${AM_HOME}/lib/bcm-bootstr
ap.jar;${AM_HOME}/lib/ppe.probe-bootstrap.jar -Xrunam


-Djlog.propertyFileDir.CYN=${AM_HOME}/etc
-Djlog.qualDir=laredoNode02.server1
server.systemproperty0=com.ibm.websphere.classloader.plugin=com.cyanea.
bcm.websphere.BcmPlugin
server.systemproperty1=java.rmi.server.codebase=file://${MS_AM_HOME}/li
b6.0/ppe.probe.jar file://${MS_AM_HOME}/lib/ppe.probe-intf.jar
file://${MS_AM_HOME}/lib6.0/ppe.was_6.jar
file://${MS_AM_HOME}/lib6.0/ppe.probe-bootstrap.jar
server.systemproperty2=appserver.platform=was6010
server.systemproperty3=am.appserver=server1
server.systemproperty4=java.security.policy=C:ITCAMDC/etc/datacolle
ctor.policy
server.systemproperty5=deploymentmgr.rmi.connection=
server.systemproperty6=am.profilename=vbdpro01
server.systemproperty7=am.nodename=laredoNode02
server.systemproperty8=am.home=C:ITCAMDC
server.environment0=PATH=C:ITCAMDC/lib;
server.environment1=QUALDIR=laredoNode02.server1
server.environment2=_JVM_THREAD_DUMP_BUFFER_SIZE=25000000
server.customservice0.displayName=am
server.customservice0.classpath=
server.customservice0.enable=true
server.customservice0.description=Custom Service for ITCAMfWAS DC
server.customservice0.classname=com.cyanea.ws6.ProbeService
server.pmiservice.initialSpecLevel=connectionPoolModule=H:wsgwModule=N:
orbPerfModule=H:cacheModule=N:webAppModule=H:threadPoolModule=H:wlmModu
le=M:webServicesModule=N:beanModule=X:jvmRuntimeModule=H:systemModule=M
:servletSessionsModule=H:transactionModule=M:j2cModule=H
server.am.install.root=C:ITCAMDC
server.am.ms.home=/C:/IBM/itcam/WebSphere/MS

Apart from the apparent configuration values, the first line indicates the server ID.
Retrieve this from the appropriate server.xml file by finding the xmi:id=”Server_
string. The command we use to activate this property is:
. $WAS_HOME/bin/setupCmdLine.bat
$WAS_HOME/bin/wsadmin.bat -conntype SOAP -hostname laredo -port 8881 -f
$ITCAM_HOME/config_DC/configDataCollector.jacl attr.properties


3.2.5 Automatically discovering the installation parameters
To analyze the installed WebSphere software in a machine, use the
vpd.properties file. Depending on your platform, the vpd.properties file resides on
the following directories:
On AIX: /usr/lib/objrepos/vpd.properties
On Windows: %WINDIR%vpd.properties or %HOMEPATH%WINDOWS
On Linux, Solaris, HP-UX: /vpd.properties or /root/vpd.properties

The entries in the vpd.properties file are delimited by a long | character field
similar to that shown in Example 3-20.

Example 3-20 vpd.properties file entries
WSBAA60|6|0|0|0|6.0.0.0|1=WebSphere Application Server|IBM WebSphere
Application Server| |IBM|
|6.0.0.0|C:IBMWebSphereAppServer|0|1|WSBAA60CoreRuntime|6|0|0|0|6.0.
0.0|1|0|2|ref_33390|1|WSBAA60|6|0|0|0|6.0.0.0|1|0|false|"_uninst"
"uninstall.jar" "uninstall.dat" ""|true|3|WSBAA60|6|0|0|0|6.0.0.0|1

The following list is aimed at helping you identify the information:
The first identifier is the product ID.
WSB* identifies the base WebSphere Application Server.
The second field provides the version number.
The twelfth field helps retrieve the WebSphere Application Server installation
directory.


Retrieve the WebSphere Application Server instances by analyzing the directory
structure of the installation. The profiles for WebSphere Application Server V6
are in the profiles directory. From the WebSphere home path, get the application
server from the config subdirectory. Figure 3-6 shows a sample directory
structure.

Figure 3-6 Configuration directory structure

In the cell level, the application server type is seen. In Figure 3-6, the file is
profiles/default/config/cells/laredoNode01Cell/cell.xml. Example 3-21 shows our
sample cell.xml.

Example 3-21 Sample cell.xml
<?xml version="1.0" encoding="UTF-8"?>
<xmi:XMI xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI"
xmlns:topology.cell="http://www.ibm.com/websphere/appserver/schemas/5.0
/topology.cell.xmi"
xmlns:ipc="http://www.ibm.com/websphere/appserver/schemas/5.0/ipc.xmi">
<xmi:Documentation>
<contact>WebSphere Application Server v5.0 Default Configuration
Files v1.7 7/31/02</contact>
</xmi:Documentation>
<topology.cell:Cell xmi:id="Cell_1" name="laredoNode02Cell"
cellDiscoveryProtocol="TCP" cellType="STANDALONE">
</topology.cell:Cell>
</xmi:XMI>


In Example 3-21 on page 62, the bold type indicates the cell type — a
STANDALONE cell or a DISTRIBUTED network deployment-based cell.
For a standalone cell, retrieve the port information from serverindex.xml
under the node name folder, for example,
config/cells/laredoNode02Cell/nodes/laredoNode02/serverindex.xml.
Example 3-22 shows parts of the server index.

Example 3-22 Sample serverindex.xml
<serverindex:ServerIndex xmi:version="2.0"
xmlns:xmi="http://www.omg.org/XMI"
xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas/5.
0/serverindex.xmi" xmi:id="ServerIndex_1"
hostName="laredo.itsc.austin.ibm.com">

. . .

<specialEndpoints xmi:id="NamedEndPoint_1144783085594"
endPointName="SOAP_CONNECTOR_ADDRESS">
<endPoint xmi:id="EndPoint_1144783085594"
host="laredo.itsc.austin.ibm.com" port="8881"/>
</specialEndpoints>

. . .

</serverindex:ServerIndex>

For a distributed cell, first select the core group information to retrieve the
node name of the deployment manager. The core group information is stored
in a sub-directory under the cell called
coregroupsDefaultCoreGroupcoregroup.xml. Get the association between
the node and the dmgr server, as shown in Example 3-23.

Example 3-23 Excerpt from coregroup.xml
. . .
<coreGroupServers xmi:id="CoreGroupServer_1130943100094"
nodeName="laredoCellManager01" serverName="dmgr"/>
. . .

The serverindex.xml is found under the appropriate node, for example,
cells/laredoCell01/nodes/laredoCellManager01. The
SOAP_CONNECTOR_ADDRESS is similar to that shown in Example 3-22.


Based on the above consideration, a shell script can be written to retrieve all the
available application servers on a machine and build the response files
individually to perform the installation.

Another option is to use an automated software distribution solution such as IBM
Tivoli Configuration Manager or IBM Tivoli Provisioning Manager.

3.3 Configuring and setting up SSL communication
ITCAM for WebSphere provides the facility to communicate with data collectors
securely over TCP/IP networks using Secure Socket Layer (SSL). Default
certificates and keystores are set up during initial installation. SSL is not enabled
by default. Therefore, manual configuration is necessary in order to have the
managing server and data collectors communicate over a secure connection.
3.3.1, “Managing server Secure Socket Layer setup” on page 64
3.3.2, “Data collector Secure Socket Layer setup” on page 65
3.3.3, “Working with custom certificates” on page 67

3.3.1 Managing server Secure Socket Layer setup
Enabling SSL and using the default certificate information to verify SSL
communication works must be your first step. This is to rule out any possible
errors while attempting to generate and use your own certificates and keystores.
1. Go to the ITCAM _HOME/etc directory. The kl1.properties and kl2.properties
kernel configuration files contain the following parameters that must be
activated for Secure Sockets Layer:
security.enabled=true
codebase.security.enabled=true
2. Set the system properties on the WebSphere Application Server where the
visualization engine is running. Add the system properties using the
WebSphere administration console. Where you set the properties depends
on the version of the WebSphere Application Server.
– certificate.path: MS_home/etc/mgmttomgmt.cer
– keystore.location: MS_home/etc/CyaneaMgmtStore
– keystore.storepass: cyanea94612
– keystore.keypass: cyanea94612
– nodeauth.userid: cyaneamgmt


Figure 3-7 shows these settings in the administration console.

Figure 3-7 New SSL-related settings

3. Restart the kernel components and the visualization engine application server
to activate the new settings.

3.3.2 Data collector Secure Socket Layer setup
The SSL setup is stored in the datacollector.properties file. This file is located in
the etc subdirectory of the data collector installation directory. Activate the
definition shown in Example 3-24.

Example 3-24 SSL settings for datacollector.properties file
security.enabled=true
certificate.path=/opt/IBM/itcam/WebSphere/DC/etc/dctomgmt.cer;/opt/IBM/
itcam/WebSphere/DC/etc/dctoproxy.cer
keystore.location=/opt/IBM/itcam/WebSphere/DC/etc/CyaneaDCStore
keystore.storepass=oakland94612
keystore.keypass=oakland94612
nodeauth.userid=cyaneadc
comm.use.ssl.dc=true


Remove the system-generated datacollector.properties file and restart the data
collector. The system-generated property file name is similar to
<nodename>.<servername>.datacollector.properties.

On restarting the data collector, the message shown in Example 3-25 is
generated in msg-dc.log, which is located in the Tivoli common log directory.
Once you receive the message, communication between the managing server
and the data collector is encrypted.

Example 3-25 SSL active information
<Message Id="CYND5237I" Severity="INFO">
<Time Millis="1143583253402"> 2006-03-28 17:00:53.402-05:00</Time>
<Server Format="IP">wsamaix1.tivlab.raleigh.ibm.com</Server>
<ProductId>CYN</ProductId>
<Component>CYN.msg.datacollector</Component>
<ProductInstance>6</ProductInstance>
<LogText><![CDATA[CYND5237I Using SSL server socket for component:
CommandAgent. All clients connecting to this component MUST connect via
SSL, else the client socket may hang till the timeout expires. No user
action is required.]]></LogText>
<Source FileName="com.cyanea.command.CommandAgent" Method="start"/>
<TranslationInfo Type="JAVA"
Catalog="com.ibm.tivoli.probe.resource.LogMessages"
MsgKey="CYND5237I"></TranslationInfo>
<Principal>wsamaix1.tivlab.raleigh.ibm.com/9.42.153.102</Principal>
</Message>

Note: Only the CommandAgent port uses SSL. Other communications
opened by the data collector, such as the ProbeController port and the data
collector - Publish Server port, do not use SSL. Therefore, when SSL is
enabled, only the data in the channels connected to the CommandAgent port
is encrypted.


All the data processed in the CommandAgent channel is encrypted when SSL is
enabled. Table 3-4 shows the data classification.

Table 3-4 Data content classification
Classification Data

Command and control data Configuring and unconfiguring data collector

User actions related to threads Starting and stopping JVM threads
Changing thread priorities
Getting thread priorities and thread status
Requesting drill information to see cookies, and so on
Generating thread dumps
Getting thread stack traces

System information Application server information
Operating system platform information
JVM information

Application information All the applications installed on the monitored application
server
Application binaries and location information
Thread pool information related to Java Message Service
(JMS), Java 2 Enterprise Edition (J2EE) Connector
Architecture (JCA), Java Transaction API (JTA), Servlet,
Enterprise JavaBeans (EJB), and so on
Data source information

Performance data All performance monitor interface (PMI) data

Transport data Object Request Broker (ORB) data
SOAP ports

Memory information Obtaining JVM Heap Snapshot™ data
Performing memory leak analysis
Performing heap dump

3.3.3 Working with custom certificates
This section discusses one approach to using custom certificates. There are
different ways of configuring the certificates. The approach that we use
categorizes the machines into three types: the managing server, the port
consolidator, and the data collector. Each type can communicate with the other
type and understand one another.


Three keystores must be generated for custom certificates. The three keystores
for a distributed environment implementation are:
MSStore
This contains the following managing server certificates:
– mgmttomgmt.cer (cn=cyaneamgmt)
– dctomgmt.cer (cn=cyaneadc)
– proxytomgmt.cer (cn=cyaneaproxy)
DCStore
This contains the following data collector certificates:
– proxytodc.cer (cn=cyaneaproxy)
– mgmttodc.cer (cyaneamgmt)
ProxyStore
This contains the following port consolidator certificates:
– mgmttoproxy.cer (cn=cyaneamgmt)
– dctoproxy.cer (cn=cyaneadc)

The keystore files are generated using the keytool command, which comes with
Java Runtime Environment (JRE). In order to run the keytool command, you
must be in the JAVA_HOME/bin directory or have it specified in the PATH.
Perform a quick check by entering the keytool command. A help list showing the
syntax must be visible on the console.

In order to better manage the keystores, we create a directory called keyfiles in
ITCAM_HOME/etc and run the keytool command from this directory.


Creating MSStore
The first keystore that must be created is for the managing server. The name of
the keystore can be customized to fit your environment. In the examples
provided here, NEWMSStore is used as the keystore name. Because three
aliases must be added to this keystore, the command must be run three times,
once for each alias.
For communication between the managing servers, create the alias
mgmttomgmt for the keystore NEWMSStore with the command shown in
Example 3-26.

Example 3-26 Command to create the alias mgmttomgmt
keytool -genkey -alias mgmttomgmt -keyalg RSA -keysize 1024 -sigalg
MD5withRSA -validity 2000 -keypass raleigh1 -keystore ./NEWMSStore
-storepass raleigh2 -dname "cn=cyaneamgmt, OU=Tivoli, O=IBM,
L=Raleigh, ST=NC, C=US"

This command creates the NEWMSStore file and the mgmttomgmt key in it.
The password, keypass, and storepass can be anything you want them to be.
For communication from the managing server to the data collector, create the
alias dctomgmt using the command shown in Example 3-27.

Example 3-27 Command to create the alias dctomgmt
keytool -genkey -alias dctomgmt -keyalg RSA -keysize 1024 -sigalg

This command adds the dctomgmt alias into the NEWMSStore file.
For communication from the managing server to the port consolidator, create
the alias proxytomgmt to the keystore using the command shown in
Example 3-28.

Example 3-28 Command to create the alias proxytomgmt
keytool -genkey -alias proxytomgmt -keyalg RSA -keysize 1024 -sigalg


Creating DCStore
The second set of keystores that must be created is for the data collector. As
with the managing server keystore, customize the data collector keystore name
to your environment. In this example, we use NEWDCStore.
Create the alias proxytodc for keystore NEWDCStore using the command

Example 3-29 Command to create the alias proxytodc
keytool -genkey -alias proxytodc -keyalg RSA -keysize 1024 -sigalg
MD5withRSA -validity 2000 -keypass raleigh1 -keystore ./NEWDCStore
-storepass raleigh2 -dname "cn=cyaneaproxy, OU=Tivoli, O=IBM,

Create the alias mgmttodc for the NEWDCStore file using the command

Example 3-30 Command to create the alias mgmttodc
keytool -genkey -alias mgmttodc -keyalg RSA -keysize 1024 -sigalg
MD5withRSA -validity 2000 -keypass raleigh1 -keystore ./NEWDCStore
-storepass raleigh2 -dname "cn=cyaneaproxy, OU=Tivoli, O=IBM,

Creating ProxyStore
ProxyStore is the last keystore that must be created before extracting the
certificates. This is used by the port consolidator in a distributed environment.
We use the file NEWProxyStore for this purpose.
Create the alias mgmttoproxy for the keystore NEWProxyStore using the
command shown in Example 3-31.

Example 3-31 Command to create the alias mgmttoproxy
keytool -genkey -alias mgmttoproxy -keyalg RSA -keysize 1024 -sigalg
MD5withRSA -validity 2000 -keypass raleigh1 -keystore
./NEWProxyStore -storepass raleigh2 -dname "cn=cyaneaproxy,
OU=Tivoli, O=IBM, L=Raleigh, ST=NC, C=US"


Create the alias dctoproxy using the command shown in Example 3-32.

Example 3-32 Command to create the alias dctoproxy
keytool -genkey -alias dctoproxy -keyalg RSA -keysize 1024 -sigalg
MD5withRSA -validity 2000 -keypass raleigh1 -keystore
./NEWProxyStore -storepass raleigh2 -dname "cn=cyaneaproxy,
OU=Tivoli, O=IBM, L=Raleigh, ST=NC, C=US"

Extracting certificates from the keystores
Once all the three keystores are created, extract the certificates from each
keystore and give them to their communication partners. These certificates are
used to validate the SSL connection between the components. Figure 3-8 shows
the keystores and certificate exchanges.

Managing Port Data
Server Consolidator Collector

MSStore ProxyStore DCStore

mgmttomgmt mgmttoproxy mgmttodc

proxytomgmt dctoproxy proxytodc key
dctomgmt

mgmttomgmt

mgmttoproxy proxytodc dctoproxy certificates
mgmttodc proxytomgmt dctomgmt

Figure 3-8 Keystores and certificate exchanges


Do the following:
Extract all the certificates from the managing server keystore NEWMSStore.
The export commands are shown in Example 3-33.

Example 3-33 Commands to extract certificates from the managing server keystore
keytool -export -alias mgmttomgmt -keypass raleigh1 -keystore
./NEWMSStore -storepass raleigh2 -file mgmttomgmt.cer
keytool -export -alias dctomgmt -keypass raleigh1 -keystore
./NEWMSStore -storepass raleigh 2 -file dctomgmt.cer
keytool -export -alias proxytomgmt -keypass raleigh1 -keystore
./NEWMSStore -storepass raleigh2 -file proxytomgmt.cer

Extract all the certificates from NEWDCStore by using the commands shown
in Example 3-34.

Example 3-34 Commands to extract all the certificates from NEWDCStore
keytool -export -alias proxytodc -keypass raleigh1 -keystore
./NEWDCStore -storepass raleigh2 -file proxytodc.cer
keytool -export -alias mgmttodc -keypass raleigh1 -keystore
./NEWDCStore -storepass raleigh2 -file mgmttodc.cer

Extract all the certificates from NEWProxyStore by using the commands

Example 3-35 Commands to extract all the certificates from NEWProxyStore
keytool -export -alias mgmttoproxy -keypass raleigh1 -keystore
./NEWProxyStore -storepass raleigh2 -file mgmttoproxy.cer
keytool -export -alias dctoproxy -keypass raleigh1 -keystore
./NEWProxyStore -storepass raleigh2 -file dctoproxy.cer

After completing the extraction of all the certificates, the directory must have
seven .cer files. Import these certificates to their appropriate partners using these
commands:
For NEWMSStore, use the commands shown in Example 3-36.

Example 3-36 Commands to import certificates: NEWMSStore
keytool -import -alias mgmttomgmt -file mgmttomgmt.cer -keypass
raleigh1 -keystore ./NEWMSStore -storepass raleigh2
keytool -import -alias mgmttoproxy -file mgmttoproxy.cer -keypass
raleigh1 -keystore ./NEWMSStore -storepass raleigh2
keytool -import -alias mgmttodc -file mgmttodc.cer -keypass raleigh1
-keystore ./NEWMSStore -storepass raleigh2


For NEWDCStore, use the commands shown in Example 3-37.

Example 3-37 Commands to import certificates: NEWDCStore
keytool -import -alias dctomgmt -file dctomgmt.cer -keypass raleigh1
-keystore ./NEWDCStore -storepass raleigh2
keytool -import -alias dctoproxy -file dctoproxy.cer -keypass
raleigh1 -keystore ./NEWDCStore -storepass raleigh2

For NEWProxyStore, use the commands shown in Example 3-38.

Example 3-38 Commands to import certificates: NEWProxyStore
keytool -import -alias proxytomgmt -file proxytomgmt.cer -keypass
raleigh1 -keystore ./NEWProxyStore -storepass raleigh2
keytool -import -alias proxytodc -file proxytodc.cer -keypass
raleigh1 -keystore ./NEWProxyStore -storepass raleigh2

Deploy these newly generated keystores to the appropriate machines. Store
NEWMSStore on the managing servers, NEWProxyStore on the port
consolidator machines, and NEWDCStore on the data collector machines. Copy
the certificate files to the respective machines, too.

Update the appropriate configuration entries specified in 3.3.1, “Managing server
Secure Socket Layer setup” on page 64, and 3.3.2, “Data collector Secure
Socket Layer setup” on page 65. Restart all the components to activate the SSL
with the custom certificates.


4

Chapter 4. Maintenance of ITCAM for
WebSphere
This chapter discusses the issues relating to maintenance of ITCAM for
WebSphere in a large-scale environment. This chapter discusses the following
topics:
4.2, “Performance and availability of the managing server” on page 76
4.2, “Performance and availability of the managing server” on page 76
4.3, “Backup and recovery configuration” on page 83
4.4, “Log files and configuration files” on page 85
4.5, “Performing product maintenance” on page 87


4.1 Operating ITCAM for WebSphere
This chapter discusses some operational issues regarding a large ITCAM for
WebSphere environment. The focus is on the issues that arise largely because
of environment size. Following is a list of these issues along with details about
the sections that describe them:
Section 4.2, “Performance and availability of the managing server” on
page 76, shows that as the size of the environment grows, even a slight
inefficiency can cause a major problem. A large-scale environment requires
the performance and availability of the managing server to be soundly
maintained.
Section 4.3, “Backup and recovery configuration” on page 83, shows that
backup and recovery of configuration is a necessity. The ability to quickly
identify and recover system configuration is critical. The large size of the
environment prohibits ad hoc backup and recovery.
Section 4.4, “Log files and configuration files” on page 85, shows that log and
configuration files must be maintained. There are two aspects of file
maintenance: keeping the size of the log files under control and ensuring that
the correct property files are used.
Section 4.5, “Performing product maintenance” on page 87, discusses
implementation of WebSphere, Database 2 (DB2), and ITCAM for
WebSphere patches. Typically, with the number of data collectors involved,
this can be a huge task.

4.2 Performance and availability of the managing server
Performance and availability of the ITCAM for WebSphere managing server are
mainly related to the IBM WebSphere Application Server that hosts the
visualization engine and the OCTIGATE database. This section discusses the
following topics:
4.2.1, “Performance of the WebSphere Application Server” on page 76
4.2.2, “Database maintenance” on page 77
4.2.3, “Data trimming” on page 78

4.2.1 Performance of the WebSphere Application Server
There are several critical tuning factors for the visualization engine. The sole use
of the visualization engine is to provide users with information regarding the
enterprise activity, not collecting and analyzing the activity. This eases the role of
the visualization engine to not being completely critical.


Tuning of the visualization engine is mainly to ensure that the WebSphere
Application Server has adequate memory. This also relates to the fact that the
visualization engine has to run on one of the managing servers that runs other
Java-based kernel components.

Each logged-in user and the activity of each user requires additional storage to
be acquired by the visualization engine. The Java garbage collection process
also plays an integral part in providing adequate memory resource.

4.2.2 Database maintenance
Database performance is largely related to how the data is stored physically and
how well the database manager understands the nature of the data. This section
primarily discusses the use of DB2 as the database manager.

The primary tools to ensure this are the REORG and RUNSTAT tools. You can
run both of these utilities from the DB2 utilities from the DB2 command line.
The REORGCHK utility reads the table information and analyzes the
necessity of running reorg. REORGCHK uses the current statistics or
generates its own statistics.
The REORG utility reorganizes the database structure to ensure that data is
allocated coherently and in a typical access sequence requested by most
queries. The syntax for running REORG is:
DB2 REORG TABLE creator.table_name INDEX primary_index_name
The RUNSTATS utility collects the statistics of the tables and indexes to
ensure that the database manager chooses the most optimized method to get
the data requested by an Structured Query Language (SQL) request. The
syntax for running RUNSTATS is:
DB2 RUNSTATS ON TABLE creator.table_name WITH <runstat_options>

Typically, you must run REORGCHK for frequently modified tables and then
determine whether you want to run REORG on those tables. After all the
modifications have been performed, run RUNSTATS to ensure that the statistics
are current and that DB2 obtains the most optimal path to the data an Structured
Query Language (SQL) requested.

Chapter 4. Maintenance of ITCAM for WebSphere 77

ITCAM for WebSphere supplies a script to run the RUNSTATS utility. This script
is called run-stat-cmds and resides in the bin directory. This command selects
specific tables that are necessary to perform RUNSTATS on. The syntax of this
command depends on the platform you execute. It can be either one of the
following:
run-stat-cmds.bat OCTIGATE amuser
export JDBC_USER=amuser
export JDBC_PASSWORD=ampasswd
run-stat-cmds.sh OCTIGATE

The table that ITCAM for WebSphere constantly modifies is the REQUEST table.
This table must be checked frequently. The run-stat-cmds automatically checks
this table and stores its output in the logs/reorg.log file. If the cluster ratio of the
REQUEST table is lower than 90, run REORG on this table and rerun
RUNSTATS on the REQUEST table. This REORG command is as follows:
db2 "REORG TABLE AMUSER.REQUEST INDEX R_TIME "

We recommend a regular, maybe monthly, maintenance window to REORG all
the tables for ITCAM for WebSphere and perform a complete RUNSTATS.

4.2.3 Data trimming
Another aspect of database performance is processing the data size. Because
monitoring information is accumulated, access to current information may be
degraded, as the database has to process more information than it must. Old
data must be trimmed to allow more efficient processing. This process is
application-specific, as the database manager does not know the relationship
between the data. This also minimizes the disk space usage for ITCAM for
WebSphere.

There are two methods of pruning the tables. Run both of them, as they address
different sets of tables. The pruning methods are:
Automatic pruning by the archive agent
This prunes data every two hours and keeps data for 48 hours. See “Archive
agent trimming” on page 79.
Manual pruning using the datatrim.sh command
We recommend that you run this regularly. A daily run to minimize the amount
of data to be pruned may be appropriate. See “Running datatrim.sh” on
page 80.


Note: You may see a datatrimmer.sh script in the <MS_HOME>/bin
directory. This script is not recommended and is not discussed in this
paper.

Archive agent trimming
Once enabled, the managing server archive agent automatically prunes the
following tables:
MEMORY_DATA
GC_DATA
MQI_QUEUEMGR_SR_OVERVIEW
MQI_QUEUE_SR_OVERVIEW
CTG_SR_OVERVIEW
WEBSERVERCHARTDATA
IMSTHREADS
VOLUMESTAT
PORTALSTATS

When data trimming is enabled, pruning occurs every two hours, keeping 48
hours worth of data in each table. These tables, and the criteria for trimming the
tables, are hard-coded in ITCAM for WebSphere, and cannot be changed.

Automatic data trimming using the archive agent is enabled by setting the
ENABLE_DATATRIMMER property to true in the etc/aa1.properties file or the
etc/aa2.properties file, or both. If both the properties files have data trimming
enabled, there is a built-in delay for the second archive agent in order to prevent
the occurrence of database deadlocks.

Note: If you are using more than two archive agents, do not enable the data
trimmer for the other archive agents. Only aa1 and aa2 must have the
ENABLE_DATATRIMMER=true property.

Stop and start the archive agents to pick up the changes made to the
ENABLE_DATATRIMMER property using the following commands:
<MS_HOME>/bin/aactl.sh stop aa1
<MS_HOME>/bin/aactl.sh start aa2


Running datatrim.sh
The datatrim.sh script is a command-line utility that can be scheduled or run
manually to trim data that is no longer required from the database. As it is
shipped with ITCAM for WebSphere, this script prunes data from the following
tables:
REQUEST
METHOD
SERVERSTATS
PMISTATS
VOLUMESTAT

APAR PK17797 allows additional tables to be trimmed, including the following:
MEMORY_DATA
IMSEVENTS

The datatrim.sh script carries out two major processes, marking and trimming:
Mark deleting process.
This process updates the rows for the REQUEST and METHOD tables that
will be deleted in the data trim process. This method is controlled by the
etc/deleterelatedtables.xml control file shown in Example 4-1.

Example 4-1 Deleterelatedtables.xml control file
<tableinfo>

<commitcount>2000</commitcount>
<useoracle>false</useoracle>
<table>
<name>request</name>
<daystokeep>7</daystokeep>

<columnname>end_time</columnname>

<startdate></startdate>
<enddate></enddate>

<adminserver></adminserver>
<appserver></appserver>
<idcolumn>request_id</idcolumn>
<delayinterval>15</delayinterval>
<relatedtables>
<name>method</name>

<columnname>start_time</columnname>
</relatedtables>
</table>
</tableinfo>

Data trim process
This process deletes the data marked in the REQUEST and METHOD tables
as a result of the marking process. This process is controlled by the
etc/deletesingletable.xml control file shown in Example 4-2.

Example 4-2 Deletesingletable.xml control file
<tableinfo>
<commitcount>2000</commitcount>
<useoracle>true</useoracle>
<table>
<name>serverstats</name>
<columnname>current_time</columnname>
<enddate></enddate>
<serverid>probe_id</serverid>
</table>
<table>
<name>pmistats</name>
<columnname>current_time</columnname>
<enddate></enddate>


<serverid>probe_id</serverid>
</table>
</tableinfo>

For both the eXtensible Markup Language (XML) files in Example 4-1 on
page 80 and Example 4-2 on page 81, the following apply:
The parameter commitcount controls the number of records to be committed
to a database in a transaction. Change the commitcount setting from a default
value of 2000 to something higher if you have large numbers of records to be
deleted. This speeds up the trimming process. The commitcount number
applies to all the tables specified in the XML files.
The useoracle setting is no longer used. Instead, the datatrim.sh script uses
the Java Database Connectivity (JDBC) driver to determine the type of
database (DB2 or Oracle) being used.
For each individual table, set either daystokeep or the startdate and enddate
pair settings. If daystokeep is set, it overrides the startdate and enddate
values.

To invoke datatrim.sh, log in using the ITCAM for WebSphere owner, typically
called amuser. Run the utility from the managing server. It will write its output in
the log/datatrim.log file. The syntax of the command is:
$MS_HOME/bin/datatrim.sh octigate amuser ampasswd

The result and status of the datatrim.sh command can be analyzed in the
logs/datatrim.log file.

The datatrim.sh command prunes the tables, but does not optimize or reclaim
free space. Perform these actions by using the REORG utility in DB2 or by
exporting and reimporting the tables in Oracle.

We recommend that you change the daystokeep value to the number of days of
data that you want to keep, and then run the datatrim.sh script every night.
Running the script every night facilitates the removal of fewer records from the
database more often. Running the command less frequently can result in the
script running for extended periods of time.


While it is not necessary to stop the archive agents (<MS_HOME>/bin/aactl.sh
stop) prior to running the datatrim.sh script, stop the archive agents to avoid
database deadlocks.

Note: Stopping the archive agents leads to loss of performance data.

4.3 Backup and recovery configuration
The backup and recovery procedure covers both WebSphere backup and
database backup. The aim of a backup policy is to quickly restore the operation.
Another objective is to minimize the backup’s impact on the operation. This
section discusses the following topics:
4.3.1, “ITCAM for WebSphere backup” on page 83
4.3.2, “WebSphere configuration backup” on page 83
4.3.3, “Database backup and restore” on page 84

4.3.1 ITCAM for WebSphere backup
Back up the installation directory structure of ITCAM for WebSphere using any
backup utility. The restore of the file system of ITCAM for WebSphere must be
owned by the ITCAM for WebSphere user ID, which is typically amuser.

4.3.2 WebSphere configuration backup
WebSphere configuration backup must be considered for both the managing
server and the data collector. Depending on whether you want to reinstall or
restore the backup, initiate a backup. As WebSphere configuration is not a
dynamic entity, taking a backup of the initial implementation and a backup
procedure to be implemented before and after a configuration change must be
sufficient.

To dump the WebSphere Application Server configuration, use the backupConfig
command. The syntax for the backupConfig command is:
backupConfig <file> -nostop -logfile <filename> -user <name> -password
<pswd>

To restore the WebSphere configuration, use the restoreConfig command. The
syntax for restoreConfig command is:
restoreConfig <file> -nostop -logfile <filename> -user <name> -password
<pswd>


4.3.3 Database backup and restore
For backing up the DB2 UDB, use the DB2 UDB backup utility that is launched
by using the DB2 command-line processor. The syntax for the DB2 backup utility
is:
db2 backup database <dbname> user <username> using <password> to <file>

Note: A directory path for the target file must exist. Manually create the path
before executing the backup command. If not, you will get an exception similar
to:
SQL2036N. The path for the file or device "/tmp/octigate" is not
valid.

A successful backup returns a message that is similar to:
Backup successful. The timestamp for this backup image is :
20041116104855

You must either keep the timestamp or rename the file according to the
timestamp. The timestamp is required to restore the database.

To restore the database, execute the following command:
db2 restore database <dbname> user <user> using <password> from <file>
taken at <timestamp> <option>

An example of running the restore with an option to prohibit rolling forward is
shown in Example 4-3. Rolling forward is a feature that allows the post-backup
changes to the database to be applied from the transaction redo log. A typical
distributed DB2 installation does not activate this feature.

Example 4-3 Restoring DB2 database
> db2 restore database octigate user itcamusr using passwd from
/tmp/octigate taken at 20041116104855 without rolling forward
SQL2539W Warning! Restoring to an existing database that is the same
as the backup image database. The database files will be deleted. Do
you want to continue ? (y/n) y
DB20000I The RESTORE DATABASE command completed successfully.


4.4 Log files and configuration files
Perform the file maintenance process regularly to reduce the amount of disk
space used and to manage monitoring configuration. Log files maintenance
applies to both managing servers and data collectors, with configuration file
maintenance applying more to data collectors. This section discusses the
following topics:
4.4.1, “Managing log files” on page 85
4.4.2, “Managing the configuration files” on page 86

4.4.1 Managing log files
Log files in ITCAM for WebSphere are used for problem determination of the
product’s working mechanism. There are several locations for log files,
depending on the components that generate them. These locations are:
Tivoli common directory
This is where most of the log files are stored. Some typical directory names
are:
– %ProgramFiles%ibmtivolicommon (Windows)
– /var/ibm/tivoli/common (UNIX)
The directory under the common directory is typically identified by a
three-character product code such as CYN for ITCAM for WebSphere. In the
product directory, there are sub-directories:
– First-failure data capture (FFDC) for first-failure data capture application
The FFDC directory contains diagnostic information pertaining to the
program exceptions that are captured.
– Logs subdirectory of Tivoli common directory
The logs subdirectory is typically filled with log files and must be
maintained.
The logs subdirectory of the installation path
These are the logs that are typically initiated either before the product is
active, such as standard output and standard error redirection, or from a
leftover log file from a previous release.
Some utility commands such as datatrim.sh also use this directory to store
their output.

Because the log files in the common directory have a maximum size and
predefined number of generations, they cannot exceed a certain size. This path
must be maintained and checked for ad hoc logging that may use up the space.


We recommend that you use a separate file system to contain the common
directory.

The contents of the logs subdirectory of the installation path must be regularly
cleaned up manually to prune the content.

4.4.2 Managing the configuration files
Figure 4-1 shows the configuration files for data collectors inter-relation. In
Version 6.1, there is a significant change in the structure of the configuration files.
This is due to the changes of the byte code instrumentation mechanism.

$DC_home custom_requests.xml
lock_analysis.xml
memory_leak_diagnosis.xml
method_entry_exit.xml
toolkit_config_ctg.xml
itcamdc/etc
toolkit_config_ims.xml
toolkit_config_jdo.xml
toolkit_config_kwj.xml
runtime toolkit_config_mqi.xml
toolkit_config_ra.xml

<wasver>.<wasnode>,<server>

custom/toolkit_custom.properties
cyn-cclog.properties
cynlogging.properties
jiti.properties
*.datacollector.properties
*.datacollector.policy
*.kwjdc.properties
*.toolkit.properties
*.toolkit.xml

Figure 4-1 Configuration file structure

From time to time, you may have to modify some XML configuration files to
enable certain product features. There is currently no mechanism in the product
to make this switch. Employ an external automation mechanism to switch these
configurations back and forth.

Configuration change is typically performed by changing the
toolkit_custom.properties and several related xml files under the itcamdc/etc
directory. This is to add or remove the classes to be used for testing with memory
allocation probe or locking probe.


4.5 Performing product maintenance
Various considerations must be taken into account when dealing with the
application of product maintenance, fixes, and fix packs. In a large environment
this must be performed remotely and automated with minimal impact to the
production system. This section discusses the following topics:
4.5.1, “Getting software updates” on page 87
4.5.2, “Updating ITCAM for WebSphere managing server” on page 87
4.5.3, “Updating ITCAM for WebSphere data collectors” on page 90

4.5.1 Getting software updates
Software updates are available at the following IBM Web sites:
For ITCAM for WebSphere:
http://www-306.ibm.com/software/sysmgmt/products/support/IBMTivoli
CompositeApplicationManagerforWebSphere.html
For DB2 Universal Database:
http://www-1.ibm.com/support/docview.wss?rs=71&uid=swg27007053
For WebSphere Application Server:

Typically, you do not have to get updates for DB2 or WebSphere on the
managing server. ITCAM for WebSphere only supports a certain level of these
products. However, you may sometimes have to fix an impending problem with a
certain software level.

4.5.2 Updating ITCAM for WebSphere managing server
The ITCAM for WebSphere managing server updates must be performed as a
scheduled maintenance. This means that:
You must have a backup of the management server system.
You must perform an orderly shutdown of the management server.
Users must be notified of the system’s unavailability.

The managing server updates for ITCAM for WebSphere software are typically
performed using a silent installer. The ITCAM for WebSphere fix packs consist of
two sets of files that are distributed as tar files:
6.1.0-TIV-ITCAMfWAS_SVR-FP000n.tar
ITCAMfWAS_V6_UpdateInstaller.tar


Fix packs and interim fixes are distributed as tar files, which are installed using
the silent command-line update installer. Multiple fixes may be applied by
combing all *.update files found in the <fix-dir>/update directory to a common
update directory.

The procedure to apply the updates is derived from the readme for the update
installer V6.1.0.pdf.
1. Modify the silentUpdate.properties file to reflect your environment.
Example 4-4 illustrates the changes required for our managing server,
srv152.

Example 4-4 The silentUpdate.properties
# product location
product.location=/opt/IBM/itcam/WebSphere/MS
# updates location
updates.location=./updates
#######################################################################
# Managing Server properties
# These properties apply only for the MS update installation,
#######################################################################
# change to false if database should not be updated
updateDb=true
# change to false if VisualEngine should not be updated
updateVe=true
# TODO: uncomment this property and specify correct WAS_HOME
updateVe.wasHome=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01
updateVe.was.soap.host=srv152.itsc.austin.ibm.com
updateVe.was.soap.port=8880
updateVe.was.node=srv152Node01
updateVe.was.server=server1
updateVe.was.user=root
updateVe.was.password=rpasswd
#ms.oracle.driver.jar=${ORACLE_JDBC_DRIVER_PATH}/ojdbc14.jar
#ms.oracle.driver.path=
# Set this property to reconfigure MS processes to bind to the
specified address.
#ms.bind.host.name=172.17.11.82
#######################################################################
# Uninstall properties
#######################################################################
# the updates to be uninstalled
uninstall.updates=last


2. Ensure that the following environment is set prior to running the silent
updater:
export JAVA_HOME=/opt/IBM/WebSphere/AppServer/java
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=JAVA_HOME:$CLASSPATH
3. Prepare the installation by executing:
./silentUpdate.sh -prepareInstall
4. Do not proceed with the installation. Clean the results of the previous
command by executing:
./silentUpdate.sh -cleanPrepared
5. Install by executing:
./silentUpdate.sh -install
6. If the install failed, rollback is achieved by:
./silentUpdate.sh -rollback
7. Display updates that have been installed by executing:
./silentUpdate.sh -displayInstalledUpdates

Example 4-5 shows an example of the output after running the command.

Example 4-5 Output from silentUpdate.sh -displayInstalledUpdates
# ./silentUpdate.sh -displayInstalledUpdates
Update installer version 6.1.0, build 20070513145421
Logging details into
/opt/IBM/itcam/WebSphere/MS/logs/update/update_20070616101145.log
Action: display installed updates
Updates installed 5/31/07 2:51 PM
6.1.0.1.1 Interim Fix 1 after Fix Pack 1
6.1.0.1.3 Interim Fix 3
Updates installed 5/31/07 11:56 AM
6.1.0.1 Fix Pack 1
Finished successfully


4.5.3 Updating ITCAM for WebSphere data collectors
Data collectors are typically run in a production environment. Updating these
machines requires strict change control. Because data collectors run as an
integral part of the WebSphere Application Server, updating the data collectors
typically involves restarting the application server. When dealing with a clustered
environment, alternately restarting the members of a cluster does not affect
application availability, while a standalone application server means that the
application is unavailable.

The primary prerequisite of this fix pack is generally to have the managing server
updated before installing the fix pack to the data collectors. Installation of data
collectors can be performed in stages. All the data collectors do not have to be
updated together.

As with the managing server updates, the data collector updates, too, come as
two tar files:
6.1.0-TIV-ITCAMfWAS_MP-FP00nn.tar.
ITCAMfWAS_V6_UpdateInstaller.tar

The following discussion is derived from our experience in installing fix pack 1 of
ITCAM for WebSphere V6.1. This fix pack is distributed as
6.1.0-TIV-ITCAMfWAS_MP-FP0001.

Note: Use the following steps following the DataCollector Fixpack1 update or
after multiple fix updates that includes Fixpack 1 -
6.1.0.1-TIV-ITCAMfWAS_MP-IF0001.

1. Download and untar the update installer tar file
ITCAM_V61_UpdateInstaller.tar.
2. Download and untar the fixes tar files. You may untar all available updates to
the same updates directory. Both fix packs and interem fixes are may be
applied together. Example 4-6 lists the fixes that we apply.

Example 4-6 A list of update files to be applied
6.1.0-TIV-ITCAMfWAS_MP-FP0001.update
6.1.0.0-TIV-ITCAMfWAS_MP-IF0003.update


3. The update installer requires the Java Runtime Environment Version 1.3.1 or
later. Either include the java command into PATH or define the JAVA_HOME
environment variable to point to the Java Runtime Environment home
directory. In UNIX or Linux, run:
export JAVA_HOME=/opt/WebSphere/AppServer/java
4. The update installer is a non-interactive silent installer that works based on
the response file. The default name of the response file is
silentUpdate.properties, and it is supplied with the update installer. Update
the response file, as in Example 4-7.

Example 4-7 The silentUpdate.properties for applying fixes to the data collector
product.location=/opt/IBM/itcam/WebSphere/DC
updates.location=./updates
updates.gclog=/tmp/gclog.log
uninstall.updates=last

5. Stop the WebSphere Application Servers that have the data collector
installed.
6. In order to prepare for installation, change to the directory where the update
installer scripts are stored. The preparation is recommended for checking the
environment. Run the command ./silentUpdate.sh -prepareInstall. A
typical output of the prepareInstall is shown in Example 4-8.

Example 4-8 The prepare for installation
# ./silentUpdate.sh -prepareInstall
/opt/IBM/itcam/WebSphere/DC/logs/update/update_20070608170706.log
Action: prepare install

7. To update the data collector with all the update files in the updates
sub-directory, run the command ./silentUpdate.sh -install. A typical
output of the install command is shown in Example 4-9.

Example 4-9 Running the patch installation
# ./silentUpdate.sh -install
. . .


8. When the installation has completed, you can verify the data collector version
using the command. ./silentUpdate.sh -displayInstalledUpdates. The
status in our environment is shown in Example 4-10.

Example 4-10 Patch installation result
# ./silentUpdate.sh -displayInstalledUpdates
Action: display installed updates
6.1.0.1 Fix Pack 1
6.1.0.0.4 High CPU usage for TEMA PMI collection
6.1.0.0.3 JITI support

Note: At least one server must first be configured before updating the
version level of the data collecter.

9. In a WebSphere Application Server V6.1 environment, activate the OSGI
configuration initialization. From the WebSphere /profiles/<profile-name>/bin
directory, run the osgiCfgInit.sh script. You should expect no messages
from this script.
10.Restart the application server to activate the data collectors.

Some patches may have to be updated manually through the WebSphere
administration console. This is typically to add certain options on WebSphere. To
perform the updates silently, build a Java Command Language (JACL) script.


5

Chapter 5. Planning for ITCAM for Response
Time Tracking

This chapter discusses areas that must be considered during the planning phase
of IBM Tivoli Composite Application Manager implementation in a large
environment. The topics discussed in this chapter are:
5.1, “Planning considerations” on page 94
5.2, “Product architecture” on page 94
5.3, “Sizing the servers” on page 96
5.4, “Deployment of ITCAM for Response Time Tracking” on page 98
5.5, “Communication and security considerations” on page 100
5.6, “Reliability and high availability” on page 102


5.1 Planning considerations
As discussed in 1.2.2, “Concerns and considerations” on page 7, the following
aspects of a large-scale implementation are discussed in this section, too:
Understanding the product architecture
This allows you to make the correct choices. Section 5.2, “Product
architecture” on page 94, describes the architecture for ITCAM for Response
Time Tracking.
Sizing the servers
This is important to correctly acquire adequate servers and choose a sound
software configuration option. Section 5.3, “Sizing the servers” on page 96,
describes an approach that can be used.
Understanding the servers’ configuration options and agent deployment
This is discussed for ITCAM for Response Time Tracking in 5.4, “Deployment
of ITCAM for Response Time Tracking” on page 98.
Planning for communication security
This is a mandatory step for an enterprise with business-critical and sensitive
information in a transaction environment. Section 5.5, “Communication and
security considerations” on page 100, discusses confidentiality and firewall
requirements.
Reliability discussion, failover, and disaster recovery
These, too, are mandatory aspects of a business-critical process in a large
enterprise. Section 5.6, “Reliability and high availability” on page 102,
discusses this issue.

5.2 Product architecture
This section discusses the product architecture of ITCAM for Response Time
Tracking. This understanding is critical for planning and deciding about server
configuration and other implementation issues. See also IBM Tivoli Composite
Application Manager V6.1 Family Installation, Configuration, and Basic Usage,
SG24-7151.

ITCAM for Response Time Tracking V6.0 has evolved from IBM Tivoli Monitoring
for Transaction Performance V5.3 and inherits the major components and
functions of IBM Tivoli Monitoring for Transaction Performance V5.3.


Figure 5-1 shows the components of ITCAM for Response Time Tracking.

RTT RTT RTT RTT
Management Store and Store and management
server forward agent forward agent agent

FIREWALL

FIREWALL
RTT RTT RTT RTT
Tivoli management management management management
Enterprise agent agent agent agent
Monitoring
Agent

Figure 5-1 Components of ITCAM for Response Time Tracking

Basically, ITCAM for Response Time Tracking is controlled from the
management server, which provides a centralized repository of policy,
configuration, and data for the ITCAM for Response Time Tracking environment.

The rest of ITCAM for Response Time Tracking is the management agent. The
management agent performs response time data collection and performance
functions on behalf of the management server. The management agent is a
single agent that can have different components deployed on it to perform
different functions.

The management server and management agent typically operate in an
unrestricted port environment. When there is a firewall between them, they
restrict the port usage to communicate. The firewall typically requests that they
use a single communication port to talk back and forth. This is where the Store
and Forward agent comes in. It bundles the communication between the
management server and the management agent to use a single port to pass
through the firewall. Because the Store and Forward agent can be cascaded, in
this sense, there can be a chain of Store and Forward agents passing through
multiple layers of firewalls.

Chapter 5. Planning for ITCAM for Response Time Tracking 95

A special management agent resides on z/OS machines. The management
agent on z/OS machines has the transaction server component activated to
receive performance information from the Customer Information Control System
(CICS) and Information Management System (IMS) data collector.

In ITCAM for Response Time Tracking V6.0, information from the management
server can be forwarded to the Tivoli Enterprise Monitoring Server for display in
the Tivoli Enterprise Portal. This is achieved using the Tivoli Enterprise
Monitoring Agent for ITCAM for Response Time Tracking.

5.3 Sizing the servers
Due to the large scale of the implementation, the size of the servers assumes
importance. Sizing determines the hardware configuration and implementation
consideration of the servers. This section discusses the following topics:
5.3.1, “Sizing parameters” on page 96
5.3.2, “Sizing estimation for ITCAM for Response Time Tracking” on page 97

5.3.1 Sizing parameters
To evaluate the size of the management server, you must understand the data
collection mechanism for ITCAM for Response Time Tracking. Overall, the load
for the management server in a steady state environment mainly relates to the
collection of response time information and serving the Web console for
operators. Additional work may be required to perform the following functions:
Running a discovery policy
Performing administrative tasks from the Web console or the command-line
interface (CLI)
Filing the uploading activities from the Synthetic Transaction Investigator or
the Rational Robot
Fulfilling an ad hoc request to reload data from the Web console

The other additional tasks are not addressed in this Redpaper as these are not
significant in a steady-state production environment.

Reporting load parameters
Reporting load is mainly local access to Database 2 (DB2) Universal Database.
This does not have a significant network overhead. The processing overhead is
the processor requirement to serve the users and DB2 buffering concerns.


This load is typically proportional to the number of operators with direct access to
ITCAM for Response Time Tracking. However, in ITCAM for Response Time
Tracking V6.0, the majority of view-only access for response time information
can be consolidated into the Tivoli Enterprise Portal, which allows a single Tivoli
Enterprise Monitoring Agent to collect information from the ITCAM for Response
Time Tracking management server.

With the Tivoli Enterprise Portal solution in place, typical Web console users
must be limited to administrative staff, because they can be easily contained.

Response time measurement
The response time information is recorded by the Application Response
Measurement (ARM) agent process from various components that perform
monitoring policies, such as listening and playback. This information is uploaded
to the management server using the eXtensible Markup Language (XML) data
structure in a predefined and scheduled interval. The size of the data transferred
is dependent on the number of response time data collected:
For playback policies, you have a fixed amount of response time information
based on the playback occurences on the time interval.
For listening policies, the response time depends on the number of
transactions actually invoked, the sampling rate, and the number of monitored
components invoked by the transactions.

5.3.2 Sizing estimation for ITCAM for Response Time Tracking
Specific to ITCAM for Response Time Tracking, consider the following
parameters for sizing:
“Communication bandwidth” on page 97
“Management server processing” on page 98
“Database size” on page 98

Communication between the management server and the management agent
primarily consists of:
Initial loading of policies and schedules
This happens when the management agent is started. The size depends on
how many policies are deployed to the management agent.
Data retrieval for response time data
Data retrieval is typically performed every hour. You can also request an ad
hoc retrieval from the Web console when data is retrieved.


Additional actions such as deploying a monitoring component or assigning a
new monitoring policy to an agent do not contribute to a large workload in a
production environment.

Management server processing
The management server is constructed in a single, large enterprise application.
The processing of the management server application is affected by the following
parameters:
The number of connected management agents
The size of data retrieved for each interval
The number of connected Web console users

Database size
The size of the database depends on the amount of response time information
loaded from the management agents. The response time information is collected
from policies such as:
Discovery policies
Listening policies
Playback policies

5.4 Deployment of ITCAM for Response Time Tracking
The considerations for installing and configuring ITCAM for Response Time
Tracking are covered in this section. The topics discussed here are:
5.4.1, “Designing the management server” on page 98
5.4.2, “Deploying the management agent” on page 99

5.4.1 Designing the management server
The ITCAM for Response Time Tracking management server consists of the
following products:
DB2 Universal Database Enterprise Server or Oracle database server
ITCAM for Response Time Tracking management server application

The ITCAM for Response Time Tracking management server is a single contact
point for all the management agents and the user Web console. This in turn
leads to the ITCAM for Response Time Tracking management server being a
single point of failure for response time monitoring.


In a large environment, we recommend that you separate the management
server into a multiple-server implementation in a clustered environment.
Figure 5-2 shows a clustered environment.

WebSphere Node
Deployment Cluster

WebSphere
Application Server

WebSphere Edge
Server DB2

WebSphere
Application Server

Figure 5-2 Clustered environment

The environment shown in Figure 5-2 introduces the following:
Load balancing using the WebSphere Edge Server for access to the
management servers
Separation of load for database access, as the database server is now a
separate, shared machine
Better reliability because if one of the management servers fails, the other
must still be able to process management request

5.4.2 Deploying the management agent
Management agent deployment for a large environment requires the following
planning:
Naming convention
Each agent must be easily identifiable by using a program to understand its
role.
Silent installation
Automated, non-interactive installation is a better option to deploy the agent
to hundreds or thousands of agents.


Communication requirement
Each agent must communicate to the management server. If there is a
firewall, create a Store and Forward agent to minimize the holes in the firewall
rule.
Mass automated installation
Perform this by using a software distribution or provisioning solution such as
IBM Tivoli Configuration Manager.

Perform management agent configuration and component deployment by using
a command-line interface. Scripting the deployment facilitates efficient,
automated, consistent, and a less error-prone deployment.

5.5 Communication and security considerations
Communication and security issues are vital in the inter-networked world we live
in. Applications and their management infrastructure must be secured to protect
resources from unauthorized sources. This section discusses some of the
planning considerations pertinent to this area:
5.5.1, “Communication security” on page 100
5.5.2, “Firewall and port considerations” on page 101

5.5.1 Communication security
Communication security relates to the confidentiality of the information
transmitted over the network. Management information that is used by IBM Tivoli
Composite Application Manager products may contain details about the
application processing internals. This requires that the content of the
management information be secured from unauthorized access.

WebSphere security
WebSphere security plays a significant role in a large-scale implementation. In
some cases, WebSphere security is not enabled during the test phase of an
implementation, but during the production environment. This requires additional
considerations such as the WebSphere user having the appropriate permissions
to, for example, issue wsadmin commands.

The configuration of data collectors involves the use of Java Command
Language (JACL) scripts, and can fail when there is a permission or rights
problem.


If any of the application servers on which the data collectors are installed has
WebSphere security enabled, the entire ITCAM for WebSphere environment
must have it enabled as well. This includes WebSphere security being enabled
on the ITCAM for WebSphere managing server.

Secure Sockets Layer communication
Secure communication between the managing server and the data collector is a
viable option if there is a requirement for data to be encrypted during
transmission. Using Secure Sockets Layer (SSL) facilitates secure data
transmission from the data collector to the managing server, and must appease
corporate security requirements, if necessary. Additional configuration must be
carried out on the managing server and the data collector when enabling the SSL
setup. A certificate key generator is included with the product. This key generator
provides the facility to use custom-generated keys.

A best practice is to complete the default installation of the managing server and
the data collector and then enable SSL for both. This isolates problems (that is,
whether the problem is caused by the basic installation or the SSL configuration).

5.5.2 Firewall and port considerations
Firewall and port issues can arise when the data collectors are on a different site,
location, or subnet from the managing server. Problems such as name resolution
can occur if the Domain Name System (DNS) is not set up correctly on either the
managing server or the data collector. Routing problems may occur if the IP
addresses used belong to different subnets. The entire network environment
must be looked at to determine where a firewall, router, or bridge can exist.

Figure 5-3 shows the communication port requirements for ITCAM for Response
Time Tracking.

Store and Store and
HTTP port Forward SnF port Forward SnF port
server agent
Agent Agent

Figure 5-3 Communication port requirements

The management server running WebSphere Application Server listens at the
listening port for a Hypertext Transfer Protocol (HTTP) request and for a Simple
Object Access Protocol (SOAP) request. The management agent talks either
directly to these ports or in a consolidated manner through the Store and
Forward agent. The Store and Forward agent acts as the proxy agent to


consolidate communication from the management agent to the management
server.

The Store and Forward agent can be cascaded to facilitate communication
through multiple firewalls. This facility is provided, as some agents may be
located across the Internet to monitor the response time from a remote agent.

5.6 Reliability and high availability
This section discusses reliability issues such as those relating to failover and
disaster recovery.

5.6.1 Failover and fault tolerance
Clustering implementation for the ITCAM for Response Time Tracking
management server facilitates a more reliable management of the management
servers to allow one of the servers to take care of the entire management load
when the other server fails. The clustering implementation uses WebSphere
Edge Server load balancing to ensure that both the management servers are
used.

The WebSphere Edge Server component automatically redirects traffic to the
available machine. In the ITCAM for Response Time Tracking management
server, there is no indication of the primary server and the secondary server.
They all perform an equal role in a WebSphere cluster.

5.6.2 Disaster recovery
There are three areas where backup is necessary for disaster recovery, with
regard to the IBM Tivoli Composite Application Manager server:
Database
Database backup must be performed regularly in order to collect the most
up-to-date information. Use the database utility function to perform the
backup function.
WebSphere Application Server configuration for the server and the agent or
the data collector
This must be backed up so that it can be restored in a disaster recovery
scenario.


IBM Tivoli Composite Application Manager servers
These must also be physically considered to facilitate recovery at the disaster
recovery site.


6

Chapter 6. Installing ITCAM for Response
Time Tracking

This chapter provides information about the installation and deployment
procedure for ITCAM for Response Time Tracking. This chapter discusses the
following topics:
6.1, “Clustering the management server” on page 106
6.2, “Deploying the management resources” on page 129
6.3, “Setting up Secure Sockets Layer certificates” on page 133


6.1 Clustering the management server
The clustering action of installing the ITCAM for Response Time Tracking
management server is performed using a WebSphere Network Deployment
cluster. The database must be located in a remote machine to overload the
processing and eliminate dependency on either of the machines. This database
may also run on a database 2 (DB2) cluster solution. The overall installation
requirement is described in Figure 6-1.

Edge server Management server Database server

WebSphere Edge WebSphere Application DB2 Universal Database
Component Server Node Deployment server
Load balancer ITCAM for RTT clustering
server

Figure 6-1 Installation requirement

Communication from the management agents and the Web console must take
place through the WebSphere Edge Server, which distributes the load for all the
management servers. This section discusses the following topics:
6.1.1, “Preparing the operating system” on page 106
6.1.2, “Installing the database” on page 107
6.1.3, “Installing WebSphere Application Server” on page 109
6.1.4, “Installing WebSphere Load Balancer” on page 118
6.1.5, “Installing the management server” on page 124

6.1.1 Preparing the operating system
Operating system preparation includes the following tasks:
Creating a user ID to own the ITCAM for Response Time Tracking server. We
create a user called itcamusr.
Modifying the following operating system (OS) parameters:
– Kernel parameters for Hewlett-Packard UNIX (HP-UX) or Solaris
See IBM Tivoli Composite Application Manager for Response Time
Tracking Installation and Configuration Guide, GC32-9482.
– User right assignment for Windows
The ITCAM for Response Time Tracking user requires Act as part of the
operating system and Log on as service rights.


Allocating the file systems that are required for:
– DB2 product
/usr/lpp/db2 or C:Program FilesIBMSQLLIB
– DB2 instance
/home/db2inst1 or C:DB2
– WebSphere Application Server:
/opt/IBM/WebSphere or C:Program FilesIBMWebSphere
– Tivoli shared path
/var/IBM/tivoli/common or C:Program FilesIBMtivolicommon
– Product path
/opt/IBM/itcam or C:Program FilesIBMitcam

6.1.2 Installing the database
This section discusses the preparations for using a DB2 database. We use IBM
DB2 Universal Database Enterprise Server Edition V8.2 with Fix Pack Level 5.

The DB2 database takes up space on the current installed drive. Make that sure
you have enough space to accommodate the ITCAM for Response Time
Tracking database.

Chapter 6. Installing ITCAM for Response Time Tracking 107

The databse installation steps are:
1. Launch the DB2 installation launch pad and select Install Product. In the
DB2 Setup Wizard window that appears (Figure 6-2), we install the base DB2
server without data warehousing by selecting Typical.

Figure 6-2 DB2 options

2. We install with mostly default options. The instance user ID that we create is
db2admin.
3. ITCAM for Response Time Tracking comes with a script to create the
database schema. Use this script to preinstall the database. Optionally, you
can also have the wizard create the database. This script is in the installation
compact-disc read-only memory (CD-ROM) in the disk1/utilities/<dbversion>
directory. The DB2 creation script is called crtdb_db2. The format for running
the script is as follows:
crtdb_db2 version dbname dbuser dbpasswd country_code create_db
was_user
In this syntax:
– version specifies the DB2 version, such as 8.1 or 8.2.
– dbname is the name of the database being created, such as ITCAMRTT.


Note: DB2 supports only eight characters for the database name.

– dbuser specifies the user name for the DB2 instance owner.
– dbpasswd specifies the password for dbuser.
– country_code is the locale name as understood by DB2.
– create_db indicates whether to create a new database, Y or N.
– was_user is the user that the WebSphere Application Server is running
under.
In our environment, we issue the following command:
crtdb_db2 8.2 ITCAMRTT db2admin xxxxxxxx US Y itcamusr

Important: You must have the proper DB2 environment to issue this
command. On Windows, run this under the DB2 command window. On
UNIX platforms, source the db2profile under the DB2 instance owner home
directory.

4. ITCAM for Response Time Tracking management server connects directly to
the database using Java Database Connectivity (JDBC) type 4. Cataloging
the database locally on the ITCAM for Response Time Tracking management
server is not required if the database is installed remotely.

6.1.3 Installing WebSphere Application Server
We use WebSphere Application Server V6.0.2 and install the network
deployment configuration. The installation first creates the network deployment
manager, dmgr instance. Individual application servers are then installed using
the application server profile. We use the default AppSrv01 profile to connect to
the deployment manager. Any of the machines can be the deployment manager,
which is used mainly to administer the cluster and not an operational runtime
tool.


The WebSphere Application Server installation process is:
1. Launch the WebSphere Application Server Version 6.0 Network Deployment
edition installation wizard, which installs the base WebSphere code. We
install WebSphere Application Server in C:IBMWebSphereAppServer
(Figure 6-3).

Figure 6-3 Installation path for WebSphere Application Server


2. As can be seen in Figure 6-4, we do not install the sample applications. The
Javadocs are optional because they do not create additional processing
requirements. Click Next.

Figure 6-4 WebSphere Application Server components


3. The WebSphere Application Server installation wizard shown in Figure 6-4 on
page 111 facilitates the launch of the profile creation wizard. Select Create an
Application Server profile, as shown in Figure 6-5, and click Next.

Note: We use default values for ports and other options.

Figure 6-5 Profile creation

Note: The deployment manager profile must only be created in the
deployment manager node.

4. After the installation concludes, launch the profile creation wizard again. On
Windows, start this by using the following command:
$WAS_HOMEbinProfileCreatorpctWindows
Alternately, use the wasprofile command to create a deployment manager
profile to host the deployment manager.


5. We create a standard deployment manager profile Dmgr01 from the Profile
creation wizard (Figure 6-6).

Figure 6-6 Application server profile

6. Install any necessary WebSphere Application Server fix packs. In our
environment, we install Refresh Pack 2 for WebSphere Application Server
V6.0.2. Copy the updateinstaller directory into the WebSphere directory. The
refresh pack must be uncompressed into the WebSphere home directory and
launched from there.

Note: Download the WebSphere fix packs or refresh packs from the
following URL:
ftp://ftp.software.ibm.com/software/websphere/appserv/support/fix
packs/<was_version>/<refreshpack or fixpack>/<Platform>

Our code is downloaded from the following URL:
ftp://ftp.software.ibm.com/software/websphere/appserv/support/fix
packs/was60/refreshpack2/Windows

When applying WebSphere fix packs on Windows servers, Java may not be
found even after sourcing the WebSphere Application Server environment with
setupCmdLine.bat. In this case, Java Virtual Machine (JVM) must be indicated
manually using the following command:
update -is:javahome %JAVA_HOME%


You may have to enclose JAVA_HOME within quotes if you have a space in
the installation path, such as program files (Figure 6-7).

C:Program FilesibmWebSphereAppServerupdateinstaller>echo
%JAVA_HOME%
C:Program FilesibmWebSphereAppServerjava

C:Program FilesibmWebSphereAppServerupdateinstaller>update
-is:javahome "%JAVA_HOME%"
Figure 6-7 Updating WebSphere Application Server

7. Start federating the application servers to the deployment manager cluster.
Perform this from the deployment manager administrative console:
http://dmgr_host:9061/ibm/console
8. In the administrative console, select System Administration → Nodes. Click
Add Node, as shown in Figure 6-8.

Figure 6-8 Federating application server


9. The add node wizard guides you to connect to the appropriate node, as
shown in Figure 6-9. The node wizard connects to the administration server to
collect information and create the node agent instance. The node agent talks
with the deployment manager to perform administrative tasks for the
deployment manager. It also includes the server1 instance into the
deployment manager cluster.

Figure 6-9 Add node wizard


10.Federate all the application server instances that you want to include in the
ITCAM for Response Time Tracking clustering server. After the nodes have
been federated, create a cluster. Outage from a single application server
results in the load being forwarded to other members of the cluster. A cluster
is created from the Administrative Console → System Administration →
Servers → Cluster menu. Click New, as shown in Figure 6-10.

Figure 6-10 Cluster list


11.Create the cluster, as shown in Figure 6-11, for defining the name and its
members. At the completion of each step, click Next. Click Finish when done.
Save the WebSphere configuration.

Figure 6-11 Cluster creation

12.The WebSphere cluster must be authenticated uniformly. To perform this
action, activate global security and use an external Lightweight Directory
Access Protocol (LDAP) server for authentication. This paper does not
discuss this, as implementation of different LDAP servers differs considerably.

Note: LDAP is required, as local OS authentication cannot span a server
for a UNIX or a Linux machine. If you are using a Windows domain user
authentication, you can use the Windows domain users for the cluster
security.


13.Install the IBM Hypertext Transfer Protocol (HTTP) server. Install this server
from the installation media of the WebSphere Application Server in the IHS
subdirectory. Install the IBM HTTP Server using the default options in the
C:IHS directory. You must also install the IBM HTTP Server plug-in to the
IBM HTTP Server system.
14.Generate the IBM HTTP Server plug-in. The plug-in generates the
configuration file from the connected WebSphere server. To recreate the
plug-in configuration file, run the configurewebserver1 command from the bin
subdirectory of the plug-in installation directory.
15.Modify the IBM HTTP Server configuration file (httpd.conf) for additional ports
to receive communication from the management agent, typically 9082. Add
the following directive to the httpd.conf file:
Listen 9082

The WebSphere Application Server is ready.

6.1.4 Installing WebSphere Load Balancer
The WebSphere Load Balancer is a component of the WebSphere Edge
Component. The WebSphere Load Balancer is installed using the WebSphere
Edge Component installation media. This is the same media that you used for
installing the WebSphere Caching Proxy for the Store and Forward agent. We
use the dispatcher component of the Load Balancer. Conceptually, the
dispatcher component works as shown in Figure 6-12.

request
srv150

srv105 (non forwarding)
rtt

srv151
Figure 6-12 Load Balancer diagram

The Load Balancer requires two Internet Protocol (IP) addresses. The physical
address, called the non-forwarding address, is used to access the box directly.


However, the Load Balancer uses a second address that is an alias to the
physical IP address. Any request to this address is forwarded to either of the
backend servers.

Perform the following tasks to install the WebSphere Load Balancer:
1. The WebSphere Load Balancer requires a loopback address to be defined as
a forwarding address.
– In AIX, issue the following command:
ifconfig lo0 alias address netmask mask
– In the Windows environment, add the Microsoft loopback adapter device
and assign the cluster IP address.
2. On the welcome panel (Figure 6-13), click Next.

Figure 6-13 Welcome dialog


3. In the license agreement dialog (Figure 6-14), accept the license agreement
by clicking Yes.

Figure 6-14 License agreement

4. The installer then checks the prerequisites. If the check is successful, it then
displays the language selection panel (Figure 6-15).

Figure 6-15 Language selection


5. Figure 6-16 shows type of installation. We use a custom installation and click
Next. Note that we use content-based routing.

Figure 6-16 Select the installation components


6. The next panel (Figure 6-17) shows the summary of selections made and
starts the installation upon clicking Next.

Figure 6-17 Installation summary dialog

7. Finally, the installation result is shown Figure 6-18. Click Finish to complete
the installation and exit the installation wizard.

Figure 6-18 Installation summary and result


Note: WebSphere Load Balancer assumes that you have the Java
Runtime Environment (JRE) javaw executable in the system path. If you do
not have the javaw in the system path, add this before proceeding.

8. Start our content-based routing server. We are running a Linux server. Run
the cbrserver command to start the server.
9. The interaction for defining the load balancer is using a command line
interface. This is performed using the cbrcontrol command.
a. Start the executor:
cbrcontrol executor start
b. Define the cluster:
cbrcontrol cluster add rtt.itsc.austin.ibm.com
c. Define the ports that the cluster listens to:
cbrcontrol port add rtt.itsc.austin.ibm.com:9081
d. Select the backend servers that the Load Balancer will try to balance. Add
all the WebSphere cluster members into this definition:
cbrcontrol server add
rtt.itsc.austin.ibm.com:9081:srv150.itsc.austin.ibm.com
cbrcontrol server add
rtt.itsc.austin.ibm.com:9081:srv150.itsc.austin.ibm.com
e. Add the rule based on which content base routing would route/load
balance the requests. Right-click the port name and select Add Rule
Content:
cbrcontrol rule add rtt.itsc.austin.ibm.com:9081:defRule type
content pattern uri=*/tmtpUI/*

Note: You can choose the type of rule depending on your need. For this
illustration we use content type as the type of rule.

f. Associate the servers in the cluster to the rule defined in previous step e:
cbrcontrol rule useserver rtt.itsc.austin.ibm.com:9081:defRule
srv150.itsc.austin.ibm.com
cbrcontrol rule useserver rtt.itsc.austin.ibm.com:9081:defRule
srv151.itsc.austin.ibm.com
g. Start the manager:
cbrcontrol manager start


h. Start the advisor on the port configured in step c on page 123:
cbrcontrol advisor start http 9081

Note: The above-mentioned steps can also be performed by GUI. Run
lbadmin to start the content-based routing GUI. For more information about
Load Balancer and various configuration commands refer to Load
Balancer Administration Guide, GC31-6858.

10.The result of the configuration is illustrated in Example 6-1.

Example 6-1 Checking result of configuration
[root@srv105 logs]# cbrcontrol server report
rtt.itsc.austin.ibm.com:9081
Cluster: rtt.itsc.austin.ibm.com Port: 9081
-------------------------------------------------------------
| Server | Map port | Total | CPS |
-------------------------------------------------------------
| srv151.itsc.austin.ibm.com |
| | 9081 | 0 | 0 |
| srv150.itsc.austin.ibm.com |
| | 9081 | 0 | 0 |
-------------------------------------------------------------

Your WebSphere Load Balancer is now ready.

6.1.5 Installing the management server
In a clustering environment, the management server is installed using a
command-line script. In WebSphere, the installation process requires access to
the DB2 Java Database Connectivity (JDBC) driver. Copy the Java subdirectory
of the DB2 installation or install the DB2 client locally for this.

Before starting the installation process, get all the following installation
parameters:
WebSphere Application Server information:
– Installation path
– Administrator user and password
– Profile name, cell name, node name, and server name
– Admin console port
– Simple Object Access Protocol (SOAP) connector port


– (Optional) Lightweight Directory Access Protocol (LDAP) user name and
password
– (Optional) Secure Sockets Layer (SSL) information
• Client Key file name and password
• Client Trust file name and password
Database information:
– Database server host name
– Database server port number
– Database name for DB2 or system ID for Oracle
– Database user ID and password
– (DB2 only) JDBC driver path

The installation of the ITCAM for Response Time Tracking clustering
management server uses a special CD-ROM image. It does not include the
installation wizard. Installation is performed interactively using an installation
script. The installation must have both the cluster CD-ROM and the common
CD-ROM images copied to the same installation directory. Example 6-2 shows a
sample copy command for UNIX platforms.

Example 6-2 Sample copy command for UNIX platforms
cp -r RTTMSclustr/shareddir/* <shared dir>
cp -r RTTMScommon/disk2/shareddir/* <shared dir>

Example 6-3 shows a sample copy command for Windows.

Example 6-3 Sample copy command for Windows
xcopy /S/I RTTMSclustrshareddir <shared dir>
xcopy /S/I RTTMScommondisk2shareddir <shared dir>

The installation script is a Java Command Language (JACL) script that
configures and installs tmtpCluster.ear into your cluster. Invoke the JACL script
using the wsadmin command from the network deployment server. The
installation command is:
wsadmin –username <userid> -password <password> –f itcamfrtt.jacl all

The existing options for itcamfrtt.jacl are:
all: This performs all the installation procedures such as install and configure.
configure: This configures the ear application only.
install: This installs the ear application only.
unconfigure: This unconfigures the ear application.
uninstall: This uninstalls the ear application.


removeall: This removes ear application and configuration.

Detailed instructions for running the JACL script are provided in ITCAM for RTT:
Installing a Management Server in a WebSphere Cluster Environment,
SC32-1804. Example 6-4 shows our installation result.

Example 6-4 Running itcamfrtt.jacl
------------------------------------------------
ITCAMfRTT Install/Configuration Script

Operation: all
Silent: false
------------------------------------------------

Have the JVM heap settings been verified? (yes|no) [yes]:
Is global security enabled? (yes|no) [yes]:
Have all nodes been federated and network connectivity verified? (yes|no) [yes]
Please enter the cluster name [ITCAMfRTTCluster]:
Available Nodes:
srv150Node01
srv151Node01
Select the desired node [srv150Node01]:
Please enter the cluster member name [ITCAMfRTTServer1]:
Please enter the path to the shared directory [/mnt/clustershare]:/rttshare
Is this a Windows node (yes|no) [no]:
Current Cluster Nodes and Members:
srv150Node01 - ITCAMfRTTServer1

Add more cluster members (yes|no) [yes]:
Available Nodes:
srv150Node01
srv151Node01
Select the desired node [srv150Node01]: srv151Node01
Please enter the cluster member name [ITCAMfRTTServer2]:
Please enter the path to the shared directory [/rttshare]:
Is this a Windows node (yes|no) [no]:

Current Cluster Nodes and Members:

Add more cluster members (yes|no) [yes]: no

--------------------------------------------------
Cluster information obtained...
--------------------------------------------------

Please enter the WebServer/LoadBalancer port [81]:


Please enter the HTTP transport port for ITCAMfRTTServer1 [9081]:
Please enter the HTTP transport port for ITCAMfRTTServer2 [9081]:

Collecting Database/Datasource Information
------------------------------------------------

Select the backend database type (db2|oracle|oracle10G) [db2]:
Please enter the database driver path for ITCAMfRTTServer1 [c:/sqllib/java]:
/home/db2inst1/sqllib/java
Please enter the database driver path for ITCAMfRTTServer2
[/home/db2inst1/sqllib/java]:
Please enter the database name (location) [ITCAMRTT]: tmtpdb
Please enter the DB2 database hostname [localhost]: lima.itsc.austin.ibm.com
Please enter the DB2 database port number [50000]:
Please enter the database username [db2admin]: db2inst1
Please enter the database password [password]: XXXXXXXX
-------------------------------------------------
Collecting Security Information for JMS
-------------------------------------------------

Please enter the system username [LocalOSUserID]: wasadmin
Please enter the system password [password]: XXXXXXXX
-------------------------------------------------
Collecting Information for Install
-------------------------------------------------

Please enter the full path name of the EAR file [tmtpCluster.ear]:
/rttclust/RTTMSclustr/ear/DB2UDB_V81/tmtpCluster.ear

------------------------------------------------
Configuring Cluster and Cluster Members
Scope: srv150Cell01(cells/srv150Cell01|cell.xml#Cell_1)
------------------------------------------------

Creating Cluster ITCAMfRTTCluster... ITCAMfRTTCluster created successfully!
Creating Cluster Member ITCAMfRTTServer1... ITCAMfRTTServer1 created
successfully!
Enabling SIB Service on ITCAMfRTTServer1... SIB Service enabled successfully!
Creating Cluster Member ITCAMfRTTServer2... ITCAMfRTTServer2 created
successfully!
Enabling SIB Service on ITCAMfRTTServer2... SIB Service enabled successfully!

------------------------------------------------
Cluster Configuration Completed!!!
------------------------------------------------

------------------------------------------------
Installing ITMTP-Cluster
-------------------------------------------------


Installing application {ITMTP-Cluster}...
Application Name: ITMTP-Cluster
Ear file: /rttclust/RTTMSclustr/ear/DB2UDB_V81/tmtpCluster.ear
Target Cluster: ITCAMfRTTCluster
Authentication Alias: TMTPDataSourceAuthData
Starting application install...

------------------------------------------------
ITMTP-Cluster Installation Completed!!!
------------------------------------------------

Do you wish to restart the cluster (yes|no) [yes]:

Starting ITCAMfRTTCluster...
Ripple start of ITCAMfRTTCluster initiated.

6.1.6 Checking the configuration of the RTT cluster application
After the installation completes, log on to the WebSphere server administration
console and check the Map virtual hosts for Web modules setting of RTT cluster
application. See whether they have been mapped to the itcamfrtt_host correctly.
Change the virtual host to itcamfrtt_host if they are not currently mapped to it.
Select OK and restart the application. See Figure 6-19.

Figure 6-19 Virtual hosts definition of RTT cluster application


6.2 Deploying the management resources
The management agent in ITCAM for Response Time Tracking is a neutral
component. Management components are deployed after the installation is
complete, using either the Web interface or the command-line interface (CLI).

6.2.1, “Silent installation of the management agent” on page 129
6.2.2, “Command-line interface for management components” on page 130

6.2.1 Silent installation of the management agent
Installing the management agent by using the silent installation method allows
unattended installation of a large number of agents to be performed. The silent
installation of the ITCAM for Response Time Tracking management agent is
performed using the following command:
setup_MA -silent -options <responsefile>

The response file sample is provided in the responsefiles subdirectory of the
installation media. The options in the response file are divided into:
License agreement setting for management agents
-G licenseAccepted=true
Installation directory for management agents
-P ma.installLocation="C:IBMitcamRTTMA”
Installation settings
-W msConnection.offline="true"
-W msConnection.hostName="fully_qualified_host_name"
-W msConnection.userName="tmtpuser"
-W msConnection.password="tmtppassword"
-W msConnection.sslValue="[true | false]"
-W msConnection.portNumber="9446"
-W msConnection.maPort="1976" (This is the default value.)
-W msConnection.epKeyStore="c:/tmtp/agent.jks"
-W msConnection.epKeyPass="changeit"
-W serviceUser.user="TMTPAgent" (Windows only)
-W serviceUser.password="tivoli" (Windows only)
Global response settings
-G replaceExistingResponse="yesToAll | yes | noToAll | no"
-G replaceNewerResponse="yesToAll | yes | noToAll | no"
-G removeExistingResponse="yesToAll | yes | noToAll | no"


-G removeModifiedResponse="yesToAll | yes | noToAll | no"
Log settings
[-W logSettings.level="trace_level"]
[-W logSettings.consoleOut="true | false"]

A sample response file MA.opt is under the management agent installation file
directory. You can use it to proceed the silent installation by modifying the default
setting in the sample response file.

6.2.2 Command-line interface for management components
To deploy the management agent, use the CLI. The CLI for ITCAM for Response
Time Tracking is provided in the MS_Home/downloads/cli directory. This
directory, which can be copied to any machine, provides the functionality from a
command line. The CLI is useful for providing a scripted deployment of ITCAM
for Response Time Tracking.

Copy the entire directory from MS_Home/downloads/cli. If you are running a
Windows system, copy w32util.dll to %WINSYSDIR%w32util.dll. Modify
tmtpcli.bat or tmtpcli.sh to define CLI_HOME to the path where you put the
directory. In our environment, we run the CLI from the management server,
which is installed in C:IBMitcamRTTMS. Therefore, CLI_HOME is set to:
CLI_HOME=C:IBMitcamRTTMSdownloadscli

You must have JAVA_HOME defined to the JRE 1.4 environment. With the
WebSphere Application Server, you can run setupCmdLine.bat or
setupCmdLine.sh to establish your environment.

Initially, generate the tmtpcli.properties file using the CreateCliProperties
command. Depending on whether you activate global security, you may have to
use Hypertext Transfer Protocol Secure (HTTPS) protocol and specify the
KeyStore file and its password. The Universal Resource Locator (URL) for the
management server is the primary URL for the application server, and not the
Web console URL. In our environment we issue the following command:
tmtpcli -CreateCliProperties –MsUrl http://khartoum:9081/ –UserName
itcamrtt –Password xxxxxxx

See IBM Tivoli Composite Application Manager for Transaction Tracking
Reference Guide, SC32-9485.


Deploy the monitoring components using the CLI. Each management component
type requires different invocation parameters.
Deploying quality of service component:
tmtpcli -DeployQoS –AgentName laredo –ProxyHostname
laredo.itsc.austin.ibm.com –ProxyPort 7081 –OriginHostname
laredo.itsc.austin.ibm.com –OriginPort 9081 -Wait
Deploying synthetic transaction investigator component:
tmtpcli -DeploySti –AgentName geneva –Wait
Deploying generic Windows component:
tmtpcli –DeployGenWin –AgentName ser1 –ProjectName trader001
–ProjectPassword pwd –ProjectUserId Administrator –Wait
Deploying Application Resource Measurement (ARM) custom application
component:
tmtpcli –DeployCustomApp –AgentName laredo –ArmAppName Trader
Deploying Java 2 Platform, Enterprise Edition (J2EE) component:
tmtpcli –DeployJ2ee –AgentName laredo –ServerType WebSphere –Version
6.0 –ServerHome C:IBMWebSphereAppServer –ServerName laredo1
–ProfileName AppSrv01 –AutoRestart –NetworkDeploy –DeployManagerName
laredo.itsc.austin.ibm.com –AdminPort 9060 –Wait

6.2.3 Defining management resources
Management resource definition for a large-scale implementation is
cumbersome if performed using the Web console. Define these management
resources, such as agent groups, policy groups, operators, and schedules, by
using the command-line, too:
Defining policy groups:
tmtpcli -CreatePolicyGroup <policy group name>
Defining agent groups:
tmtpcli -CreateAgentGroup <agent group name> -Agents <comma
separated agent list>
Defining users:
tmtpcli -CreateUser <user>
Defining schedules:
tmtpcli -CreateSchedule <xml filename>


The schedule itself must be defined in an eXtensible Markup Language (XML)
file. Example 6-5 shows the schedule XML file definition.

Example 6-5 Schedule XML file
<soapenv:Envelope
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<ns1:ScheduleData xmlns:ns1="urn:TMTP">
<arg0 href="#every5min"/>
</ns1:ScheduleData>
<multiRef id="every5min" soapenc:root="0"
soapenv:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
xsi:type="ns2:com.ibm.tivoli.transperf.core.ejb.common.ScheduleData"
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:ns2="typeNS">
<created xsi:type="xsd:dateTime">1970-01-01T00:00:00.000Z</created>
<creator xsi:type="xsd:string"></creator>
<deleted xsi:type="xsd:dateTime">1970-01-01T00:00:00.000Z</deleted>
<deletor xsi:type="xsd:string"></deletor>
<description xsi:type="xsd:string">Changed temporally by
Danilo</description>
<end xsi:type="xsd:int">0</end>
<endDateTime
xsi:type="xsd:dateTime">1970-01-01T00:00:00.000Z</endDateTime>
<endTime xsi:type="xsd:string">00:00</endTime>
<isDeleted xsi:type="xsd:boolean">false</isDeleted>
<iteration xsi:type="xsd:int">5</iteration>
<iterationType xsi:type="xsd:int">0</iterationType>
<iterationUnit xsi:type="xsd:string">MINUTES</iterationUnit>
<lastUpdated
xsi:type="xsd:dateTime">2005-10-14T15:37:07.672Z</lastUpdated>
<lastUpdator xsi:type="xsd:string"></lastUpdator>
<name xsi:type="xsd:string">Every_5_5min</name>
<objectVersion xsi:type="xsd:int">1</objectVersion>
<run xsi:type="xsd:int">1</run>
<runFri xsi:type="xsd:boolean">false</runFri>
<runMon xsi:type="xsd:boolean">false</runMon>
<runSat xsi:type="xsd:boolean">false</runSat>
<runSun xsi:type="xsd:boolean">false</runSun>
<runThu xsi:type="xsd:boolean">false</runThu>
<runTues xsi:type="xsd:boolean">false</runTues>


<runWed xsi:type="xsd:boolean">false</runWed>
<start xsi:type="xsd:int">0</start>
<startDateTime
xsi:type="xsd:dateTime">2005-10-14T21:36:00.656Z</startDateTime>
<startTime xsi:type="xsd:string">00:00</startTime>
<timeZone xsi:type="xsd:string">US/Central</timeZone>
<type xsi:type="xsd:string">PLAYBACK_SCHEDULE</type>
<useDST xsi:type="xsd:boolean">true</useDST>
<uuid xsi:type="xsd:int">221</uuid>
</multiRef>

6.3 Setting up Secure Sockets Layer certificates
To secure communication between various components of ITCAM for Response
Time Tracking, activate SSL. ITCAM for Response Time Tracking provides a
default set of certificates to let you implement SSL encryption using the default
setting.

The default setting has the communication encrypted, and provides the quickest
way to implement SSL. It also lets you test the function with the SSL enabled, but
without the risk of potential authentication problems with custom certificates.
Anybody with the installation image can extract the certificate and unscramble
the information.

6.3.1, “Secure Sockets Layer for ITCAM for Response Time Tracking” on
page 133
6.3.2, “Working with custom certificates” on page 134

6.3.1 Secure Sockets Layer for ITCAM for Response Time Tracking
Secure Sockets Layer in ITCAM for Response Time Tracking is handled using
the Java Secure Socket Extension (JSSE). In this configuration, SSL certificates
and keys are stored in a jks (Java key store) file. Management of these jks files is
performed using the IBM GSKIT.

Note: The IBM GSKIT is different from the keytool command that is supplied
in the Java environment.


Perform Secure Sockets Layer activation directly with the installation wizard or
the silent installation options when you install ITCAM for Response Time
Tracking. The default port assignments for SSL communication are:
Management server port: 9446
Store and Forward Agent port: 443 for the management agent or downstream
Store and Forward Agent, and 1976 for the management server or upstream
Store and Forward Agent

To activate SSL communication after you have ITCAM for Response Time
Tracking running without SSL, perform the following tasks:
1. Activate SSL on the management server.
The SSL setting on the management server is stored in the WebSphere
Application Server settings. Changing to SSL requires modification in the
following paths:
– Select Security → SSL and modify
host/TMTPSettings_Without_ClientAuth and
host/TMTPSettings_With_ClientAuth to specify the jks file and its
password.
– Select Environment → Manage WebSphere Variables and modify the
TMTP_KEYSTORE and TMTP_KEYPASS variables. The
TMTP_KEYPASS password contains an encrypted password that can be
created by using pass_util.jar.
2. Activate SSL on the Store and Forward agent.
The Store and Forward agent uses WebSphere Caching Proxy. SSL
modification requires changes in ITCAM for Response Time Tracking and
WebSphere Caching Proxy.
– Modify the SnF/config/endpoint.properties file and the change
endpoint.keystore and endpoint.keypass parameters.
– Modify etc/cp/en_US/ibmProxy.conf and the KeyRing and KeyRingStash
variables.
3. Activate SSL on the management agent.
Modify the config/endpoint.properties file and change the endpoint.keystore
and endpoint.keypass parameters.

6.3.2 Working with custom certificates
You may want to implement your own set of custom certificates. The basic
operation of certificates lets you define your identity certificate to encrypt the


information that you send and lets everyone you trust have a public key to
decrypt your information.

Although it is possible to define a separate certificate identity for every machine
that communicates, the sheer number of machines and operation changes
involved in managing them are prohibitive, especially in a large environment. We
implemented the following set of certificates:
Management server certificate
Store and Forward agent certificate
Management agent certificate that communicates directly to the management
server
Management agent certificate that communicates through the Store and
Forward agent

Configuration involves the following steps:
1. The jks files are built and managed using the iKeyMan command. The
command resides in the WebSphere Application Server’s bin directory.
Create the following jks files:
– For the management server: jks file for the management server. We call
this file prodms.jks.
– For the Store and Forward agent: jks file for the Store and Forward agent.
We call this file prodsnf.jks.
– For the management agent: jks file for the management agent. We call
this file prodma.jks.
Besides these files, the iKeyMan tool also generates a password key file with
the same name and extension of arm.
2. Run the gsk7ikm command from the WebSphere Application Server’s
gskit7install directory. This is used to create a key database (kdb file) for the
Store and Forward agent, and possibly the quality of service proxy. The
quality of service SSL is not discussed in this paper as it relates to the
application requirement.
This task produces a kdb file, a stash file (sth) (where the kdb password is
stored), and an arm file.
3. The jks and kdb file sets must be installed and activated on the ITCAM for
Response Time Tracking, as discussed in 6.3.1, “Secure Sockets Layer for
ITCAM for Response Time Tracking” on page 133.


7

Chapter 7. Maintenance of ITCAM for
Response Time Tracking

This chapter discusses the issues relating to the maintenance of ITCAM for
Response Time Tracking in a large-scale environment. This chapter discusses
the following topics:
7.1, “Operational issues pertaining to a large environment” on page 138
7.2, “Performance and availability of management server” on page 138
7.3, “ITCAM for Response Time Tracking files” on page 140


7.1 Operational issues pertaining to a large
environment
This chapter discusses operational issues regarding a large-scale ITCAM for
Response Time Tracking environment, and focuses on the issues that arise
largely because of the environment size. The topics discussed in this chapter
are:
7.2, “Performance and availability of management server” on page 138
As the size of the environment grows, even a slight inefficiency can cause a
major problem. A large-scale environment requires the performance and
availability of the managing server to be soundly maintained.
7.3, “ITCAM for Response Time Tracking files” on page 140
This is a necessity. This section discusses specific backup and recovery
considerations and maintenance of files.
This discusses the implementation of WebSphere, Database 2 (DB2), and
ITCAM for Response Time Tracking patches. Typically, with the number of
data collectors involved, this can be a huge task.

7.2 Performance and availability of management server
Performance and availability of the ITCAM for Response Time Tracking
management server is related mainly to the IBM WebSphere Application Server
that hosts it and the availability of its database. This section discusses the
following topics:
7.2.1, “Performance of WebSphere Application Server” on page 138
7.2.2, “Database maintenance” on page 139

7.2.1 Performance of WebSphere Application Server
In ITCAM for Response Time Tracking, the management server runs entirely on
the WebSphere Application Server process. The performance of the
management server and the Web interface are greatly dependent on the
performance of the WebSphere Application Server. This makes the performance
tuning of the WebSphere Application Server critical.


The WebSphere Application Server basically runs on a Java Virtual Machine
(JVM). The typical performance indicators for a JVM are:
Adequate processing power from the original machine
The clustering environment ensures that you can distribute the processing
power across enough server machines. These machines must be dedicated
and must not be running any unnecessary tasks.
Class loading disk performance
WebSphere boot classes and other jar files from ITCAM for Response Time
Tracking must be stored in a contiguous and fast disk. If possible, separate
the system paging area with the jar files.
Adequate memory size for processing
As JVMs use a lot of memory, provide adequate memory for the server. You
may want to specify the minimum JVM size from the WebSphere Application
Server configuration. You must be able to find the size after running the
server for a while. Allocating memory on the fly is expensive.
Another aspect of JVM memory tuning is the garbage collection. The
frequency, as well as the garbage collection options, must be tuned to not
interfere with the processing and waste critical memory space.
Network connection to the database engine and from the WebSphere Edge
Server
Primary network connection considerations are request and uploading data to
the management server, and data to be put into the database. If possible, the
cluster and the WebSphere Edge Server must be running, using a separate
local network to minimize packet collision with other applications on the
Ethernet bus.

7.2.2 Database maintenance
The performance of the management server is also related to database
performance, which is in turn related to how the data is stored physically and how
well the database manager understands the nature of data. This section is
primarily aimed at discussing the use of DB2 as the database manager.

The primary tools to ensure this are the REORG and RUNSTAT tools. Run both
of these utilities from the DB2 utilities from the DB2 command line. Following is a
description of these tools:
The REORGCHK utility reads the table information and analyzes the
necessity of running reorg. REORGCHK used current statistics or generates
its own statistics.

Chapter 7. Maintenance of ITCAM for Response Time Tracking 139

The REORG utility reorganizes the database structure to ensure that data is
allocated coherently and in a typical access sequence requested by most
queries. The syntax for running REORG is:
DB2 REORG TABLE creator.table_name INDEX primary_index_name
The RUNSTATS utility collects the statistics of the tables and indexes to
ensure that the database manager chooses the most optimized method to get
the data requested by a Structured Query Language (SQL) request. The
syntax for running RUNSTATS is:
DB2 RUNSTATS ON TABLE creator.table_name WITH <runstat_options>

Typically, you should run REORGCHK for frequently modified tables, and then
determine whether you want to run REORG on those tables. After all the
modifications are performed, run RUNSTATS to ensure that the statistics are
current and the DB2 obtains the most optimal path to the data that an SQL
requested.

We also recommend a regular, maybe monthly, maintenance window to REORG
all the tables for ITCAM for Response Time Tracking and perform a complete
RUNSTATS.

Database performance also relates to the size of the data itself. Regular pruning
of the collected data must be performed to ensure database performance. This
pruning action must be carried out before any REORG or RUNSTATS utilities are
performed.

7.3 ITCAM for Response Time Tracking files
This section discusses the file considerations for ITCAM for Response Time
Tracking and provides information about the following topics:
7.3.1, “Backup and recovery” on page 140
7.3.2, “Managing the log files” on page 141

7.3.1 Backup and recovery
The backup and recovery procedure for ITCAM for Response Time Tracking is
similar to the backup and recovery procedure for ITCAM for WebSphere
discussed in 4.3, “Backup and recovery configuration” on page 83. It consists of
server backup, WebSphere Application Server configuration, and DB2 database
backup.


Note: WebSphere Application Server backup must be performed on the
network deployment engine instead of on individual application servers.

An additional requirement for backup may arise from the WebSphere Edge
Server. Typical backup of the product installation directories must be adequate.

7.3.2 Managing the log files
Log files in ITCAM for Response Time Tracking are used for problem
determination of the product’s working mechanism. Several locations exist for
the log files, depending on the components that generate them. The locations
are:
Tivoli common directory
This is where most of the log files are stored. The typical directory names are:
– %ProgramFiles%ibmtivolicommon (Windows)
– /var/ibm/tivoli/common (UNIX)
The directory under the common directory is typically identified by a
three-character product code, such as BWM for ITCAM for Response Time
Tracking. In the product directory, the following sub-directories exist:
– First-failure data capture (FFDC) for first failure data capture application
The FFDC directory contains diagnostic information pertaining to the
program exceptions that are captured.
– The logs subdirectory of Tivoli common directory
This is typically filled with log files and must be maintained.
WebSphere log files
WebSphere log files are located on each WebSphere Application Server
instance under the path $WAS_HOME/profiles/<profile>/logs/<server>.
As these log files have a maximum size and a predefined number of
generations, they cannot exceed a certain size. This path must be maintained
and checked for ad hoc logging that may use up the space. We recommend
that you use a separate file system to contain the common directory.


7.4 Performing product maintenance
Various considerations must be taken into account when dealing with the
application of product maintenance, fixes, and fix packs in a large environment,
as this must be performed remotely, and automated with minimal impact to the
production system. This section discusses the following topics:
7.4.1, “Getting software updates” on page 142
7.4.2, “Updating ITCAM for Response Time Tracking management server” on
page 142
7.4.3, “Updating ITCAM for Response Time Tracking management agents” on
page 143

7.4.1 Getting software updates
Obtain the software updates from the following IBM Web sites:
For ITCAM for Response Time Tracking:
http://www-306.ibm.com/software/sysmgmt/products/support/IBMTivoliCo
mpositeApplicationManagerforResponseTimeTracking.html
For DB2 Universal Database:
For WebSphere Application Server:

Typically, you do not have to obtain updates for DB2 or WebSphere on the
managing server. ITCAM for Response Time Tracking only supports a certain
level of these products. However, you may sometimes require a certain software
level to fix a problem.

7.4.2 Updating ITCAM for Response Time Tracking management
server
The ITCAM for Response Time Tracking management server updates must be
performed as a scheduled maintenance. This means that:
You must have a backup of the management server system.
You must perform an orderly shutdown of the management server.
Users must be notified of the system’s unavailability.

The patch for ITCAM for Response Time Tracking is distributed as a
self-extracting executable called 6.0.0.0-TIV-RTT-FP000n_<platform> with the


bin or exe extensions for each supported management server platform. It also
provides zip files for cluster-based installation.

Non-cluster installation is performed by launching the executable and following
the prompt, while cluster-based implementation is performed by extracting the
zip file and running the supplied Java Command Language (JACL) scripts with
the wsadmin command. You may have to update the ITCAM for Response Time
Tracking database using the supplied script programs. See the corresponding
readme files for more information.

Restart the WebSphere Application Server after applying the update.

7.4.3 Updating ITCAM for Response Time Tracking management
agents
Perform management agent updates in ITCAM for Response Time Tracking
directly from the management server process. Use the automatic agent through
the Web console. Select System Administration → Agent Updates. Install the
agents manually using the following command:
<MA_BASEDIR>jre142jrebinjava -Dtmtp.user.dir=<MA_BASEDIR> -jar
ma_fixpack.jar -silent

Restriction: Manually install the management agent for z/OS and OS/400
using a platform interface such as System Modification Program/Extended
(SMP/E).

Reinstall the Java 2, Enterprise Edition (J2EE) component after updating the
agent. Use the Web interface or the command line for this. For a large
installation, perform both the steps using the command-line interface, as shown
in Example 7-1.

Example 7-1 Updating J2EE components using the command line
tmtpcli -RemoveComponent -AgentName <agent> -ComponentType J2EE_WAS
tmtpcli -DeployJ2ee –AgentName <agent> –ServerType WebSphere –Version
6.0 –ServerHome serverHome –ServerName serverName –ProfileName
profileName [–AutoRestart] [–IsSecure –AdminUserName <adminUserName>
–AdminPassword <adminPassword>] [–NetworkDeploy –DeployManagerName
<deployManagerName> –AdminPort <adminPort>] [–Wait]


Abbreviations and acronyms
AIX Advanced Interactive JMX™ Java Management
eXecutive Extensions
API Application programming JNI™ Java Native Interface
interface JRE Java Runtime Environment
ARM Application Response JVM Java virtual machine
Measurement
JVMPI Java Virtual Machine Profiler
BCM Byte Code Modification Interface
CD-ROM Compact-disc read-only JVMTI Java Virtual Machine Tool
memory Interface
CLI Command-line interface LDAP Lightweight Directory Access
CPU Central processing unit Protocol
DB2 Database 2 MSC Multiple Systems Coupling
EJB Enterprise JavaBeans MVS™ Multiple Virtual Storage
ETE™ End-to-end PCB Program control block
FFDC First-failure data capture PDF Portable Document Format
FMID Function modification PMI Performance monitor
identifier interface
GUI Graphical user interface RMI Remote Method Invocation
HTML Hypertext Markup Language RTE runtime environment
HTTP Hypertext Transfer Protocol SNMP Simple Network Management
HTTPS HTTP Secure Protocol

IBM International Business SOAP Simple Object Access
Machine Corp. Protocol

ITIL Information Technology SQL Structured Query Language
Infrastructure Library SSH Secure SHell
ITM IBM Tivoli Monitoring SSL Secure Sockets Layer
ITSO International Technical STI Synthetic Transaction
Support Organization Investigator
J2C Java2 Connectivity TCP/IP Transmission Control
J2EE Java 2 Platform, Enterprise Protocol/Internet Protocol
Edition TEMA Tivoli Enterprise Monitoring
JAR Java archive Agent

JDBC Java Database Connectivity TEMS Tivoli Enterprise Monitoring
Server
TEP Tivoli Enterprise Portal


TEPS Tivoli Enterprise Portal Server
UDB Universal Database
UML Universal Markup Language
URI Universal Resource Identifier
URL Universal Resource Locator
WSDL Web Services Definition
Language
XML eXtensible Markup Language


Related publications

The publications listed in this section are considered particularly suitable for a
more detailed discussion of the topics covered in this Redpaper.

IBM Redbooks
For information about ordering these publications, see “How to get IBM
Redbooks” on page 149. Note that some of the documents referenced here may
be available in softcopy only.
Installing WebSphere Studio Application Monitor V3.1, SG24-6491
Unveil Your e-business Transaction Performance with IBM TMTP 5.1,
SG24-6912
WebSphere Studio Application Monitor V3.2 Advanced Usage Guide,
SG24-6764

Other publications
These publications are also relevant as further information sources:
ITCAM for Response Time Tracking publications
– IBM Tivoli Composite Application Manager for Response Time Tracking
Administrator’s Guide, SC32-9483
Checking Performance and Availability, SC32-9484
Installation and Configuration Guide, GC32-9482
– IBM Tivoli Composite Application Manager for Response Time Tracking:
Installing a Management Server in a WebSphere Cluster Environment,
SC32-1804
Prerequisites, SC32-9486
Problem Determination Guide, SC32-9513


Reference Guide, SC32-9485
V6.0 Program Directory, GI11-4099
ITCAM for WebSphere publications
– IBM Tivoli Composite Application Manager for WebSphere Installation and
Customization Guide, GC32-9506
– IBM Tivoli Composite Application Manager for WebSphere: Installing and
Configuring the Tivoli Enterprise Monitoring Agent, SC32-1801
– IBM Tivoli Composite Application Manager for WebSphere Operator’s
Guide, SC32-9508
– IBM Tivoli Composite Application Manager for WebSphere Problem
Determination Guide, SC32-9509
– IBM Tivoli Composite Application Manager for WebSphere: Tivoli
Enterprise Monitoring Agent Problem Determination Guide, SC32-1800
– IBM Tivoli Composite Application Manager for WebSphere User’s Guide,
SC32-9507

Online resources
These Web sites and URLs are also relevant as further information sources:
Microsoft link for InstallShield error
http://support.microsoft.com/default.aspx?scid=kb;en-us;295278
Microsoft Windows Services for UNIX
http://www.microsoft.com/windows/sfu/default.asp
Open group Web site for Application Response Time Management
http://www.opengroup.org/arm
Product prerequisites information
http://publib.boulder.ibm.com/tividd/td/ITCAMRTT/prereq60/en_US/HTML
/Version60.html

http://publib.boulder.ibm.com/tividd/td/ITCAMWAS/prereq60/en_US/HTML
/itcam6.html


Product Web pages
http://www-306.ibm.com/software/tivoli/products/composite-applicatio
n-mgr-rtt/
http://www-306.ibm.com/software/tivoli/products/composite-applicatio
n-mgr-websphere/
WebSphere fix pack links
ftp://ftp.software.ibm.com/software/websphere/appserv/support/fixpac
ks/was60/refreshpack1/Windows
ftp://ftp.software.ibm.com/software/websphere/appserv/support/fixpac
ks/was60/refreshpack1/Linux

How to get IBM Redbooks
You can search for, view, or download Redbooks, Redpapers, Hints and Tips,
draft publications and Additional materials, as well as order hardcopy Redbooks
or CD-ROMs, at this Web site:
ibm.com/redbooks

Help from IBM
IBM Support and downloads
ibm.com/support

IBM Global Services
ibm.com/services

Related publications 149

Index
gsk7ikm 135
A iKeyMan 135
application management 2
install-DC 50
pillar 5
java 44
application monitor 20
keytool 68
data collector agent 20
restoreConfig 83
managing server 20
run-stat-cmds 78
Application Response Measurement
SetupCmdLine.bat 113
ARM 97
setupCmdLine.bat 130
Application Response Measurement, See ARM
su 36
application server 19
tmtpcli 130
archive agent 23, 26, 48
update 113
trimming 79
wasprofile 112
ARM 11
wsadmin 29
Application Response Measurement 97
communication
port requirement 31
B security 100
backup communication security 29
database 84 composite application 2
ITCAM for WebSphere 83 confidentiality support 8
backupConfig command 83 config_DC command 56
configuration files 44, 86
configuration management database, See CMDB
C configurewebserver1 command 118
certificate key generator 30
CreateCliProperties command 130
CICS 6
crtdb_db2 command 108
Customer Information Control System 96
custom certificate 67
CLI
custom certificates 134
command-line interface 96
Customer Information Control System
Client Application Tracker 13
CICS 96
cluster 106
Customer Information Control System, See CICS
environment 99
CMDB 3
command-line interface 130 D
CLI 96 data collection mechanism 96
commands data collector 19, 96
backupConfig 83 additional 56
config_DC 56 deploying 28
configurewebserver1 118 deployment 50
CreateCliProperties 130 data retrieval 97
crtdb_db2 108 data trimming 78
datatrim.sh 80 database
db2cmd 37, 39 cataloging 37
db2install 36 connection pools 19


engine 139 sponse Time Tracking
installation 107 backup, recovery 140
installation scripts 36 components 12
maintenance 139 management agent 12
table information 24 management server 12
gcdata 24 Store and Forward Agent 12
memorydata 24 deployment 98
methods 24 features 11
pmidata 24 file systems 107
requests 24 functions 11
volumestats 24 installation, deployment 105
database maintenance 77 maintenance 137
datatrim.sh command 80 management agents, update 143
DB2 management server, processing 98
Client Application Enabler 35 management server, update 142
database 107 overview 11
Universal Database platforms supported 14
Enterprise Server 25, 98 server sizing
installation 35 communication bandwidth 97
db2cmd command 37, 39 IBM Tivoli Composite Application Manager for Web-
db2install command 36 Sphere 19
designing managing server 25 architecture 19–20
disaster recovery 32, 102 availability 6
backup, recovery configuration 83
component
E Tivoli Enterprise Monitoring Agent 10
EJB 19
components 9
usage 19
archive agent 26, 28
embedded installation 27
data collector 10
Enterprise JavaBeans, See EJB
global publishing server 26
ETEWatch 13
kernels 26
managing server 9
F message dispatcher 27
failover 102 polling agent 27
fault tolerance 102 publish server 26, 28
file system 107 visualization engine 27
firewall 101 data sources 19
considerations 30 features 9
support 8 functions 9
implementation 25
managing server 20, 25
G
global publish server 43 communication bandwidth 22
gsk7ikm command 135 database size 24
sizing 22
overview 8
I platforms supported 10
IBM GSKIT 133 product architecture 18
IBM IT Service Management approach 3 IBM Tivoli Composite Application Manager for Web-
IBM Tivoli Composite Application Manager for Re- Sphere managing server


processing requirement 24 J
IBM Tivoli Composite Application Manager portfolio J2EE
5 application server 19
for IBM CICS Transactions 5 monitoring agent 13
for IBM IMS Transactions 5 JACL scripts 29
for Response Time Tracking 5 java command 44
for SOA 5 Java Database Connectivity, See JDBC
for WebSphere 5 Java Runtime Environment, See JRE
IBM Tivoli OMEGAMON® XE for IBM Web- Java Virtual Machine Tool Interface, See JVMTI
Sphere Business Integration 5 JDBC 9
IBM Tivoli Composite Application Manager Re- JRE 43
sponse Time Tracking JVM thread pools 19
designing management server 98 JVMTI 19
IBM Tivoli software portfolio 4
business service management 5
composite application management 4 K
kernel 23, 26
event correlation and automation 4
parameters 106
orchestration and provisioning 4
keystore 68
resource monitoring 4
extraction of certificates 71
iKeyMan command 135
keytool command 68
IMS 6, 19
Information Management System 96
in-depth resource usage 6 L
Information Management System large-scale environment 3
IMS 96 definition 6
Information Management System, See IMS large-scale implementation
installation parameters 61 concerns, considerations 7
install-DC command 50 deploying agents 7
IT Infrastructure Library, See ITIL maintenance 8
ITCAM for Response Time Tracking reliability 8
planning considerations 94 security 8
product architecture 94 sizing servers 7
sizing parameters 96 listening policies 97
sizing servers 96 load balancing 99
ITCAM for WebSphere log files 85, 141
components 26
database installation, configuration 35
features 9
M
management agent 95, 135
installation of managing server 34
deployment 99
configuration 34
management resources
maintenance 75
definition 131
operational issues 76
deployment 129
ITCAM for WebSphere managing server
management server 95
sizing
installation 124
memory size 23
parameters 124
ITIL 3
performance, availability of 138
managing server 19
global publishing server 26

Index 153

managing server split server 35 reliability 102
mass automated installation 29, 100 remote database 34
message dispatcher 27, 43 reporting load 96
Microsoft Windows Services for UNIX, See SFU request table 78
modular application development 3 response time 6
monitoring components 131 data 97
Monitoring On Demand 22 response time measurement 97
monitoring policies 97 restoreConfig command 83
run-stat-cmds command 78
N
name resolution 30 S
naming convention 99 Secure Sockets Layer communication 30
Secure Sockets Layer, See SSL
security
O communication, firewall, port usage 29
OCTIGATE database 35
service level agreement 5
OMEGAMON XE 18
setenv.sh 23
operating system preparation 106
SetupCmdLine.bat 113
operational management pillar 4
setupCmdLine.bat 130
Oracle database server 25, 98
SFU 36
overhead
silent installation 28, 50, 99, 129
data collector 25
Simple Network Management Protocol, See SNMP
overseer components 24
sizing servers 21
parameters 21
P complexity of transaction 21
Performance Management Interface, See PMI data collection filter 21
planning considerations 18 instrumentation level 21
playback policies 97 listening policy mask 21
PMI 19 monitoring level 21
policy number of data collectors 21
discovery 98 sampling rate 21
listening 98 transaction rate 21
playback 98 SNMP 27
polling agent 27, 43 software updates 87, 142
port 101 split server
consolidation 31 configuration 42
issues 30 installation, setup 43
product maintenance 87, 142 SSL 30, 64
pruning tables 78 certificates 133
publish server 23, 26, 48 communication 101
data collector setup 65
managing server setup 64
Q
Quality of Service monitoring agent 13 SSL communication 30
STI 13
Store and Forward Agent 95, 135
R su command 36
Rational Robot 12, 96 Synthetic Transaction Investigator 96
Redbooks Web site 149 Synthetic Transaction Investigator, See STI
Contact us xi


T
Tivoli common directory 141
Tivoli Enterprise Monitoring Agent 13
Tivoli Enterprise Monitoring Server 96
Tivoli Enterprise Portal 10, 96
tmtpcli command 130
tmtpcli.properties 130

U
update
managing server 87
update command 113
user right assignment 106
utilities
REORGCHK 139
RUNSTATS 140
utility
REORGCHK 77
RUNSTATS 77

V
visualization engine 24, 27, 41, 43, 76

W
wasprofile command 112
WebSphere
log files 141
security 100
WebSphere Application Server 25, 98, 109
installation 109
performance 76, 138
WebSphere Edge Component
installation 118
WebSphere Edge Server 106
WebSphere security 29
WebSphere Studio Application Monitor 18
WebSphere Studio Application Monitor, See WSAM
Windows workstation 12
wsadmin command 29
WSAM 18

Index 155

Back cover ®

Large-Scale Implementation of
IBM Tivoli Composite Application
Manager for WebSphere and
Response Time Tracking Redpaper

Planning for This IBM® Redpaper discusses large-scale implementation
performance of of IBM Tivoli® Composite Application Manager for INTERNATIONAL
management WebSphere® and IBM Tivoli Composite Application Manager TECHNICAL
infrastructure for Response Time Tracking. Large-scale implementation is SUPPORT
typically characterized by the number of monitoring agents ORGANIZATION
deployed and the number of transactions load-managed. A
Implementing with
typical large-scale implementation of a monitoring product
multiple servers
contains the following challenges:
Keeping up the performance of the monitoring tools BUILDING TECHNICAL
Performing mass to accommodate the processing load from the INFORMATION BASED ON
update of agents agents. PRACTICAL EXPERIENCE
Automation of installation, update, and maintenance
of monitoring agents based on silent installation and
IBM Redbooks are developed by
automated update.
the IBM International Technical
Specific day-to-day maintenance actions to ensure Support Organization. Experts
performance and availability of the monitoring from IBM, Customers and
solution. Partners from around the world
This IBM Redpaper addresses these issues with regard to the create timely technical
implementation of ITCAM for WebSphere and ITCAM for information based on realistic
Response Time Tracking on distributed platforms. The scenarios. Specific
recommendations are provided
discussion is divided into planning issues, implementation to help you implement IT
guides, and maintenance considerations. solutions more effectively in
your environment.

For more information:
ibm.com/redbooks

Large scale implementation of ibm tivoli composite application manager for web sphere and response time tracking redp4162

More Related Content

What's hot (17)

Similar to Large scale implementation of ibm tivoli composite application manager for web sphere and response time tracking redp4162 (20)

More from Banking at Ho Chi Minh city (20)

Recently uploaded (20)

Large scale implementation of ibm tivoli composite application manager for web sphere and response time tracking redp4162