Ibm tivoli key lifecycle manager for z os redp4472

Front cover

IBM Tivoli Key Lifecycle
Manager for z/OS
Features and benefits

Planning, installation, and use

Troubleshooting tips

Karan Singh
Steven Hart
William C. Johnston
Lynda Kunz
Irene Penney

ibm.com/redbooks Redpaper

International Technical Support Organization

IBM Tivoli Key Lifecycle Manager for z/OS

August 2009

REDP-4472-00

Note: Before using this information and the product it supports, read the information in “Notices” on
page ix.

First Edition (August 2009)

This edition applies to Version 1, Release 0 of Tivoli Key Lifecycle Manager for z/OS (product number
5698-B35).

This document created or updated on August 6, 2009.

© Copyright International Business Machines Corporation 2009. All rights reserved.
Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.

Contents

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
The team who wrote this paper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Become a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii

Chapter 1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Tivoli Key Lifecycle Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 How tape encryption works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 How DS8000 encryption works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Why use Tivoli Key Lifecycle Manager and Tape/DS8000 encryption . . . . . . . . . . . . . . 5
1.5 Encryption key management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.5.1 Tivoli Key Lifecycle Manager services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.5.2 Key exchange . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.6 Encryption key methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.6.1 System-managed encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.6.2 Library-managed encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.6.3 Encrypting and decrypting with SME and LME . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.6.4 Application-managed encryption. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.6.5 Mixed mode example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Chapter 2. Planning for Tivoli Key Lifecycle Manager and its keystores. . . . . . . . . . . 23
2.1 Planning for encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.2 What data to encrypt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.2.1 Encrypting data on disk. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.2.2 Encrypting data on tape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3 Where does the data reside? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.4 Rekeying considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.5 Performance and capacity considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.5.1 Performance considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.5.2 Capacity considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.6 Keys and certificates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.7 Tivoli Key Lifecycle Manager considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.7.1 Multiple Tivoli Key Lifecycle Managers for redundancy . . . . . . . . . . . . . . . . . . . . 27
2.7.2 Tivoli Key Lifecycle Manager location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.7.3 Database selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.7.4 Keystore considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.8 Additional deployment considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.8.1 Sysplex versus monoplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.8.2 Active/Active . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.8.3 Primary/Secondary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.8.4 Cloning z/OS Tivoli Key Lifecycle Manager instances . . . . . . . . . . . . . . . . . . . . . 32
2.8.5 Data sharing on z/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.8.6 VIPA and Sysplex distributor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.9 Additional considerations for encrypting data on tape cartridges . . . . . . . . . . . . . . . . . 33
2.9.1 Encryption method comparison. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.9.2 In-band and out-of-band . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

© Copyright IBM Corp. 2009. All rights reserved. iii

2.10 Disaster recovery considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.11 IBM Encryption Key Manager to Tivoli Key Lifecycle Manager migration . . . . . . . . . . 38
2.12 Tivoli Key Lifecycle Manager configuration planning checklist . . . . . . . . . . . . . . . . . . 38
2.13 Tivoli Key Lifecycle Manager planning quick reference . . . . . . . . . . . . . . . . . . . . . . . 40
2.13.1 Other resources that can help with the planning process . . . . . . . . . . . . . . . . . . 40

Chapter 3. Tivoli Key Lifecycle Manager installation . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.1 Installation overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.2 Solution components. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.2.1 Tivoli Key Lifecycle Manager for z/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.2.2 IBM DB2 for z/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.2.3 IBM System Services Runtime Environment for z/OS, Resource Recovery Service,
and Integrated Solutions Console. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.2.4 RACF/SAF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.2.5 ICSF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.2.6 SMF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.3 z/OS System Services Runtime Environment installation and configuration . . . . . . . . 49
3.3.1 System Services Runtime Environment installation and configuration overview . 50
3.3.2 Preparing the host system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.3.3 Create System Services Runtime Environment configuration file. . . . . . . . . . . . . 57
3.3.4 Creating a System Services Runtime Environment instance . . . . . . . . . . . . . . . . 61
3.3.5 Verify the System Services Runtime Environment configuration . . . . . . . . . . . . . 63
3.4 Tivoli Key Lifecycle Manager installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.4.1 Tivoli Key Lifecycle Manager installation overview . . . . . . . . . . . . . . . . . . . . . . . . 65
3.4.2 SMP/E install Tivoli Key Lifecycle Manager and SMP/E install Tivoli Key Lifecycle
Manager Fix Pack 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.4.3 Host system requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.4.4 System Services Runtime Environment configuration changes . . . . . . . . . . . . . . 68
3.4.5 Install Tivoli Key Lifecycle Manager product tar file created during the SMP/E
install. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.4.6 Run DB2 SPUFI scripts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.4.7 Create the Tivoli Key Lifecycle Manager response file by running the
createResponseFile.sh script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.4.8 Install Tivoli Key Lifecycle Manager by running the installTKLM.sh script . . . . . . 80
3.4.9 Perform post installation steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.4.10 Stop and restart System Services Runtime Environment . . . . . . . . . . . . . . . . . . 85
3.4.11 Verify installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
3.5 Defining a master keystore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
3.5.1 Create RACF profiles for JCERACFKS or JCECCARACFKS keystores . . . . . . . 86
3.5.2 Define the keystore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
3.6 Deploying additional Tivoli Key Lifecycle Manager servers in a Sysplex . . . . . . . . . . . 88
3.6.1 Install System Services Runtime Environment on a second LPAR . . . . . . . . . . . 89
3.6.2 Install Tivoli Key Lifecycle Manager on the second LPAR . . . . . . . . . . . . . . . . . . 90
3.6.3 Back up the primary Tivoli Key Lifecycle Manager server . . . . . . . . . . . . . . . . . . 90
3.6.4 Restore the primary Tivoli Key Lifecycle Manager backup to the second Tivoli Key
Lifecycle Manager server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
3.6.5 Shut down and restart the second Tivoli Key Lifecycle Manager server. . . . . . . . 90
3.7 Managing the SSRECFG user ID password . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

Chapter 4. Tivoli Key Lifecycle Manager backup and restore. . . . . . . . . . . . . . . . . . . . 93
4.1 Backup and restore of Tivoli Key Lifecycle Manager data . . . . . . . . . . . . . . . . . . . . . . 94
4.2 Backup procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.2.1 Backing up Tivoli Key Lifecycle Manager configuration data . . . . . . . . . . . . . . . . 95

iv IBM Tivoli Key Lifecycle Manager for z/OS

4.2.2 Backing up DB2 tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.2.3 Backing up a JCEKS keystore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.2.4 Backing up a JCERACFKS keyring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.2.5 Backing up a JCECCARACFKS keyring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
4.2.6 Backing up ICSF datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
4.3 Restore procedures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
4.3.1 Restoring Tivoli Key Lifecycle Manager configuration data . . . . . . . . . . . . . . . . 100
4.3.2 Restoring DB2 Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.3.3 Restoring a JCEKS keystore. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.3.4 Restoring a JCKRACFKS keyring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
4.3.5 Restoring a JCECCARACFKS keyring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
4.3.6 Restoring ICSF datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

Appendix A. Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
A.1 Problems with System Services Runtime Environment installation and configuration 108
A.1.1 +BBOJ0095W: JAVA VERSION/LEVEL IS NOT SUPPORTED BY WEBSPHERE
FOR Z/OS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
A.1.2 Problem starting up System Services Runtime Environment: INSUFFICIENT
AUTHORITY TO OPEN applyPTF.sh. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
A.1.3 RACF ICH408I permission messages for SSRECFG and SSREADM. . . . . . . . 109
A.1.4 System Services Runtime Environment PDSE is not APF authorized . . . . . . . . 109
A.1.5 System Services Runtime Environment PDSE is not cataloged . . . . . . . . . . . . 109
A.1.6 System Services Runtime Environment file system is not mounted or the path is
incorrect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
A.1.7 System Services Runtime Environment was started but modifySSRE.sh or
equivalent security setup commands were not executed . . . . . . . . . . . . . . . . . . 110
A.1.8 Trying to start System Services Runtime Environment but the Configuration file
system is not mounted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
A.1.9 Multiple browsers windows are logged into the same System Services Runtime
Environment instance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
A.1.10 Unable to resolve the System Services Runtime Environment hostname and get to
the ISC admin console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
A.1.11 Unable to make updates on the Tivoli Key Lifecycle Manager GUI . . . . . . . . . 111
A.1.12 Security errors from running the System Services Runtime Environment
scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
A.1.13 Cell name and port number conflicts with System Services Runtime
Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
A.1.14 System Services Runtime Environment errors, abends, hang conditions . . . . 111
A.1.15 Collecting data for IBM support center when opening a PMR . . . . . . . . . . . . . 113
A.1.16 Additional diagnostic requests by IBM support center . . . . . . . . . . . . . . . . . . . 114
A.1.17 Taking a console dump of System Services Runtime Environment . . . . . . . . . 114
A.1.18 Dynamic tracing with ISC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
A.1.19 Dynamic tracing using Modify. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
A.2 Additional resources for troubleshooting System Services Runtime Environment
configuration problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
A.2.1 First failure data capture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
A.2.2 Garbage collection tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
A.2.3 Debugging applications via RAD V7 (prior to deploying on z/OS) . . . . . . . . . . . 119
A.2.4 z/OS Debugging tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
A.2.5 Additional diagnostic references. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
A.3 System Services Runtime Environment runtime logs . . . . . . . . . . . . . . . . . . . . . . . . . 120
A.3.1 How to view logs in TSO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
A.3.2 How to create a data set from logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

Contents v

A.3.3 How to retrieve logs via FTP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
A.4 System Services Runtime Environment application deployment problems . . . . . . . . 120
A.4.1 Application not correctly signed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
A.5 Java problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
A.5.1 Generating additional trace information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
A.6 Problems during the Tivoli Key Lifecycle Manager post SMP/E install. . . . . . . . . . . . 121
A.6.1 Locating Tivoli Key Lifecycle Manager log files . . . . . . . . . . . . . . . . . . . . . . . . . 121
A.6.2 Unable to allocate memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
A.6.3 Out of disk space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
A.6.4 Using wrong user ID to execute Tivoli Key Lifecycle Manager post SMP/E
scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
A.6.5 Not having the correct permissions set up on the
TKLM_POST_SMPE_INSTALL_HOME directory and its contents . . . . . . . . . . 122
A.6.6 Not having correct permission and ownership values on the System Services
Runtime Environment config hfs container . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
A.6.7 Tivoli Key Lifecycle Manager post SMP/E install script return codes . . . . . . . . . 123
A.7 General errors resulting from the Tivoli Key Lifecycle Manager post SMP/E Install. . 130
A.7.1 *** SSL SIGNER EXCHANGE PROMPT *** SSL signer from target host null is not
found in trust store safkeyring:///WASKeyring.System Services Runtime
Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
A.7.2 FSUM7343 cannot open "/SYSTEM/tklmProductInstall/logs/.output" for output:
EDC5111I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
A.7.3 Attempting to run the bin/migrateEKM.sh, bin/installTKLM.sh or
bin/uninstallTKLM.sh script while System Services Runtime Environment is already
and running. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
A.7.4 Using an unauthorized user to run the Tivoli Key Lifecycle Manager post SMP/E
install scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
A.7.5 Tivoli Key Lifecycle Manager product files are not synchronized with Tivoli Key
Lifecycle Manager database in DB2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
A.7.6 Trying to use a hardware keystore but the IBMJCECCA provider not specified in the
java.security file within System Services Runtime Environment's embedded
Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
A.7.7 Forgot to install the Java unrestricted policy files . . . . . . . . . . . . . . . . . . . . . . . . 134
A.7.8 Attempting to create a file-based keystore in a path that does not exist . . . . . . 134
A.7.9 Attempting to create a file-based keystore in a read only directory . . . . . . . . . . 135
A.7.10 Attempting to create a file-based keystore in a directory that the SSREGRP group
does not have authority to write to . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
A.8 Problems configuring Tivoli Key Lifecycle Manager . . . . . . . . . . . . . . . . . . . . . . . . . . 135
A.8.1 Kicked out of ISC console and Tivoli Key Lifecycle Manager panels because the
"Session has become invalid". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
A.8.2 Tivoli Key Lifecycle Manager panel pops up in a second browser window . . . . 136
A.8.3 DB2 is not active: CODE=-4499, SQLSTATE=08001DSRA0010E: SQL State =
08001, Error Code = -4,499 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
A.8.4 CTGKM0597E - Error occurred while generating the secret key . . . . . . . . . . . . 136
A.8.5 WebSphere transaction timed out: BBOO0222I: WTRN0006W. . . . . . . . . . . . . 136
A.8.6 Problems starting System Services Runtime Environment: BBOO0222I: J2CA0090I
when starting System Services Runtime Environment . . . . . . . . . . . . . . . . . . . . 137
A.8.7 Lexical error when running Tivoli Key Lifecycle Manager CLI commands
from OMVS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
A.8.8 IRR.RAUDITX Access Errors due to RACF setup for Tivoli Key Lifecycle Manager
auditing not being performed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
A.8.9 Unable to authenticate to Tivoli Key Lifecycle Manager MBeans: BBOO0222I:
SECJ0305I in the servant job log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

vi IBM Tivoli Key Lifecycle Manager for z/OS

A.8.10 DB2's WLM Environment has stopped: SQLCODE: -471, SQLSTATE: 55023 140
A.8.11 Unable to import certificates into RACF using the Tivoli Key Lifecycle Manager
import function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
A.8.12 Tivoli Key Lifecycle Manager has a known problem with SSL certificates using
mixed case alias names . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
A.8.13 Tivoli Key Lifecycle Manager panel pops up and creates 2nd active windows for the
Tivoli Key Lifecycle Manager GUI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
A.8.14 Status message on Tivoli Key Lifecycle Manager indicates that I'm ready to serve
keys however my device can't make a connection . . . . . . . . . . . . . . . . . . . . . . . 141
A.8.15 Unable to update the Tivoli Key Lifecycle Manager configuration after recycling
System Services Runtime Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
A.8.16 Receiving NOT AUTHORIZED error messages when running the
samples/racfpermissions.rexx script to setup permissions to my RACF keyring 144
A.9 Information to gather when Tivoli Key Lifecycle Manager deployment fails . . . . . . . . 144
A.10 Enabling System Services Runtime Environment trace . . . . . . . . . . . . . . . . . . . . . . 145
A.11 Enabling Tivoli Key Lifecycle Manager trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

Appendix B. Basics of cryptography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
B.1 Introduction to cryptography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
B.2 Cryptographic algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
B.2.1 Symmetric key algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
B.2.2 Asymmetric key algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
B.3 Padding. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
B.4 Encryption modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
B.5 Hybrid encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
B.6 Digital signatures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
B.7 Digital certificates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
IBM Redbooks publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
How to get Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

Contents vii

viii IBM Tivoli Key Lifecycle Manager for z/OS

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area. Any
reference to an IBM product, program, or service is not intended to state or imply that only that IBM product,
program, or service may be used. Any functionally equivalent product, program, or service that does not
infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not give you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such
provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION
PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR
IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT,
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of
express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any
manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the
materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring
any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to the names and addresses used by an actual business
enterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrate programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs.

© Copyright IBM Corp. 2009. All rights reserved. ix

Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines
Corporation in the United States, other countries, or both. These and other IBM trademarked terms are
marked on their first occurrence in this information with the appropriate symbol (® or ™), indicating US
registered or common law trademarks owned by IBM at the time this information was published. Such
trademarks may also be registered or common law trademarks in other countries. A current list of IBM
trademarks is available on the Web at http://www.ibm.com/legal/copytrade.shtml

The following terms are trademarks of the International Business Machines Corporation in the United States,
other countries, or both:
AIX® Rational® VTAM®
DB2® Redbooks® WebSphere®
DS8000® Redbooks (logo) ® z/OS®
FICON® System p® z/VM®
IBM® System Storage™ z/VSE™
Language Environment® System z9® z9®
OS/390® System z® zSeries®
Parallel Sysplex® Tivoli®
RACF® TotalStorage®

The following terms are trademarks of other companies:

SUSE, the Novell logo, and the N logo are registered trademarks of Novell, Inc. in the United States and other
countries.

Red Hat, and the Shadowman logo are trademarks or registered trademarks of Red Hat, Inc. in the U.S. and
other countries.

SAP, and SAP logos are trademarks or registered trademarks of SAP AG in Germany and in several other
countries.

J2EE, Java, Java runtime environment, JDBC, JVM, Solaris, Sun, Sun Java, ZFS, and all Java-based
trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Windows Server, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United
States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

x IBM Tivoli Key Lifecycle Manager for z/OS

Preface

This IBM® Redbooks® publication provides details of a new offering called IBM Tivoli® Key
Lifecycle Manager. We introduce the product, provide planning suggestions, and detail the
installation of IBM Tivoli Key Lifecycle Manager on the z/OS® operating system running on a
System z® server.

Tivoli Key Lifecycle Manager is IBM’s latest storage device encryption solution. It allows
enterprises to create, manage, back up, and distribute their cryptographic key material from a
single control point. Tivoli Key Lifecycle Manager has evolved from the existing IBM
Encryption Key Manager solution. Unlike IBM Encryption Key Manager, which only provided
a key server, Tivoli Key Lifecycle Manager provides real key management, security policy
capabilities, and a Web-based user interface for ease of use. It leverages the existing security
strengths of the z/OS platform by using Integrated Cryptographic Services Facility (ICSF),
System Authorization Facility (SAF), and Java™-based keystores to store all the key
material.

The team who wrote this paper
This paper was produced by a team of specialists from around the world working at the
International Technical Support Organization, Poughkeepsie Center.

Karan Singh is a Project Leader with the International Technical Support Organization
(ITSO) in Poughkeepsie, NY. His areas of expertise include core z/OS technologies.

Steven Hart is a Staff Software Engineer who has worked for IBM Systems and Technology
group for 6 years. He is a Certified Information Systems Security Professional who has
worked in the design, development, function test, and service phases for critical z/OS security
software, such as Trusted Key Entry and Encryption Facility. As the Tivoli Key Lifecycle
Manager for z/OS Team Lead, Steve led the z/OS team to successful completion of Tivoli Key
Lifecycle Manager for z/OS V1.

William C. Johnston is experienced in working with large system installations to deploy
encryption key management solutions, including performing enterprise system security
assessments, educating client teams on security-related topics, and bringing “best practices”
to client processes. For more than ten years he was responsible for the design and
implementation of the test approach definitions for security-related elements of the z/OS
operating system, including their interaction with other components, the base OS, and other
platforms such as Linux® and Windows® XP. Prior to that, he performed code development,
functional and system level testing, and project management duties.

Lynda Kunz is an IT Architect experienced in architecting and deploying encryption solutions
for large systems. Her current areas of infrastructure expertise include large scale tape and
encryption solutions. Her past experience includes code design and development on a variety
of IBM products including LE, AOC, VM and VTAM®, z/OS Project Office and IBM
Management.

Irene Penney is a Certified IT Architect in Poughkeepsie, NY. She has over 26 years of
experience in various areas of IT support. She is currently in the Optimization team within the
CIO Organization. Her areas of expertise include infrastructure, particularly System p®, and

© Copyright IBM Corp. 2009. All rights reserved. xi

SAP® Architecture and infrastructure. She also has extensive experience with SAP Basis
and AIX®, VM and MVS Systems Administration and Operations.

Thanks to the following people for their contributions to this project:

Rich Conway, Bob Haimowitz
International Technical Support Organization, Poughkeepsie Center

Jonathan Barney, Tom Benjamin, John Dayka, James Ebert, Krishna Yellepeddy
IBM

Become a published author
Join us for a two- to six-week residency program! Help write a book dealing with specific
products or solutions, while getting hands-on experience with leading-edge technologies. You
will have the opportunity to team with IBM technical professionals, Business Partners, and
Clients.

Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you
will develop a network of contacts in IBM development labs, and increase your productivity
and marketability.

Find out more about the residency program, browse the residency index, and apply online at:
ibm.com/redbooks/residencies.html

Comments welcome
Your comments are important to us!

We want our papers to be as helpful as possible. Send us your comments about this paper or
other IBM Redbooks publications in one of the following ways:
Use the online Contact us review Redbooks form found at:
ibm.com/redbooks
Send your comments in an e-mail to:
redbooks@us.ibm.com
Mail your comments to:
IBM Corporation, International Technical Support Organization
Dept. HYTD Mail Station P099
2455 South Road
Poughkeepsie, NY 12601-5400

xii IBM Tivoli Key Lifecycle Manager for z/OS

1

Chapter 1. Introduction
This chapter introduces Tivoli Key Lifecycle Manager.

© Copyright IBM Corp. 2009. All rights reserved. 1

1.1 Tivoli Key Lifecycle Manager
Tivoli Key Lifecycle Manager provides you a simplified key management solution that is easy
to install, deploy, and manage. Tivoli Key Lifecycle Manager allows you to create, back up,
and manage the keys and certificates your enterprise uses. Through its graphical and
command line interfaces you can manage symmetric keys, asymmetric keys, and certificates.

Tivoli Key Lifecycle Manager provides:
Key serving with lifecycle management using a graphical user interface and a command
line interface.
Support for encryption-enabled IBM System Storage™ TS1100 Family Tape Drives (3592
tape drives).
Support for IBM Systems Storage Linear Tape-Open (LTO) Ultrium Generation 4 Tape
Drives.
Support for the DS8000® Storage Controller (IBM System Storage DS8000 Turbo drive).
This support requires the appropriate microcode bundle version on the DS8000 Storage
Controller, Licensed Internal Code level 64.2.xxx.0 or higher.
Backup and recovery to protect your keys and certificates.
Notification on expiration of certificates.
Audit records to allow you to track the encryption of your data.
Support for RACF® and ICSF protected keystores.
Auto roll-over of key groups and certificates. This capability applies to 3592 and LTO
drives; it does not apply to DS8000. Provides key life-cycle management function that
allows a user to define when a new key group should be used with LTO drives or new
certificates with 3592 drives.

While other encryption solutions require processor power, encryption using Tivoli Key
Lifecycle Manager in concert with IBM encryption-capable tape and disk drives is done with
little or no impact on performance. You can easily exchange encrypted tapes with your
business partners or data centers that have the necessary key information to decrypt the
data.

With the introduction of the Tivoli Key Lifecycle Manager, IBM has made available the next
generation of Key Manager software to enable serving keys to encrypting drives. Tivoli Key
Lifecycle Manager is intended to give a consistent look and feel for Key Management tasks
across the brand, while simplifying those same key management tasks.

Tivoli Key Lifecycle Manager and IBM encryption-capable tape drives provide high
performance data encryption. Encryption is performed by the tape drive hardware at native
drive speeds. It also supports encryption of large amounts of tape data for backup and
archive purposes. Utilizing the TS1130 Tape Drive, TS1120 Tape Drive, or LTO4 Tape Drive
offers a cost-effective solution for tape data encryption by offloading encryption tasks from
servers, leveraging existing tape infrastructure incorporated in standard IBM Tape Libraries,
and eliminating the need for unique appliance hardware.

Tivoli Key Lifecycle Manager and the DS8000 drives provide high performance data
encryption for all your data on disk. Encryption is performed by the disk drive hardware at
native drive speeds, providing economical encryption for large amounts of data on disk.
Utilizing the DS8000 disk drives to encrypt your data provides a cost-effective solution for disk
data encryption by offloading encryption tasks from the servers, leveraging existing disk
infrastructure and eliminating the need for unique appliance hardware.

2 IBM Tivoli Key Lifecycle Manager for z/OS

Adding encryption to the enterprise by using IBM encrypting devices and Tivoli Key Lifecycle
Manager is transparent to the applications and operations using the devices and therefore
adds valuable security and loss prevention for data without expensive changes to the
applications or operations procedure.

See Appendix B, “Basics of cryptography” on page 149 for an overview of cryptographic
concepts.

1.2 How tape encryption works
Encryption, implemented in the tape drive, encrypts the data before it is written to the
cartridge. When tape compression is enabled, the tape drive first compresses the data then
encrypts it. This means that there is no loss of capacity with IBM Tape Encryption. If the
encryption solution encrypts the data first, then the tape drive tries to compress the data,
there will be very little space saved because encrypted data does not compress well.

To encrypt the data, the tape drive needs a key. This key is provided by Tivoli Key Lifecycle
Manager in an encrypted form to make the Tape Encryption solution secure.

Figure 1-1 summarizes the process flow for Tape Encryption using TS1130 and TS1120.

1. Load cartridge, specify
encryption

Encryption 2. Tape drive requests a data key
Key
Manager Encrypted “Data Key”

5. Tape drive writes encrypted
3. Key manager 4.Encrypted keys data and stores encrypted data
generates key and transmitted to tape drive key on cartridge
encrypts it

Encrypted “Data Keys”

Figure 1-1 TS1120 and TS1130 Tape Encryption process flow

Figure 1-2 on page 4 summarizes the LTO4 Tape Encryption process flow.

Chapter 1. Introduction 3

1. Load cartridge, specify
encryption

Encryption 2. Tape drive requests a data key
Key
Manager
5. Tape drive decrypts the data
key, writes encrypted data and
3. Key manager keyid on the cartridge
4.Encrypted data key
retrieves key and transmitted to tape drive
encrypts it for
transmission

LTO 4 Encryption
Encrypted “Data Key”

Figure 1-2 LTO4 Tape Encryption process

1.3 How DS8000 encryption works
Encryption, implemented in the disk drive, encrypts the data before it is written to the disk.
When compression is enabled, the disk drive first compresses the data to be written, then
encrypts it. This means that there is no loss of capacity with IBM Disk Encryption. If the
encryption solution encrypted the data first, then tried to compress it, there would be little
space savings because encrypted data does not compress well.

To encrypt the data, the disk drive needs a key. This key is provided by Tivoli Key Lifecycle
Manager in an encrypted form to make the Disk Encryption solution secure.

When a DS8000 is installed the protected AES key is requested from Tivoli Key Lifecycle
Manager. This key is used to wrap and unwrap the keys the DS8000 will use to encrypt the
data on disk. Unlike tape, the AES key request from Tivoli Key Lifecycle Manager is a one
time occurrence and is used to wrap all the data keys used by this disk. When sent from Tivoli
Key Lifecycle Manager to the DS8000, the AES key is wrapped with a different key for secure
transfer back to the DS8000 where it is stored.

Figure 1-3 on page 5 summarizes the process flow for Disk Encryption using a DS8000.


Tivoli Key Lifecycle Manager

1) Power on DS8000
2) Request unlock key from TKLM

3) Key manager
generates key and
encrypts (wraps) it
4) Encrypted (wrapped) key is sent back to the DS8000

5) DS8000 unwraps key.
Data is encrypted when written
to disk, and decrypted when
read from disk

Figure 1-3 DS8000 Turbo drive encryption process

1.4 Why use Tivoli Key Lifecycle Manager and Tape/DS8000
encryption
Tape and disk encryption is used to hide and protect sensitive data. If a retired DS8000 unit
or tape cartridge leaves the data centers, the data is no longer protected through Resource
Access Control Facility (RACF) or similar access protection mechanisms. Tape and DS8000
encryption will secure the data and can help you fulfill security regulations.

Important and sensitive data can be protected in many ways. Data can be encrypted by
means of special software programs, hardware adapters, hardware appliances, or by the
tape/disk drive as the data is written. Encrypting data with software programs utilizes
processor power, and encrypting data with hardware appliances requires additional
investment in hardware. Using the disk or tape drive needed to write the data on media
provides encryption in a cost-effective manner.

One of the advantages of IBM Tape and DS8000 Encryption is that the data is encrypted after
compression. This saves space on tape cartridges and disk drives, thus sparing the cost of
additional hardware investments. Data on cartridges does not have to be “degaussed” or
overwritten with patterns of x’FF’ at the end of life of the cartridge, which will provide a cost
savings when the tape cartridge or disk reaches end of life. This is true for both Write Once
Read Many (WORM) cartridges and normal tape cartridges. DS8000 units, with the use of
encryption, can have disk drives replaced or discarded without removing the data contained
on the unit, thus saving time and money.

Additionally, a clever use of encryption is for data shredding. If you delete an encryption key,
all the data that encryption key protected becomes, in effect, garbage. This use of the feature
requires extreme care. You need to know exactly what data was encrypted with the key you
are deleting. Remember that without the key you cannot decrypt the data.


Finally, one of the most important aspects of using Tivoli Key Lifecycle Manager with IBM
encryption-capable devices is transparent encryption. An enterprise gains the ability to
secure data without having to make costly changes to the code of existing applications that
use the devices or to the existing operations procedures. With IBM encryption-capable
devices and Tivoli Key Lifecycle Manager, a security administrator can quickly and easily set
up the encrypting environment and turn on encryption without having to make any other
changes to the applications or procedures.

1.5 Encryption key management
A large number of symmetric keys, asymmetric keys, and certificates can exist in your
enterprise. All of these keys and certificates need to be managed. Key management can be
handled either internally by an application, such as Tivoli Storage Manager, or externally by
an Key Manager such as IBM Encryption Key Manager or Tivoli Key Lifecycle Manager.

The Tivoli Key Lifecycle Manager product is an application that will perform key management
tasks for IBM encryption-enabled hardware (for example, the IBM encryption-enabled
TS1100 family of tape drives, Linear Tape-Open (LTO) Ultrium 4 tape drives, and the
DS8000 Turbo drives) by providing, protecting, storing, and maintaining encryption keys that
are used to encrypt information being written to, and decrypt information being read from,
tape and disk media. Tivoli Key Lifecycle Manager operates on a variety of operating
systems. Currently, the supported operating systems are:

Supported with initial release installed:
AIX 5.3 64-bit1
AIX 6.1 64-bit1
Red Hat® Enterprise Linux 4 32-bit
Solaris™ 10 SPARC 64-bit1
SUSE® Linux Enterprise Server 9 32-bit
SUSE Linux Enterprise Server 10 32-bit
Windows Server® 2003 R2 32-bit
z/OS Version 1 Release 9 or later

Supported with fix pack 1 installed
Red Hat Enterprise Linux 5 32-bit
Red Hat Enterprise Linux 5 64-bit1
Solaris 9 SPARC 64-bit1
SUSE Linux Enterprise Server 10 64-bit1
Windows Server 2003 64-bit1 . Requires both new installation image and Fix Pack 1 (or
later).
Windows Server 2008 32-bit. Requires both new installation image and Fix Pack 1 (or
later).
Windows Server 2008 64-bit1 . Requires both new installation image and Fix Pack 1 (or
later).

Tivoli Key Lifecycle Manager is designed to be a shared resource deployed in several
locations within an enterprise. It is capable of serving numerous IBM encrypting tape and
1 Tivoli Key Lifecycle Manager runs as a 32-bit application on 64-bit operating systems.


DS8000 drives regardless of where those drives reside (for example, in tape library
subsystems, connected to mainframe systems through various types of channel connections,
or installed in other computing systems).

1.5.1 Tivoli Key Lifecycle Manager services
You can use Tivoli Key Lifecycle Manager to manage encryption keys and certificates. Tivoli
Key Lifecycle Manager allows you to create, back up, and manage the lifecycle of keys and
certificates that your enterprise uses. This includes the management of symmetric keys,
asymmetric keys, and certificates. Tivoli Key Lifecycle Manager waits for and responds to key
generation or key retrieval requests that arrive through TCP/IP communication for a tape
library, tape controller, tape subsystem, device drive, tape drive, or DS8000 drive. Tivoli Key
Lifecycle Manager provides you with additional functions beyond those offered in the
previous IBM key management product (IBM Encryption Key Manager), including:
Lifecycle functions
– Notification of certificate expiration
– Automated rotation of certificates
– Automated rotation of groups of keys
Usability enhancements
– Provides a graphical user interface
– Initial configuration wizards
– Migration wizards
– Provides a command line interface through WSAdmin
Integrated backup and restore of Tivoli Key Lifecycle Manager file
– One button to create and restore a single backup packaged as a jar file
Security policy
– Leverages the Security Infrastructure of the IBM System Services Runtime
Environment
Audit enhancements
– Provides audit records in SMF Type 83 sub-type 6 format

DB2
Tivoli Key Lifecycle Manager stores the drive table in DB2®, giving the user a more robust
interface for managing drives and the keys and certificates that are associated with those
drives. With IBM Encryption Key Manager, the previous key management product, the only
place to determine the key used to encrypt a tape cartridge, and similar audit information, was
in the IBM Encryption Key Manager audit log and the IBM Encryption Key Manager
metadata.xml file. With Tivoli Key Lifecycle Manager this information is stored in the Tivoli
Key Lifecycle Manager DB2 tables, enabling the user to search and query that information
with ease.

Tip: The option to automatically accept unknown tape drives can facilitate the task of
populating the drive table with your drives. For security reasons, you might want to turn off
this option as soon as all of your drives have been added to the table. In a business and
continuity recovery site, however, it may be required to accept unknown tape drives.

Configuration file
Tivoli Key Lifecycle Manager also has an editable configuration file with additional
configuration parameters that are not accessible through the GUI. The file can be text edited.


However, the preferred method is modifying the file through the Tivoli Key Lifecycle Manager
command line interface (CLI).

Java security keystore
The keystore is defined as part of the Java Cryptography Extension (JCE) and is an element
of the Java Security components, which are, in turn, part of the Java Runtime Environment. A
keystore holds the certificates and keys (or pointers to the certificates and keys) used by
Tivoli Key Lifecycle Manager to perform cryptographic operations. A keystore can be either
hardware-based or software-based.

Tivoli Key Lifecycle Manager supports several types of Java keystores, offering a variety of
operational characteristics to meet your needs.

Tivoli Key Lifecycle Manager on distributed systems
Tivoli Key Lifecycle Manager on distributed systems supports the JCEKS keystore. This
keystore supports both symmetric keys and asymmetric keys. Symmetric keys are used for
LTO 4 encryption drives, while asymmetric keys are used for the TS1100 family of tape drives
and the DS8000 drives.

Cryptographic services
Tivoli Key Lifecycle Manager uses the IBM Java Security components for its cryptographic
capabilities. Tivoli Key Lifecycle Manager does not provide cryptographic capabilities and
therefore does not require, nor is it allowed to obtain, FIPS 140-2 certification. However, Tivoli
Key Lifecycle Manager takes advantage of the cryptographic capabilities of the IBM Java
Virtual Machine in the IBM Java Cryptographic Extension component and allows the selection
and use of the IBMJCEFIPS cryptographic provider, which has a FIPS 140-2 level 1
certification. By setting the FIPS configuration parameter to ON in the Configuration
Properties file, either through text editing or using the Tivoli Key Lifecycle Manager CLI, you
can make Tivoli Key Lifecycle Manager use the IBMJCEFIPS provider for all cryptographic
functions.

For more information about the IBMJCEFIPS provider, its selection and use, see:
http://www.ibm.com/developerworks/java/jdk/security/50/FIPShowto.html

1.5.2 Key exchange
Tivoli Key Lifecycle Manager acts as a process awaiting key generation or key retrieval
requests sent to it through a TCP/IP communication path between Tivoli Key Lifecycle
Manager and the tape library, tape controller, tape subsystem, device driver, tape drive, or
DS8000 drive. When a drive writes encrypted data, it first requests an encryption key from
Tivoli Key Lifecycle Manager. The tasks that the Tivoli Key Lifecycle Manager performs upon
receipt of the request are different for the asymmetric keys used by the TS1100 family of tape
drives and the DS8000 drives, and symmetric keys used by the TS1040 tape drive.

Asymmetric and symmetric keys
Tivoli Key Lifecycle Manager requests an Advanced Encryption Standard (AES) key from the
cryptographic services and serves it to the drives in one of the following forms:
Encrypted or wrapped, using Rivest-Shamir-Adleman (RSA) key pairs. This form is used
for the TS1100 family of tape drives and the DS8000 drives.


Separately wrapped for secure transfer to the tape drive, where it is unwrapped upon
arrival and the key inside is used to encrypt the data being written to tape. This form is
used for the TS1040 tape drives.
Additionally, the libraries now support SSL-encrypted connections between the Tivoli Key
Lifecycle Manager and library for key exchanges. When SSL is not used for key
exchange, the key material will be encrypted in another fashion. The transport of the keys
is always secure across the TCP/IP connection.

Note: For z/OS systems at or below Integrated Cryptographic Services Facility version
7740, the zOSCompatibility flag should be set in the Tivoli Key Lifecycle Manager
configuration file. This setting can be turned on using either the Tivoli Key Lifecycle
Manager CLI or by editing the Tivoli Key Lifecycle Manager configuration file. When
true is specified, Triple Data Encryption Standard (Triple DES or DESede) symmetric
keys are used instead of AES symmetric keys.

TS1100 family of tape drives and DS8000
When an encrypted tape cartridge is read by a TS1100 tape drive, the protected AES key on
the tape is sent to Tivoli Key Lifecycle Manager, where the wrapped AES key is unwrapped.
The AES key is then wrapped with a different key for secure transfer back to the tape drive,
where it is unwrapped and used to decrypt the data stored on the tape. Tivoli Key Lifecycle
Manager also allows protected AES keys to be rewrapped, or rekeyed, using different RSA
keys from the original keys that were used when the tape was written. Rekeying is useful
when an unexpected need arises to export volumes to business partners whose public keys
were not included; it eliminates the need to rewrite the entire tape and enables a tape
cartridge’s data key to be reencrypted with a business partner’s public key.

Rekeying of the DS8000 is currently not available and would require a complete
re-initialization of the drive.

LTO Ultrium 4 tape drives
The Tivoli Key Lifecycle Manager fetches an existing AES key from a keystore and wraps it
for secure transfer to the tape drive, where it is unwrapped upon arrival and used to encrypt
the data being written to tape.

When an encrypted tape is read by an LTO Ultrium 4 tape drive, the Tivoli Key Lifecycle
Manager fetches the required key from the keystore, based on the information in the Key ID
on the tape, and serves it to the tape drive wrapped for secure transfer.

1.6 Encryption key methods
Tape methods
There are three methods of tape encryption management supported by the IBM Tape
Encryption solution. These methods differ in where the encryption policy engine resides,
where key management is performed, and how Tivoli Key Lifecycle Manager is connected to
the drive. Encryption policies control which volumes need to be encrypted.

Key management and the encryption policies can be located in any one of the following
environmental layers:
System layer
Library layer
Application layer


In accordance with the layers we call these methods:
System-managed encryption (SME)
Library-managed encryption (LME)
Application-managed encryption (AME)

Only two of these methods, SME and LME, require the implementation of an external
component, the Tivoli Key Lifecycle Manager, to provide and manage keys. With AME, key
provisioning and key management are handled by the application. All three methods allow
you to specify which tape cartridges will be encrypted and which will not.

Not all operating systems, applications, and tape libraries support all of these methods, and
where they are supported, not all of the methods are equally suitable. When you plan for tape
encryption, select the encryption method depending on your operating environment. In the
following sections, we explain the characteristics of AME, SME, and LME.

DS8000 methods
Full Disk Encryption (FDE) is provided for the DS8000. All data on the disk will be encrypted.

1.6.1 System-managed encryption
In a system-managed encryption (SME) implementation, encryption policies reside within the
system layer. This method of tape encryption requires a key server (Tivoli Key Lifecycle
Manager) for key management. SME is fully transparent to the application and library layers.
Figure 1-4 on page 11 shows an illustration of system-managed encryption.

System-managed encryption is supported on z/OS, z/VM®, z/VSE™, z/TPF, zLinux, and a
number of distributed system platforms. On z/OS, z/VM, z/VSE, z/TPF, and zLinux,
system-managed encryption is the only encryption method supported. SME is supported on
z/OS using Data Facility Storage Management Subsystem (DFSMS). On distributed systems
platforms, the IBM tape device driver is used for specifying encryption policies on a per-drive
basis.

The following distributed systems operating systems are currently supported:
AIX
Windows
Linux
Solaris

System-managed encryption offers you centralized enterprise-class key management, which
facilitates tape interchange and migration. Another advantage is its support for stand-alone
drives. The drawbacks of SME are its policy granularity on distributed systems, additional
responsibilities for the storage administrator, and the dependency of data access on the
availability of the key server and the key path.

SME shares most of its advantages and disadvantages with library-managed encryption
(LME), but there are two major differences. Naturally, LME does not support stand-alone tape
drives. However, in a distributed systems environment, LME gives you better policy
granularity than SME because you can control encryption on a per-volume basis with TS3500
and 3494 tape libraries. On z/OS, you can control encryption on the volume level through the
use of DSMFS.

In a System z environment that does not support encryption, or in an distributed systems
environment with stand-alone drives and an application that does not support encryption,
SME is the only choice. In all other environments, consider LME as an alternative.


Application
Layer

Tivoli Key
Lifecycle
Manager Policy
System
Layer

Library
Layer

Figure 1-4 System-managed encryption (SME)

System-managed encryption for distributed systems
Encryption policies specifying when to use encryption are set up in the IBM tape device
driver. For details about setting up system-managed encryption on tape drives in a distributed
systems environment, refer to the IBM Tape Device Driver Installation and User’s Guide,
GC27-2130, and the Planning and Operator Guide for your tape library.

On distributed systems, this support can be described as in-band, meaning tape drive
requests to the Tivoli Key Lifecycle Manager component travel over the Fibre Channels to the
server hosting the Tivoli Key Lifecycle Manager.

System-managed encryption for System z
On z/OS, policies specifying when to use encryption are set up in DFSMS. You can also use
additional software products, such as IBM Integrated Cryptographic Service Facility (ICSF)
and IBM Resource Access Control Facility (RACF). Key generation and management is
performed by the Tivoli Key Lifecycle Manager, running on the host or externally on another
host. Policy controls and keys pass through the data path between the system layer and the
encrypting tape drives. Encryption is transparent to the applications.

For TS1120 tape drives that are connected to an IBM Virtualization Engine TS7700,
encryption key labels are assigned using the Maintenance Interface on a per-storage-pool
basis. DFSMS storage constructs are used by z/OS to control the use of storage pools for
logical volumes, resulting in an indirect form of encryption policy management. For more
information, refer to the white paper, IBM Virtualization Engine TS7700 Series Encryption
Overview, which is available at:
http://www.ibm.com/support/docview.wss?&uid=ssg1S4000504

For details about setting up system-managed encryption on the TS1120 tape drive in a
System z platform environment, refer to z/OS DFSMS Software Support for IBM System
Storage TS1120 Tape Drive (3592), SC26-7514.


Encryption key paths
System-managed encryption on z/OS can use either the in-band or out-of-band encryption
key flow. For in-band the key request flows from the tape drive over the ESCON/FICON®
channel to the server proxy (a component of z/OS), which will translate the request into IP
protocols. The server proxy will then send the key request to Tivoli Key Lifecycle Manager
using its TCP/IP connection. In an out-of-band configuration, the tape controller establishes
the communication to the Tivoli Key Lifecycle Manager server over a TCP/IP connection. The
use of out-of-band support requires the use of a router for the control unit.

Out-of-band support runs on VM, VSE, TPF, and zLinux, and is your only option on those
operating system platforms. The TS7700 Virtualization Engine only uses out-of-band support.

In-band key flow
In-band key flow, illustrated in Figure 1-5, occurs between Tivoli Key Lifecycle Manager and
the tape drive through a FICON proxy on the FICON/ESCON interface. The FICON proxy
supports failover to the secondary key path on failure of the first-specified Tivoli Key Lifecycle
Manager path addresses. Impact on controller service requirements is minimal.

The controller does the following:
Reports drive status in SMIT displays
Passes encryption-related errors from the drive to the host
Reports “encryption failure unit checks” to the host
Must be reconfigured whenever new encryption drives are introduced for attachment or
when an encryption-capable drive is enabled for encryption

System z

Tivoli Key
Lifecycle Library Manager
Manager 3953 / 3494

Library
Manager
Interface

IOS Key
Exchange
Interface
FICON Subsystem TS1120
Proxy Proxy Drive Tape Drive
Interface
Encryption ESCON/ TS1120 Tape
FICON
Control Controller
Interface
or 3592-J70

Figure 1-5 In-band encryption key flow

Out-of-band key flow
Out-of-band key flow, shown in Figure 1-6 on page 13, occurs between Tivoli Key Lifecycle
Manager and the tape drive through a subsystem proxy that is located in the 3592 controller
or TS7700 Virtualization Engine on the Tivoli Key Lifecycle Manager interface. Impact on


service requirements can be greater than for in-band key flow due to the introduction of two
routers on the Tivoli Key Lifecycle Manager interface, to and from the controller.

The controller and the TS7700:
Support failover to the secondary key path on failure of the first-specified Tivoli Key
Lifecycle Manager path addresses
Report drive status in SMIT displays
Pass encryption-related errors from the drive to the host
Report “encryption failure unit checks” to the host
Must be reconfigured whenever new encryption drives are introduced for attachment or
when an encryption-capable drive is enabled for encryption

You can enter up to two Tivoli Key Lifecycle Manager IP/domain addresses (and up to two
ports) for each controller, as well as two Domain Name Server IP addresses.

Tivoli Key TS7700
Tivoli Key Lifecycle Manager Interface
Lifecycle Virtualization
Manager Library Engine
Tivoli Key Manager
Lifecycle Library Manager Interface
Manager
Interface 3953 / 3494 Subsystem
Proxy
Library Manager
Interface

Drive
System z Interface

TS1120
Tape Drive
FICON Subsystem (Back End)
Proxy Proxy
ESCON/
Encryption FICON TS1120 Tape Drive
Control Interface Interface TS1120
Controller
or 3592-J70 Tape Drive

Figure 1-6 Out-of-band encryption key flow

1.6.2 Library-managed encryption
In a library-managed encryption (LME) implementation, encryption policies reside within the
tape library. This method of tape encryption requires a Tivoli Key Lifecycle Manager for key
management. LME is fully transparent to the application and system layers. Figure 1-7 on
page 14 shows an example of library-managed encryption.

Library-managed encryption offers you the broadest range of application and operating
system support. Centralized enterprise-class key management facilitates tape interchange
and migration. If you implement LME on a TS3500 or 3494 tape library, you get policy
granularity on a per-volume basis. LME comes with additional responsibilities for the storage


administrator as compared to AME. Data access depends on the availability of Tivoli Key
Lifecycle Manager and the key path.

In most distributed systems environments, LME is the preferred method for tape encryption.

Application
Layer

Tivoli Key
Lifecycle
Manager System
Layer

Library
Policy
Layer

Figure 1-7 Library-managed encryption (LME)

LME can be implemented:
On a distributed systems-attached TS3500 tape library with TS1120 and LTO Ultrium 4
tape drives
On an distributed systems-attached 3494 or TS3400 tape library with TS1120 tape drives
On a TS3310, TS3200, or TS3100 tape library with LTO Ultrium 4 tape drives

Key generation and management is handled by Tivoli Key Lifecycle Manager, running on a
host with a TCP/IP connection to the library. Policy control and keys pass through the
library-to-drive interface; therefore, encryption is transparent to the applications.

For TS3500 and IBM 3494 tape libraries, you can use barcode encryption policies (BEPs) to
specify when to use encryption. On an IBM TS3500 Tape Library, you set these policies
through the IBM System Storage Tape Library Specialist Web interface. On a 3494 tape
library, you can use the Enterprise Automated Tape Library Specialist Web interface or the
Library Manager Console. With BEPs, policies are based on cartridge volume serial numbers.
Library-managed encryption also allows for encryption of all volumes in a library, independent
of barcodes.

For certain applications, such as Symantec Netbackup, library-managed encryption includes
support for Internal Label Encryption Policy (ILEP). When ILEP is configured, the TS1120 or
LTO Ultrium 4 Tape Drive automatically derives the encryption policy and key information from
the metadata written on the tape volume by the application. For more information, refer to
your Tape Library Operator’s Guide.

The following IBM tape libraries support library-managed encryption:
IBM System Storage TS3500 Tape Library
IBM TotalStorage® 3494 Tape Library



Note: System-managed encryption and library-managed encryption interoperate with one
another. A tape that is encrypted using SME can be decrypted using LME, and the other
way around, provided that they both have access to the same keys and certificates.

1.6.3 Encrypting and decrypting with SME and LME
Encrypting and decrypting with system-managed encryption and with library-managed
encryption have identical process flows.

SME and LME encryption processes
Figure 1-8 on page 16 describes the flow of encrypted data to tape, and how keys are
communicated to the tape drive and then stored on the tape media. In this particular example,
assume a TLKM is running on an abstract server, and that the tape library and, consequently,
the tape drives are connected to another abstract server. These can be the same server or
different servers, because whether the server is the same or not does not affect the outcome.

Assume that a certificate from a business partner had been imported into this keystore. It only
has a public key associated with it; the business partner has the corresponding private key.

Now, the server sends a write request to the drive. The drive is encryption-capable, and the
host has requested encryption. As part of this initial write, the drive obtains from the host or a
proxy two Key Encrypting Key (KEK) labels, which are aliases for two Rivest-Shamir-
Adleman (RSA) algorithm KEKs. The drive requests that the Tivoli Key Lifecycle Manager
send it a data key (DK), and encrypt the DK using the public KEKs aliased by the two KEK
labels.

Tivoli Key Lifecycle Manager validates that the drive is in its list of valid drives or that
accept.Unknown.drives is specified. After validation, Tivoli Key Lifecycle Manager obtains a
random DK from cryptographic services. Tivoli Key Lifecycle Manager then retrieves the
public halves of the KEKs aliased by the two KEK labels. Tivoli Key Lifecycle Manager then
requests that cryptographic services create two encrypted instances of the DK using the
public halves of the KEKs, thus creating two Externally Encrypted Data Keys (EEDKs).

Tivoli Key Lifecycle Manager sends both EEDKs to the tape drive. The drive stores the
EEDKs in the cartridge memory (CM) and three locations on the tape. The Tivoli Key
Lifecycle Manager also sends the DK to the drive in a secure manner. The drive uses the
separately secured DK to encrypt the data.

There are two modes for creating the EEDK:
The first mode is CLEAR or LABEL. In this mode, the KEK label is stored in the EEDK.
The second mode is Hash. In this mode, a Hash of the public half of the KEK is stored in
the EEDK.

When sharing business partner KEKs, we recommend using the Hash mode. The Hash mode
lets each party use any KEK label when importing a certificate into their keystore. The
alternative is to use the CLEAR or LABEL mode and then have each party agree on a KEK
label.


Obtains KEK labels/methods

Requests DK using
KEK labels/methods

Validates drive in Drive Table

Requests a Data Key (DK)

Generates a random DK

Requests KEKs using
KEK labels/method

Retrieves KEK pairs

Requests DK to be wrapped
with public half of KEKs
generating two EEDKs

Creates EEDKs

Sends EEDKs
Writes EEDKs to
three locations on
tape and into CM

Encrypts write data using DK

Tivoli Key
Keystore Crypto Services Lifecycle Manager TS1120

Figure 1-8 Key and data flow for encryption using SME or LME

SME and LME decrypting processes for TS1120
Figure 1-9 on page 17 shows the key and data flow for decrypting data. In this example, we
assume that the data was encrypted at another site. For the decrypting process, the tape has
two EEDKs stored in its cartridge memory. We call these EEDK1 and EEDK2. EEDK1 was
stored with the CLEAR (or LABEL) mode selected, and EEDK2 was stored with the Hash
mode selected.

An encrypted tape is mounted for a read or a write append. The two EEDKs are read from the
tape. The drive asks the Tivoli Key Lifecycle Manager to decrypt the DK from the EEDKs. The
Tivoli Key Lifecycle Manager validates that the drive is in its list of valid drives. After validation,
the Tivoli Key Lifecycle Manager requests the keystore to provide the private half of each
KEK used to create the EEDKs. The KEK label associated with EEDK1 cannot be found in
the keystore, but the Hash of the public key for EEDK2 is found in the keystore.

The Tivoli Key Lifecycle Manager asks cryptographic services to decrypt the DK from EEDK2
using the private half of the KEK associated with EEDK2. The Tivoli Key Lifecycle Manager
then sends the DK to the drive in a secure manner. The drive then decrypts the data on the
tape. In our example, we described reading from an encrypted tape. Exactly the same
communication between tape drive and the Tivoli Key Lifecycle Manager takes place for a
write-append.


Reads EEDKs from
tape or from CM

Requests unwrap of
DK from EEDKs

Validates drive in Drive Table

Requests KEKs
for EEDKs

Retrieves KEK pairs

Requests unwrap of DK
from EEDKs using KEKs

Unwraps DK from EEDKs

Sends DK

Encrypts/decrypts
data using DK

Tivoli Key
Keystore Crypto Services Lifecycle Manager TS1120

Figure 1-9 Key and data flow for decrypting using SME or LME

1.6.4 Application-managed encryption
For application-managed encryption, illustrated in Figure 1-10 on page 18, the application
has to be capable of generating and managing encryption keys and of managing encryption
policies. At the time of writing, the only application with this capability is Tivoli Storage
Manager. Policies specifying when encryption is to be used are defined through the
application interface. The policies and keys pass through the data path between the
application layer and the encrypting tape drives. Encryption is the result of interaction
between the application and the encryption-enabled tape drive and does not require any
changes to the system and library layers.

AME is the easiest encryption method to implement and adds the fewest responsibilities for
the storage administrator. Because the data path and the key path are the same, there is no
additional risk to data and drive availability. Policy granularity depends on the application.
With Tivoli Storage Manager, you control encryption on a storage pool basis. There is no
centralized key management with AME because the application generates, stores, and
manages the encryption keys. The lack of centralized key management makes tape
interchange and migration more difficult.

AME can be the most convenient solution when Tivoli Storage Manager is the only application
that utilizes tape encryption.

Tivoli Storage Manager does not restrict you to using AME. You can also choose SME or
LME to encrypt Tivoli Storage Manager data.


Note: Tape volumes written and encrypted using the application-managed encryption
method can only be decrypted with an application-managed encryption solution. In
addition, because the data keys reside only in the Tivoli Storage Manager database, the
same database must be used.

Policy
Application
Layer

System
Layer

Library
Layer

Figure 1-10 Application-managed encryption

Application-managed encryption on IBM TS1120 and LTO Ultrium 4 tape drives can use
either of two encryption command sets, the IBM encryption command set developed for Tivoli
Key Lifecycle Manager or the T10 command set defined by the International Committee for
Information Technology Standards (INCITS).

Application-managed encryption is supported in the following IBM tape drives and libraries.

TS1120 Tape Drives:
IBM TotalStorage 3494 Tape Library

LTO Ultrium 4 Tape Drives:
IBM System Storage TS2340 Tape Drive Express Model S43 and by use of Xcc/HVEC
3580S4X

For details about setting up application-managed encryption, refer to your Tivoli Storage
Manager documentation or the following Web site:
http://publib.boulder.ibm.com/infocenter/tivihelp/v1r1/index.jsp


Note: Tivoli Key Lifecycle Manager or IBM Encryption Key Manager is not required or used
by application-managed encryption.

Application-managed encryption example
In this section, we describe how data is encrypted to tape using Tivoli Storage Manager as
the key manager. Figure 1-11 shows a high-level diagram depicting how data and keys travel
to and from the tape when writing from the beginning of the tape (BOT).

In this example, a tape drive mounts a tape for encryption. The tape drive then sends its tape
ID or VOLSER to Tivoli Storage Manager. Tivoli Storage Manager generates a 256-bit AES
data key, encrypts the data key, and stores the encrypted data key and the tape identifier in
the Tivoli Storage Manager database. Tivoli Storage Manager then sends the data key to the
tape drive. Using the AES algorithms and the data key that was sent to it, the tape drive
encrypts data to the tape.

TSM
Server
Tape
Drive
2 Tape ID
3 Generate Data Key
4 Store Data Key in DB 6 Encrypt Data
5 Data Key with Data Key

Database 7 Encrypted 1 Tape ID
Tape ID Data Key Data

Tape1 AES Data Key1
Tape2 AES Data Key2
Tape
Tape3 AES Data Key3
AES 256
... ... Encrypted Data

Figure 1-11 Application-managed encryption data flow for encryption

Figure 1-12 on page 20 depicts the flow of data and keys when using application-managed
encryption to read or write append an encrypted cartridge. In this diagram, Tivoli Storage
Manager is the application that controls encryption.

In our example, an encrypted tape is mounted to be read. The tape drive reads the tape ID or
VOLSER and sends that information to Tivoli Storage Manager. Tivoli Storage Manager looks
up that tape ID in its internal database and finds the entry associated with it. Tivoli Storage
Manager then decrypts the data key from the entry and sends the data key to the tape drive.

Now, the tape drive can read data from the tape, decrypting the data as it reads using the
256-bit AES data key and the AES algorithms to yield clear data.


TSM
Server
2 Tape ID Tape
3 Search Taple for 5 Data Key
Drive
Tape ID
4 Retrieve DataKey 8 Decrypted 6 Decrypt Data
with Data Key
from Database Data

Database 7 Encrypted 1 Tape ID
Tape ID Data Key Data

Tape1 AES Data Key1
Tape2 AES Data Key2
Tape
Tape3 AES Data Key3
AES 256
... ... Encrypted Data

Figure 1-12 Application-managed encryption data flow for decrypting data

1.6.5 Mixed mode example
In the previous sections, we described how encryption works with different tape encryption
methods. This section describes a mixed environment, using different types of tape
encryption methods. In reality, an enterprise is likely to have several types of systems. The
tape solutions can commingle and allow all of those disparate setups to take advantage of
tape encryption.

Figure 1-13 on page 21 illustrates the case of an enterprise taking advantage of both in-band
and out-of-band encryption. In addition, this picture shows both a system-managed
encryption solution and a library-managed encryption solution implemented concurrently in
the same enterprise.

In this example, a Tivoli Key Lifecycle Manager is running on a z/OS system, taking
advantage of hardware cryptography for its keystore. There is also a miscellaneous IBM
server system with a Tivoli Key Lifecycle Manager running and a distributed systems server
attached to a TS3500 or 3494 tape library. The z/OS system and the distributed systems
server are attached to two separate libraries. The IBM server, which is running a backup
Tivoli Key Lifecycle Manager, is not attached to any libraries, but it is attached to the shared
LAN/WAN.

We can see that the distributed systems server is running library-managed encryption on its
library. All data is sent across Fibre Channel to the library to be encrypted to tape. The keys
that this library uses for encryption, however, come across the network from either the z/OS
Tivoli Key Lifecycle Manager or the IBM server Tivoli Key Lifecycle Manager.

The library that is attached to the z/OS system shows both in-band and out-of-band
encryption. The z/OS system attached to the library is using in-band encryption. Tivoli Key
Lifecycle Manager communication to the library is across the Fibre Channel, and data is also
sent across the Fibre Channel. In addition, this library is pointing to the backup Tivoli Key
Lifecycle Manager on the IBM server system. The keys that are served to the library from the
IBM server flow across the network, which requires the library’s control unit to have network
access.


z/OS System z Open Systems
KS Server
Server
DFSMS
Uses ICSF's
ICSF
(Any instance
Linux; zLinux;
crypto
services of IBM JVM)
i5/OS, AIX;
Tivoli & key store Windows;
Key Solaris; data
Lifecycle HP-UX
Manager

IBM JVM
Device
FICON
proxy EKM KS Driver
Eth

IB keys OB keys FC
(z/OS) DD
DFSMS keys
tells drive Key
to encrypt req. LAN/WAN any
ATL
a given LM
OB keys
cartridge keys
(non-z/OS)

CU Library
(J70 or C06) Eth Proxy
FC
FC Key is
served if TS3500 or 3494
drive is <OB to drive>
<IB to authorized
drive> RS422
3592 3592
ATL

Figure 1-13 Encryption in a mixed environment


2

Chapter 2. Planning for Tivoli Key Lifecycle
Manager and its keystores
In this chapter, we discuss planning for Tivoli Key Lifecycle Manager and your encryption
environment. Tivoli Key Lifecycle Manager or the Encryption Key Manager must be
implemented before you can encrypt data using system-managed encryption (SME),
library-managed encryption (LME) or full disk encryption (FDE). Application-managed
encryption (AME) does not require either a Tivoli Key Lifecycle Manager or IBM Encryption
Key Manager implementation. It is recommended that you begin your planning early, even
before your hardware and software arrives.

Tivoli Key Lifecycle Manager provides lifecycle management for keys and certificates of an
enterprise by allowing you to centrally create, distribute, back up, archive, and manage the
lifecycle of an enterprise’s keys and certificates. It can be used as a key server for
encryption-capable IBM storage devices such as the TS1120, TS1130 and IBM LTO4 tape
drive, and the DS8000.

You must decide what platform (or platforms) you will implement Tivoli Key Lifecycle Manager
on. In this chapter, we provide a discussion to help you decide how to implement your
encryption solution, including platforms where Tivoli Key Lifecycle Manager can run and
keystore implementation considerations.


2.1 Planning for encryption
Protecting, controlling access to, and verifying authenticity of your data while maintaining its
availability is key to data center management. How do you assure security without
complicating an already complicated storage center? Careful planning for your encryption
rollout will assist you in achieving these goals. Remember that deploying encryption is not
your typical tape library or DS8000 upgrade. Significant new function and infrastructure must
be implemented when you deploy an encryption solution. Planning is vital to a smooth rollout
of an encryption solution into your existing environment. Before you read this chapter and
begin your planning, review Appendix B, “Basics of cryptography” on page 149 to familiarize
yourself with encryption concepts and terminology.

When starting to plan encryption management, there are several important considerations for
you to keep in mind. What solutions are available and what will fit in your environment are key
starting points. Depending on your environment and your operating system platforms, you
might choose to implement one or more methods of encryption. It is not required that you use
the same method of encryption for all your implementations.

2.2 What data to encrypt
When designing an encryption solution begin with a clear understanding of your corporate
encryption goals, and how your company’s data is used and who uses it. If your company’s
encryption goal is end to end security, you will need to secure your data both “in motion” and
“at rest.” In motion refers to data being transmitted; this can be protected using session
encryption. Data at rest refers to data on tape cartridges or disk. Tivoli Key Lifecycle Manager
and IBM encryption-capable tape and disk hardware can help you protect data at rest.

2.2.1 Encrypting data on disk
Simply encrypting all data in your data center may cause your users some unexpected
problems. For data on disk, you may think encrypting all data is a safe thing to do because it
is not transported outside your data center and access to Tivoli Key Lifecycle Manager is
available. But if you encrypt the data needed to IPL the system where Tivoli Key Lifecycle
Manager resides you will have issues because you will need to decrypt the data before you
can IPL your system. Unless there is a Tivoli Key Lifecycle Manager that your disk drive can
access to decrypt the data you will not be able to IPL your system. See “Environments with
disk encryption” on page 27 for more information about Tivoli Key Lifecycle Manager
considerations for disk encryption.

2.2.2 Encrypting data on tape
Data on tape is more complex and you will need to understand what your enterprise data
exchange needs are. Remember that to read data on an encrypted tape cartridge the reader
needs access to the key to decrypt the data.

Data exchange within your enterprise
Is the data on tape cartridges exchanged within your company at the same site or multiple
sites? All sites that have encrypted tape cartridges sent to them will need access to the key to
decrypt the data. Allowing those sites access to your primary and secondary Tivoli Key
Lifecycle Manager will solve this concern by allowing them access to the key needed to
decrypt the data. If you were planning to have unique Tivoli Key Lifecycle Manager instances


at each site, or communication between the sites is not available, then you will need to import
and export certificates when sending tape cartridges between sites. This topic is explored
further in section 2.7.2, “Tivoli Key Lifecycle Manager location” on page 27.

Data exchange with business partners or different platforms
If you are exchanging data with business partners or vendors you will need to understand
their encryption solution and whether they can decrypt the data you send them. If the data
you are sending them is not sensitive, you might choose to not encrypt that data. If it is
sensitive, or your data center policy states that all data on tape cartridges must be encrypted,
then you will need to work with your business partners and vendors to develop data exchange
procedures, including the exchange of public keys.

If you plan to share the same information with multiple business partners, you have additional
planning considerations. If you share information today by sending multiple tape cartridges at
the same time to different business partners, you can continue to do that, but you need to
encrypt the tape for each business partner using the individual business partner’s unique
public key.

2.3 Where does the data reside?
You will need to identify which type of server is writing and reading the tape and disk data. Are
all of the servers of one type or one operating system, or do you have multiple operating
systems that will be encrypting data?

If you have only one platform that reads and writes the data, your solution is greatly simplified.
You can have a single encryption solution that you can deploy for all your systems.

If you have multiple platforms that require their data to be encrypted, you will need to decide if
a single Tivoli Key Lifecycle Manager solution servicing all supported platforms or a different
Tivoli Key Lifecycle Manager solution for each platform is best suited for your enterprise.
While Tivoli Key Lifecycle Manager does provide the ability to have a single solution for both
tape and disk across multiple operating systems and multiple platforms, there might be
operational and staff organizational considerations that would make supporting and
managing a consolidated solution difficult in your enterprise. Additionally, for cross-platform
solutions, there are limitations on keystore usage that might not fit with your encryption goal.
See 2.7.4, “Keystore considerations” on page 28 for information on the keystores.

2.4 Rekeying considerations
You also need to consider rekeying needs in your planning.

Rekeying should not be done to a disk. When the disk is configured and you IML the disk a
key is assigned for the entire disk. This key should not be changed.

For tape, rekeying can be required as part of sharing the tape with your business partners, or
if a key has been compromised. This is analogous to having a locksmith rekey all of the locks
in your house. The old house key cannot be used to get into your house. Rekeying can be
used to make the tape unreadable by anyone possessing the compromised certificate or
private key. Details for performing rekeying in z/OS are provided in the IBM Redbooks
publication IBM System Storage Tape Encryption Solutions, SG24-7320.

Chapter 2. Planning for Tivoli Key Lifecycle Manager and its keystores 25

2.5 Performance and capacity considerations
This section addresses performance and capacity considerations related to Tivoli Key
Lifecycle Manager and supported encryption-enabled hardware.

2.5.1 Performance considerations
Unlike software encryption or encryption appliances, the encryption-capable TS1130,
TS1120, LTO4, and DS8000 disk drives—in concert with Tivoli Key Lifecycle Manager—
encrypt data with minimal data transfer performance impact and without requiring additional
equipment in the computing environment. Performance testing has shown there is little
degradation to data transfer performance with encryption-enabled drives, and the data rate
claims of the drive remain unchanged.

2.5.2 Capacity considerations
The IBM TS1100 family of tape drives and the DS8000 disk drives compress the data prior to
encrypt it, thus you do not loose any storage capacity. Be aware that if you use an application
to encrypt your data prior to the data being compressed, when the tape drive or disk drive
writes the data very little space will be saved when the drive compresses the data, thus you
will loose approximately two thirds of your storage capacity (assuming a 3:1 compression
ratio).

2.6 Keys and certificates
Keeping your keys available and safe is crucial to the success of any encryption solution.

We recommend that you:
Take care to not lose your keys and certificates.
Once a tape cartridge or disk has been encrypted using a key, you must have that key to
decrypt the data. If you lose the key, the data is forever lost. There is no back door to
recover the information.
Back up your keys and certificates and verify that you can retrieve them from the backup
copy.

Important: Although IBM has services that can help you to recover data from a damaged
tape or disk, if the damaged tape or disk is encrypted, what you get back is still encrypted
data. So, if you lose your keys, you have lost your data.

Do not leave your keys and certificates exposed.
The security provided to your data is only as good as your key protection strategy. For
example, putting keys and certificates on users’ general purpose laptops is not
recommended. A better solution is to back up your keys to a DVD that is placed in a
secured, locked facility.
Maintenance, backup, and restoration of key and certificate information can be
accomplished using the Tivoli Key Lifecycle Manager GUI interface or CLI commands.


2.7 Tivoli Key Lifecycle Manager considerations
If your Tivoli Key Lifecycle Manager is down, then none of your encrypted data can be read,
thus it is obvious that you will need more than one Tivoli Key Lifecycle Manager instance in
your environment. This section addresses how many Tivoli Key Lifecycle Manager instances
you need and where to place them.

2.7.1 Multiple Tivoli Key Lifecycle Managers for redundancy
IBM tape hardware allows you to specify up to two IP addresses, thus you can specify up to
two Tivoli Key Lifecycle Manager instances to each tape library. For tape encryption, it is
recommended that you have at least two instances of Tivoli Key Lifecycle Manager installed
and running. In addition to the two active instances, you can have other stand-by instances of
Tivoli Key Lifecycle Manager that can be made available to your environment quickly.

IBM disk hardware allows you to specify up to four IP addresses, thus you can specify up to
four Tivoli Key Lifecycle Manager instances to each disk unit. Disk encryption requires that
you have at least two active instances of Tivoli Key Lifecycle Manager at all times and at least
one of those must be an isolated Tivoli Key Lifecycle Manager server.

You can keep multiple Tivoli Key Lifecycle Manager instances synchronized by backing up
one server and restoring the backup configuration on the other server. You should plan on
doing this backup and restore process when the following events take place:
Initial configuration
Adding keys or devices
Key or certificate replacement intervals
Certificate Authority requests
Upgrades to Tivoli Key Lifecycle Manager or middleware like DB2

2.7.2 Tivoli Key Lifecycle Manager location
When planning for deployment of your Tivoli Key Lifecycle Manager instances you must
consider whether you are planning for a tape encryption only solution or for one that includes
disk encryption as well as tape. Consider your environment’s encryption needs as you select
the locations for your Tivoli Key Lifecycle Manager instances.

Environment with tape encryption only
Selecting a system for your Tivoli Key Lifecycle Manager that is highly available is critical
because Tivoli Key Lifecycle Manager must be available to access data that has been
encrypted. While Tivoli Key Lifecycle Manager is supported on the platforms mentioned in
section 1.5, “Encryption key management” on page 6, the backup/restore mechanism used to
create redundant Tivoli Key Lifecycle Manager instances requires the same platform and
configuration. Thus selecting the same operating system platform will simplify your
administrative and operational process. If the tape unit is unable to access one of your Tivoli
Key Lifecycle Manager instances to obtain a key, it will not be able to encrypt or decrypt data.

With z/OS a RACF keystore is available, which adds additional protection to your encryption
environment.

Environments with disk encryption
Selecting a system for your Tivoli Key Lifecycle Manager that is a highly available is less
critical for disk encryption. Unlike tape, disk will only request a key from Tivoli Key Lifecycle


Manager during the IML process, then use that key to encrypt the data key for the life of the
unit. The disk unit does require that two Tivoli Key Lifecycle Manager instances be up and
active or it will Call Home to raise a red flag about one of the Tivoli Key Lifecycle Manager
instances being down. Despite issuing these alerts, the disk unit will continue to operate
without interruption.

With z/OS a RACF keystore is available, which adds additional protection to your encryption
environment.

Disk encryption solutions in the System z environment have an additional consideration that
you will need to pay close attention to as you design your encryption solution.

You must have Tivoli Key Lifecycle Manager up and operational prior to configuring your
DS8000. When you specify that a DS8000 should be encrypted, the DS8000 is “locked” on
power up and will become “unlocked” when Tivoli Key Lifecycle Manager serves the DS8000
with a key via a TCP/IP connection. Until the DS8000 is unlocked with this key it is unable to
serve data that has been encrypted on its disks.

It is important to architect the deployment of Tivoli Key Lifecycle Manager instances to avoid
deadlock and data dependency situations. For example, an IPL of an LPAR will fail if data
required for the IPL is stored on an encrypted DS8000 and the system being IPLed is the only
Tivoli Key Lifecycle Manager server available.

It is required for you to have an isolated Tivoli Key Lifecycle Manager key server to avoid any
issues during IPL of z/OS LPARs that have dependencies on the encryption-capable
DS8000.

One option is to place a Tivoli Key Lifecycle Manager instance on a dedicated x86 key server.
This will mean that you must have an x86 key server at each site. Human intervention will be
required to keep your primary Tivoli Key Lifecycle Manager synchronized with the one on the
x86 server. On your primary Tivoli Key Lifecycle Manager you will need to save your keys in a
pkcs#12 + password format, then move the keys to the x86 server. Additionally, because the
distributed server does not support z/OS cryptographic hardware or RACF, you will need to
select a keystore that is not hardware protected or RACF based.

2.7.3 Database selection
Tivoli Key Lifecycle Manager utilizes DB2 to store the drive tables and key metadata. If you
have DB2 9, Tivoli Key Lifecycle Manager will use your existing DB2 installation. On
distributed systems, if DB2 9 is less than DB2 9.1 fix pack 4, the Tivoli Key Lifecycle Manager
installer will upgrade your DB2 installation. If you have a prior version of DB2 or do not have
DB2 on the target machine, Tivoli Key Lifecycle Manager will install DB2 9.1 fix pack 4. The
Tivoli Key Lifecycle Manager Backup/Restore function backs up the Tivoli Key Lifecycle
Manager relevant tables. On z/OS the requisite DB2 levels must be installed prior to installing
Tivoli Key Lifecycle Manager.

2.7.4 Keystore considerations
Tivoli Key Lifecycle Manager supports four keystore types on z/OS. Selecting the keystore
you will deploy is driven by regulations and requirements your business needs to meet.
Consider the requirements for a specific keystore type before you install and configure Tivoli
Key Lifecycle Manager. Changing the keystore type after you install and configure Tivoli Key
Lifecycle Manager requires you to un-install, re-install, and re-configure Tivoli Key Lifecycle
Manager.


Tivoli Key Lifecycle Manager supports the following keystore types:
JCEKS (JCE software provider)
Use this keystore type if you are using only Java software. This keystore is supported for
all operating systems and TS1100 family tape drives, LTO tape drives, and DS8000 Turbo
drives. Ensure that the flat file JCEKS keystore resides in a restricted area of the file
system on the Tivoli Key Lifecycle Manager system. Use a JCEKS keystore for all
operating systems other than z/OS. You can also use this keystore type on a z/OS system
if you want to use JCE software and a flat file to store keys, or you want to use the same
keystore for Tivoli Key Lifecycle Manager instances that are running on multiple operating
systems.
JCERACFKS (JCE software provider)
The JCERACFKS keystore makes use of all the security advantages of RACF by storing
keys and certificates in a RACF database structure referred to as a RACF keyring. Use
this keystore type to store key material in your RACF keyring that is not using Integrated
Cryptographic Services Facility (ICSF). This keystore type is only available if Tivoli Key
Lifecycle Manager is running on the z/OS operating system and the attached encrypting
hardware is TS1100 family tape drives and DS8000 Turbo drives. Note this keystore does
not support symmetric keys, thus it is not used with LTO tape drives.
JCECCAKS (IBMJCECCA provider)
JCECCAKS is a file-based keystore that is only supported on the z/OS operating system.
Both symmetric and asymmetric keys can be stored in JCECCAKS, thus it can be used
with the TS1100 family of tape drives, LTO tape drives, and DS8000 Turbo drives. Use
this keystore type when you desire a file-based keystore that has the added security
benefit of being hardware protected. By leveraging Integrated Cryptographic Services
Facility (ICSF), your flat file JCECCAKS keystore will reside in a restricted area of the file
system on the Tivoli Key Lifecycle Manager system. JCECCAKS is primarily intended to
store asymmetric keys that are to be used with the cryptographic hardware coprocessors.
It can also be used to access symmetric keys kept in the ICSF CKDS. Note that to use
hardware protection you will need to have Crypto Express2 cards in your processor.
JCECCARACFKS (IBMJCECCA provider)
The JCECCARACFKS keystore makes use of all the security advantages of RACF and
hardware protection by storing a PKDS label in a RACF database entry, referred to as a
key ring. The actual keys and certificates are stored encrypted with the coprocessor
Master Key in the ICSF PKDS. Only asymmetric keys can be stored in JCECCARACFKS,
thus it can be used with the TS1100 family of tape drives and DS8000 Turbo drives. LTO
drives cannot use this type of keystore. Use this keystore type when you need a RACF
protected keystore that has the added security benefit of being hardware protected and
you do not have LTO drives that are encrypting data. Note that hardware protection
requires a Crypto Express2 card in your processor.

In general, if you plan to run Tivoli Key Lifecycle Manager on multiple platforms and want to
use backup/restore to keep your Tivoli Key Lifecycle Manager keystores synchronized, then
consider using JCEKS because it is supported on all platforms. If you are running Tivoli Key
Lifecycle Manager on z/OS it is recommended that you use a RACF protected keystore
(JCERACFKS or JCECCARACFKS). If your environment requires a hardware-protected
keystore then select JCECCAKS or JCECCARACFKS.

The capabilities of the keystore types are summarized in Table 2-1 on page 30.


Table 2-1 Comparison of keystore types
Keystore type Tivoli Key TS1120, LTO4 DS8000 RACF- Hardware-
Lifecycle TS1130 (stores Turbo drives protected protected
Manager (stores symmetric (stores
Platforms asymmetric keys) asymmetric
supported keypairs and keypairs and
certificates) certificates)

JCEKS ALL Yes Yes Yes No No

JCERACFKS z/OS Yes No Yes Yes No

JCECCAKS z/OS Yes Yes Yes No Yes

JCECCARACFKS z/OS Yes No Yes Yes Yes

2.8 Additional deployment considerations
This section covers additional topics to consider when deploying your Tivoli Key Lifecycle
Manager instances.

2.8.1 Sysplex versus monoplex
Tivoli Key Lifecycle Manager for z/OS can be installed within a standalone system, across
LPARs within a Parallel Sysplex®, or across multiple Sysplexes. During the Sysplex install
the user follows the monoplex install instructions to install and configure Tivoli Key Lifecycle
Manager on their primary LPAR. The instructions for completing a single system install are
documented in the IBM Tivoli Key Lifecycle Manager: Installation and Configuration Guide.
This guide is part of the Tivoli Key Lifecycle Manager Information Center. Following is a link
for your convenience:

http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/index.jsp?topic=/com.ibm.tk
lm.doc/welcome.htm

Once Tivoli Key Lifecycle Manager is fully installed and configured on the primary LPAR, all
subsequent instances of Tivoli Key Lifecycle Manager can be synchronized using the backup
and restore function of Tivoli Key Lifecycle Manager. Leveraging DB2 datasharing, and by
sharing ICSF, RACF, and the HFS across LPARs within a Parallel Sysplex, the Tivoli Key
Lifecycle Manager database and any key material protected by ICSF and RACF will
automatically be propagated to each instance of Tivoli Key Lifecycle Manager. Note that each
instance of Tivoli Key Lifecycle Manager contains a unique set of product configuration files
located within the Tivoli Key Lifecycle Manager_HOME directory that must be manually
propagated to each LPAR running Tivoli Key Lifecycle Manager and restored. The Tivoli Key
Lifecycle Manager backup and restore function will aid with syncing up subsequent Tivoli Key
Lifecycle Manager instances once your primary is fully installed and configured.

The first step in the Sysplex installation of the Tivoli Key Lifecycle Manager is to install
System Services Runtime Environment on each LPAR where you plan to have an instance of
Tivoli Key Lifecycle Manager running. Each instance of Tivoli Key Lifecycle Manager requires
a separate System Services Runtime Environment configuration HFS for hosting and
execution. Multiple instances of System Services Runtime Environment running within a
common Sysplex can point to the same System Services Runtime Environment product file
system and Load Libraries contained within the System Services Runtime Environment
PDSE dataset. When using a shared HFS, symbolic links can be set up on each LPAR so
that each instance of System Services Runtime Environment can point to the same System


Services Runtime Environment product file system. This eliminates the need for mounting the
System Services Runtime Environment product file system on every LPAR.

When installing and configuring multiple instances of System Services Runtime Environment
on the same system, using the same hostname, the following values must be unique for each
instance:
_SSRE_CELL_NAME_
_SSRE_PROC_PREFIX_
_SSRE_CONFIG_FS_
_SSRE_CONFIG_DIRECTORY_
_SSRE_PORT_BASE_

You are prompted for these values during execution of the configSSRE.sh shell script, which
is used to configure an instance of System Services Runtime Environment.

When installing multiple instances on different systems, or on the same Sysplex and using
different hostnames, the following values must be unique for each instance:
_SSRE_CELL_NAME_
_SSRE_PROC_PREFIX_
_SSRE_CONFIG_FS_
_SSRE_CONFIG_DIRECTORY_
_SSRE_SYSTEM_IPNAME_

Again, you are prompted for these values during execution of the configSSRE.sh shell script,
which is used to configure an instance of System Services Runtime Environment.

During the System Services Runtime Environment installation and configuration you will be
prompted for the Sysplex name. This value is saved in the System Services Runtime
Environment configuration file variable:
_SSRE_SYSPLEX_NAME_

When installing and configuring multiple instances of System Services Runtime Environment
within a Parallel Sysplex, ensure that this value is the same for all instances.

System Services Runtime Environment can coexist with WebSphere® Application Server for
z/OS, which can be installed in the same target zone. Tivoli Key Lifecycle Manager, however,
cannot be deployed within WebSphere Application Server for z/OS because the System
Services Runtime Environment contains a specific Java level and Integrated Solutions
Console level that are required by Tivoli Key Lifecycle Manager.

2.8.2 Active/Active
Installation and configuration of redundant Tivoli Key Lifecycle Managers across the
enterprise can be useful when trying to achieve high availability for encrypting LTO tape
drives, 3592 tape drives, or DS8000 turbo drives. It is recommended to have at least two
redundant instances of Tivoli Key Lifecycle Manager for any installation.

Placing instances of Tivoli Key Lifecycle Manager within logical locations in the enterprise will
take careful planning and preparation. Given Tivoli Key Lifecycle Manager's integration with
ISCF, RACF, and DB2, you will need a plan to determine how to transfer key material
between versions of ICSF and RACF running on completely separate systems and how to


transfer the Tivoli Key Lifecycle Manager database between DB2 instances running on
completely separate systems.

If the user also plans to run Tivoli Key Lifecycle Manager on distributed systems, careful
planning will be needed to ensure all key material, configuration data, and the Tivoli Key
Lifecycle Manager database can be propagated between z/OS and the distributed system.
Distributed Tivoli Key Lifecycle Managers do not have hardware keystore support. If Tivoli
Key Lifecycle Manager for z/OS is configured to use hardware protection the user will not be
able to export key material from z/OS to the distributed Tivoli Key Lifecycle Manager
instances. Thus hardware protection cannot be used if the user intends to manually
synchronize distributed and z/OS Tivoli Key Lifecycle Manager systems. Another limitation is
that the backup and restore function does not work across platforms. Any configuration
updates made to your z/OS Tivoli Key Lifecycle Manager instance will then have to be
manually reflected on the distributed Tivoli Key Lifecycle Manager instance.

2.8.3 Primary/Secondary
It is required to at least have a primary and secondary Tivoli Key Lifecycle Manager system.
This should be part of the user's disaster recovery plan. It is especially important with
encrypting DS8000 turbo drives to have a dedicated key server that can be relied on to
always be able to serve keys to your encrypting tape and DASD devices. The introduction of
encrypting DSAD does increase the chance that system packs will be mistakenly stored on
an encrypted device. System packs needed to IPL the system should never be stored on an
encrypted device because this could cause a potential deadlock situation if Tivoli Key
Lifecycle Manager is unable to initialize and serve keys to the encrypted device.

2.8.4 Cloning z/OS Tivoli Key Lifecycle Manager instances
The backup and restore function of Tivoli Key Lifecycle Manager provides a convenient way
of cloning Tivoli Key Lifecycle Manager instances running on z/OS. On z/OS, the backups
contain Tivoli Key Lifecycle Manager configuration information from within the Tivoli Key
Lifecycle Manager_HOME directory. This information can then be sent to other instances of
Tivoli Key Lifecycle Manager running on z/OS and can be used to restore or create a clone on
that system. For Tivoli Key Lifecycle Manager on z/OS, in addition to this configuration
information, to complete a clone of Tivoli Key Lifecycle Manager you must also propagate the
Tivoli Key Lifecycle Manager database and any key material stored within ISCF and RACF to
the other instances of Tivoli Key Lifecycle Manager running on z/OS. Leveraging a shared
ICSF, RACF, and DB2 in datasharing mode within a Parallel Sysplex can be used to
automate this part of the cloning work.

If you are attempting to clone multiple instance of Tivoli Key Lifecycle Manager that are not
running within the same Parallel Sysplex or do not share ICSF, RACF, and DB2 in
datasharing mode, you will need to create a process for manually propagating your Tivoli Key
Lifecycle Manager key material and Tivoli Key Lifecycle Manager database between
systems. In this case the Tivoli Key Lifecycle Manager CLI Import and Export commands will
come in handy for moving key material between systems.

2.8.5 Data sharing on z/OS
When running DB2 in datasharing mode, the sample SPUFI file for creating the Tivoli Key
Lifecycle Manager database, POST_SMPE_TKLM_HOME/samples/ tklmsql_zos_install.db2
only needs to be executed once. After the sample is executed the Tivoli Key Lifecycle
Manager database should be created for all instances of Tivoli Key Lifecycle Manager.


Configuring DB2, ICSF, and RACF in data sharing mode across LPARs within a Parallel
Sysplex can aid with cloning multiple instances of Tivoli Key Lifecycle Manager. This was
described in the previous section.

2.8.6 VIPA and Sysplex distributor
A VIPA/Sysplex distributor can be set up such that key requests coming in from encrypting
devices can be distributed to multiple instances of Tivoli Key Lifecycle Manager running
within a Sysplex. This provides high availability because a given key server IP address is
actually backed by multiple Tivoli Key Lifecycle Manager key servers that are ready to
consume and respond to the request.

To implement this the user should add a VIPADEFINE MOVEABLE IMMED statement in the
system TCP/IP profile that defines Sysplex distributed dynamic VIPA of the Tivoli Key
Lifecycle Manager keyserver IP and ports. Then a DESTIP ALL statement should be added
to instruct the Sysplex distributor to spray all network connections arriving with that specific
IP:port combination to ALL images in the Sysplex. It is recommended to use dynamic routing
with VIPA and optionally use VIPAROUTE statements to optimize routing.

Example 2-1 shows the TCP/IP profile statements that define a Sysplex distributed dynamic
VIPA 9.12.20.243 for ports 3801 and 1443.

Example 2-1 Dynamic VIPA example
VIPADYNAMIC
;
VIPADEFINE MOVEABLE IMMED 255.255.255.0 9.12.20.243
VIPADISTRIBUTE DEFINE SYSPLEXP DISTM BASEWLM 9.12.20.243 PORT 3801 1443
DESTIP ALL
;

VIPAROUTE DEFINE 172.18.1.32 172.1.1.32 ;JA0
VIPAROUTE DEFINE 172.18.1.33 172.1.1.33 ;JB0
VIPAROUTE DEFINE 172.18.1.34 172.1.1.34 ;JC0
VIPAROUTE DEFINE 172.18.1.36 172.1.1.36 ;JE0
VIPAROUTE DEFINE 172.18.1.38 172.1.1.38 ;Z0
VIPAROUTE DEFINE 172.18.1.40 172.1.1.40 ;TPN
VIPAROUTE DEFINE 172.18.1.42 172.1.1.42 ;JH0

ENDVIPADYNAMIC

For additional information about VIPA/Sysplex distributed, see the following document:

Communications Server for z/OS V1R9 TCP/IP Implementation Volume 3: High Availability,
Scalability, and Performance, SG24-7534.

2.9 Additional considerations for encrypting data on tape
cartridges
When encrypting data on tape cartridges you have additional decisions that do not exist when
encrypting on disk. This section discusses those considerations.


2.9.1 Encryption method comparison
For disk encryption only FDE is supported in both the System z and distributed environments.
For tape encryption you will need to decide which method of encryption you will deploy.

System z encryption methods
In System z environments (z/OS, z/VM, z/VSE, and z/TPF), you will use system-managed
encryption. Application-managed and library-managed encryption are not supported for these
operating systems.

Distributed systems encryption methods
In the distributed systems environments (including Linux on System z), you have a choice of
encryption method to use. On most operating systems, all three methods of encryption are
available: Application-Managed, System-Managed, and Library-Managed encryption.

Table 2-2 compares several of the differences and considerations for distributed systems
solutions.

Table 2-2 Comparison of distributed systems encryption methods
Method Policy granularity Advantages Disadvantages

Application-managed Encryption policy is configured at the Fewer new Key management is
encryption application GUI. responsibilities for not centralized.
Granularity is application-dependent. storage administrators Only available
currently in Tivoli
Storage Manager.

System-managed Encryption is configured (on/off) at the host Centralized enterprise- Requires ISV support
encryption for each device driver instance, for class key management. for IBM tape drive
(using device drivers) example, the host-to-drive relationship. Broadest library and device drivers.
non-library coverage

Library-managed Encryption is configured (on/off) at the Centralized enterprise- Not available for
encryption library GUI for each logical grouping of class key management. drives outside an
drives (for example, all drives in a TS3500 Broadest application IBM tape library.
logical library). and operating system
Encrypt with default Tivoli Key Lifecycle (OS) coverage.
Manager or IBM Encryption Key Manager
keys.
or
Barcode encryption policy (BEP) for
VOLSER ranges of cartridges associated
with logical grouping of drives.
or
Internal Label encryption policy (ILEP)
currently supported by Netbackup.

Note: LME Barcode Encryption Policy is supported only on the TS3500 and 3494 tape
libraries. LME-Encrypt All, LME-Internal Label - Selective Encryption, and LME-Internal
Label - Encrypt All are supported on all of the tape libraries.

Application-managed encryption might be the most advantageous when a single application
is the primary user of tape, for example, when all of the tape processing in an distributed
systems environment is related to a single software application, such as a backup/restore


application (Tivoli Storage Manager). In this case, having the backup/restore application
manage the keys might be the most convenient solution.

System-managed encryption and library-managed encryption are perhaps the most logical
approaches in environments where tape assets are shared across multiple applications. This
is due to the transparency of encryption offered through the use of the IBM Encryption Key
Manager or Tivoli Key Lifecycle Manager. As with application-managed encryption, updates
might be needed for certain aspects of the overall system, such as device drivers, operating
systems, DFSMS, or controllers, to fully enable encryption.

Selecting an encryption method
You must decide which encryption method or methods are best for your environments.
System-managed encryption will be used for z/OS, z/VM, z/VSE, and z/TPF operating
systems because it is the only supported method. In general, library-managed encryption will
be used for distributed systems operating systems.

2.9.2 In-band and out-of-band

z/OS Java Virtual Machine
Key Manager (TCP/IP)
Key Store
Common Platform Java
Code Another z/OS
(or Open Systems)
TCP/IP Host

Translates in- FICON Proxy to Key -OR- TCP/IP
band key Manager
exchanges In-band Key Key
Management Manager
TCP/IP Across ESCON/FICON
(out-of-band Encrypt?
Key Key Labels?
Management)

SMS Policy
DFSMS TCP/IP
Data Class

TCP/IP to Key
Manager under
z/OS or elsewhere
ESCON / FICON (in-band key management)

J70 or C06 Control 3592 Model Open Systems
Unit E05 Hosts

(VM, VSE, TPF, or even z/OS) - out-of-band Key Management (TCP/IP)
Control Unit across TCP/IP to z/OS or elsewhere

Figure 2-1 z/OS in-band and out-of-band centralized key management

System-managed encryption on z/OS has two configuration options that are shown in
Figure 2-1. Which type of setup makes sense for you?

The first method of encryption that we describe is in-band encryption, where a tape drive
request for a key travels over the ESCON/FICON channels to a FICON proxy that is
TCP/IP-connected to Tivoli Key Lifecycle Manager. The second method is out-of-band
encryption, where the tape hardware establishes communication to Tivoli Key Lifecycle


Manager server over TCP/IP connections between the tape hardware and Tivoli Key
Lifecycle Manager server.

In-band encryption
In-band encryption uses a FICON proxy (FICON is the I/O supervisor component of z/OS) to
exchange key management information between the tape drive and Tivoli Key Lifecycle
Manager. The tape drive communicates to the FICON proxy over the existing ESCON/FICON
connection. If your tape solution does not include a TS7700, or you are only encrypting data
on tapes that are not in the TS7700, then the use of in-band encryption will save you the cost
of deploying a secondary IP network in your tape area if you do not already have a redundant
network available. The reliability and physical security of the existing I/O attachments are
significant reasons that many clients choose the in-band key management path to Tivoli Key
Lifecycle Manager. The z/OS proxy interface communicates with the tape drive (through the
control unit) in the current data path and then uses a TCP/IP connection to communicate with
Tivoli Key Lifecycle Manager. In-band encryption has the added advantage of being
managed by the I/O supervisor (IOS). This allows you the ability to display and alter Tivoli
Key Lifecycle Manager primary and secondary server addresses from the operator console.
In-band is only supported by the z/OS operating system. If the drives will be attached to other
operating systems, then in-band cannot be used.

Out-of-band encryption
Out-of-band encryption allows the hardware to communicate directly to Tivoli Key Lifecycle
Manager. You will need to order a switch to allow the control unit to attach to your TCP/IP
network. Additionally, you will want to have a redundant TCP/IP network installed in your tape
area. Remember, if the tape hardware cannot communicate with the Tivoli Key Lifecycle
Manager, it cannot read or write an encrypted tape. For out-of-band encryption the Tivoli Key
Lifecycle Manager server addresses are only visible on the Library Manager Console. The
TS7700 uses out-of-band encryption only. z/VM, z/VSE and z/TPF support out-of-band
encryption only.

Tape controller out-of-band support
For ESCON/FICON System z environments utilizing out-of-band support for encryption,
routers are required to allow the tape controller to communicate with Tivoli Key Lifecycle
Manager. Feature code FC5593 (Router for EKM Attach) provides dual routers to allow
redundant connections between the tape controller and the IBM Encryption Key Manager or

The installation of features required for out-of-band support is dependent on the automation
platform supporting the TS1120 or TS1130 tape drives:
For TS1120 and TS1130 tape drives in the TS3500 Tape Library:
Order one FC5593 on the 3953 Model F05 containing FC5505, Base Frame. This feature
supports up to sixteen 3592 tape controllers in a TS3500/3953 Library Manager tape
system.
For TS1120 and TS1130 tape drives in the 3494 Tape Library:
Order one FC5593 on the 3494 Library Manager (Model L10, L12, L14, L22, or L24). This
feature supports up to seven 3592 tape controllers. FC5246, Dual Path Concentrator, is
required on the Library Manager before FC5593 can be installed. A second FC5593 can
be installed on the 3494 Library Manager to support up to a total of fourteen 3592 tape
controllers.


Note: The IBM 3494 Tape Library will support up to fifteen tape controllers. However,
the maximum quantity of two FC5593s in the 3494 Library Manager will only support up
to fourteen 3592 tape controllers.

For TS1120 or TS1130 tape drives in the IBM TS3400 Tape Library:
Order FC5247, Enhanced Router, on the TS1120 Model C06 controller to which the
TS1120 drives are attached. This feature is required when attaching TS1120 drives in the
TS3400 library to a TS1120 Model C06 controller, even when no encryption is needed.
For TS1120 or TS1130 tape drives in the Storagetek 9310 Powerhorn Automated Tape
Library:
– When the tape drives are attached to the 3952 Model J70, order FC5593 on the 3590
Model C10 to support the first Model J70. One of FC4860, FC4862, FC9861, or
FC9862 is required on the 3590 Model C10.
– To support a second 3592 Model J70 in the 3590 Model C10 frame, order FC5594
(Attach Additional CU to Router) on the Model C10. FC5593 is required on the Model
C10 to install FC5594.
– When the tape drives are attached to the TS1120 Model C06, order FC5593 on the
3952 Model F05. FC7315 is required on the 3952 Model F05.
– To support more than one TS1120 Model C06 controller in the same 3952 Model F05
frame, order one FC5594 (Attach Additional CU to Router) for each additional Model
C06 to use out-of-band support. FC5593 is required on the 3952 Model F05 to install
FC5594. The maximum quantity for FC5594 on the 3952 Model F05 is two.
For TS1120 or TS1130 tape drives in a client-supplied rack:
Order one FC5593 on the 3592 Model J70 or TS1120 Model C06 tape controller, which is
contained in 3953 Model F05 containing FC5505, Base Frame. This feature supports up
to sixteen 3592 or TS1120 tape controllers in a 3953 Tape System.

For the latest details about specific hardware, software, and Fibre Channel support for the
IBM TS1120 Tape Drive, refer to the following Web site:
http://www-03.ibm.com/systems/storage/tape/pdf/compatibility/ts1120_interop.pdf

Considerations when selecting in-band or out-of-band encryption
In general, for tape drives that are only used by z/OS it is recommended that you use in-band
encryption. z/VM, z/VSE and z/TPF only support out-of-band encryption. If you are sharing
the same tape drives between z/OS and z/VM, z/VSE, or z/TPF then you must use out-of
band encryption. The TS7700 only supports out-of-band encryption.

2.10 Disaster recovery considerations
If you plan to recover your systems using encrypted data you need to consider how you will
decrypt that data. Your options are:
1. Have a complete mirror of your data and applications, including Tivoli Key Lifecycle
Manager at an alternative site using a product like IBM Global Mirror.
2. Supply an attachment from your DR site to one of your Tivoli Key Lifecycle Manager
instances. When considering this alternative you need to determine if your primary and
secondary Tivoli Key Lifecycle Manager instances are geographically far enough apart for
one of them to be up and active if a disaster occurs at the other site.


3. Have an operational Tivoli Key Lifecycle Manager at your DR site that has recently been
synchronized with your Tivoli Key Lifecycle Manager.
4. Install and maintain a Tivoli Key Lifecycle Manager with all your keys on an alternative
platform that is portable (like Windows on a laptop) and bring it to your DR site.

2.11 IBM Encryption Key Manager to Tivoli Key Lifecycle
Manager migration
If you have an existing Encryption Key Manager (EKM) installed you can migrate all your
keys, keystore, drive tables, and config files to Tivoli Key Lifecycle Manager. This will allow
you to decrypt data that was encrypted by keys stored in IBM Encryption Key Manager.

Migration from an IBM Encryption Key Manager installation to a Tivoli Key Lifecycle Manager
installation requires stopping the IBM Encryption Key Manager server. You can either
schedule a maintenance window to shut down the IBM Encryption Key Manager or if you
have redundant IBM Encryption Key Manager instances you can stage this migration.
Following is a quick overview of things to be aware of when migrating from IBM Encryption
Key Manager to Tivoli Key Lifecycle Manager:
Back up your IBM Encryption Key Manager configuration prior to migration.
Write down the path to the IBM Encryption Key Manager. This path should not contain any
spaces.
Schedule an outage for your IBM Encryption Key Manager server. Note that if you have
redundant IBM Encryption Key Manager instances you do not need to stop all IBM
Encryption Key Manager instances, just the one being migrated to Tivoli Key Lifecycle
Manager. Deactivate one of your IBM Encryption Key Manager instances, leaving your
other IBM Encryption Key Manager instance up to service key requests. Migrate the
deactivated IBM Encryption Key Manager information to Tivoli Key Lifecycle Manager,
then activate Tivoli Key Lifecycle Manager using the IP address of the IBM Encryption Key
Manager instance you are replacing with this Tivoli Key Lifecycle Manager instance. Once
this Tivoli Key Lifecycle Manager is up and active, you can deactivate your second IBM
Encryption Key Manager and use the backup/restore feature to configure your second
Migration types that are not supported:
– Administrator SSL keystores or truststores
– PKCS11Impl keystores and truststores

For more information about IBM Encryption Key Manager migration to Tivoli Key Lifecycle
Manager, see Tivoli Key Lifecycle Manager Installation and Configuration Guide SC23-9977,
and the following Web site:
lm.doc/install/cpt/cpt_ic_plan_migration.html

2.12 Tivoli Key Lifecycle Manager configuration planning
checklist
This section provides a review of the issues you should consider when doing the
configuration planning for Tivoli Key Lifecycle Manager:


Make sure you have selected a supported software and hardware platform. Note that the
machine name cannot start with the following:
– IBM
– SQL
If you are migrating an existing IBM Encryption Key Manager server verify that the server
meets the Tivoli Key Lifecycle Manager server recommendations, and follow the migration
procedures described in the previous section.
If this is a new Tivoli Key Lifecycle Manager install:
– Know the recipients for the tapes to be written and read.
For each recipient, an associated X.509 certificate must exist when using TS1120 or
TS1130.
For each recipient, a key or range of keys must be pre-generated within the keystore
when using LTO4.
– Identify the tape drives that will be used.
For each tape drive, determine the drive serial number for input into the Tivoli Key
Lifecycle Manager drive table. You can also configure Tivoli Key Lifecycle Manager to
accept unknown drives.
– Have existing keys and certificates that you plan to use.

Note: When deploying multiple Tivoli Key Lifecycle Manager servers to handle requests
from the same set of tape drives, the information in the associated keystores must be kept
the same.

This consistency is required so that no matter which Tivoli Key Lifecycle Manager server is
contacted, the necessary information is available to support the request from the tape
drive.


2.13 Tivoli Key Lifecycle Manager planning quick reference
Table 2-3 compares the encryption characteristics of the TS1120, TS1130, and LTO4 tape
drives and the DS8000 Turbo drive.

Table 2-3 Encryption implementation characteristics comparison
Description TS1120 tape drive TS1130 tape drive LTO4 tape drive DS8000

Encryption stan- AES (256-bit) AES (256-bit) AES (256-bit) AES (256-bit)
dard

Encryption pro- Symmetric AES (256) Symmetric AES (256) Symmetric AES (256) Symmetric AES (256)
cess for data

Encryption key Wrapped key Wrapped key Direct key Wrapped key
model

Encryption type for Public/private key Public/private key None, since data key Public/private key
data keys (Asymmetric) (Asymmetric) is not stored on car- (Asymmetric)
tridge.

Data keys used Unique data key for Unique data key for Keylist: A list or range Unique data key for
each cartridge each cartridge of data keys used, each volume
pregenerated in key-
store

Data keys stored? Wrapped (encrypted) Wrapped (encrypted) Stored in keystore Wrapped (encrypted)
data keys (2) stored data keys (2) stored only. Key Identifier data keys (2) stored
on cartridge, called on cartridge, called stored on tape on DS8000, called
EEDKs EEDKs EEDKs

Keystore Contents Contains private keys Contains private keys Contains data keys Contains private keys
and certificates repre- and certificates repre- that have been pre- and certificates repre-
senting public keys. senting public keys. generated senting public keys.
Preloaded in keystore Preloaded in keystore Preloaded in keystore

2.13.1 Other resources that can help with the planning process
For those of you who are implementing new IBM tape drives or tape libraries, here are some
other IBM Redbooks publications that can help you with library planning and implementation
details:
IBM TS3500 Tape Library with System z Attachment: A Practical Guide to TS1120 Tape
Drives and TS3500 Tape Automation, SG24-6789. This book describes the TS1120 tape
drive in a TS3500 tape library using z/OS or other System z platforms; most concepts
should map to the TS1130 as well.
IBM System Storage Tape Library Guide for Open Systems, SG24-5946. This book
describes the TS1120, TS1130, and the LTO4 tape drive in Open Systems
implementations. This book also discusses Open Systems IBM tape libraries: the TS3500,
3494, TS3400, TS3310, TS3200, TS3100, TS2900.
IBM TotalStorage 3494 Tape Library: A Practical Guide to Tape Drives and Tape
Automation, SG24-4632. This book explains how to plan for and how to install the tape
products and library in enterprise platforms. It considers day-to-day operations and
integration with other products and applications and also provides information about data
migration and operational considerations.


IBM System Storage Virtualization Engine TS7700: Tape Virtualization for System z
Servers, SG24-7312. This book describes the TS7700 Virtualization Engine using 3592,
TS1120, TS1130 tape drives in a TS3500 tape library or a 3494 tape library.
IBM System Storage Tape Encryption Solutions, SG24-7320


3

Chapter 3. Tivoli Key Lifecycle Manager
installation
This chapter describes the installation of Tivoli Key Lifecycle Manager for z/OS.


3.1 Installation overview
This chapter covers the installation of Tivoli Key Lifecycle Manager for z/OS and its
prerequisite, IBM System Services Runtime Environment for z/OS.

The installation guides for IBM System Services Runtime Environment for z/OS and Tivoli
Key Lifecycle Manager detail the installation processes thoroughly, so rather than reproduce
that material, we outline the installation process as performed on our systems and provide
details on our configuration.

Once IBM System Services Runtime Environment for z/OS is installed and verified, we
deploy a Tivoli Key Lifecycle Manager instance on a single LPAR in a Sysplex and then
create another IBM System Services Runtime Environment for z/OS and Tivoli Key Lifecycle
Manager instance on a second LPAR in the same Sysplex and DB2 data sharing group.
Finally, we perform a backup of the Tivoli Key Lifecycle Manager configuration of the first
Tivoli Key Lifecycle Manager instance and restore it to the second Tivoli Key Lifecycle
Manager instance, thus synchronizing (cloning) the two instances.

Note: Whenever updates are made on the first Tivoli Key Lifecycle Manager instance, a
backup of the Tivoli Key Lifecycle Manager should be made and restored to the second
Tivoli Key Lifecycle Manager instance.

Skill requirements for installation
On z/OS systems, installing and configuring Tivoli Key Lifecycle Manager requires
coordination and assistance from your z/OS System administrators, DB2 and WebSphere
Application Server administrators, and Security administrators. This document assumes that
the reader is familiar with z/OS systems (including UNIX® System Services), and also with
the accompanying products, WebSphere Application Server for z/OS, DB2 for z/OS, and
RACF.

3.2 Solution components
A Tivoli Key Lifecycle Manager for z/OS V1.0 installation requires several components:
z/OS V1.9 or later, including RACF and SMF.
IBM Tivoli Key Lifecycle Manager for z/OS V1.0 with Fixpack 1, APAR OA28422, PTF
UA47192.
IBM DB2 for z/OS, v1.8 or later (including JDBC™).
IBM System Services Runtime Environment for z/OS V1.1 with Service Update 1, APAR
PK72726 (this includes IBM Integrated Solutions Console V7.1).
Optional - For keystore types JCECCAKS or JCECCARACFKS, Integrated Cryptographic
Services Facility, version HCR7751 or later (for hardware encryption with Crypto Express2
Coprocessor). See IBM Tivoli Key Lifecycle Manager Installation and Configuration Guide
for restrictions on using versions of ICSF that are earlier than HCR7751.
IBM System z9® EC or z9 BC, z10 EC or z10 BC.
Refer to the IBM Tivoli Key Lifecycle Manager Installation and Configuration Guide for the
latest hardware and software version requirements.


Note: To avoid errors during installation of the Tivoli Key Lifecycle Manager, pay close
attention to the sections that describe the permissions required by various components.

3.2.1 Tivoli Key Lifecycle Manager for z/OS
Tivoli Key Lifecycle Manager for z/OS works with IBM encryption-enabled storage
components to generate, protect, store, and maintain encryption keys that are used to
encrypt information being written to and decrypt information being read from storage media.

Tivoli Key Lifecycle Manager executes in the IBM Java runtime environment™ and uses IBM
Java security components for the cryptographic capabilities used.

Tivoli Key Lifecycle Manager provides simplified key lifecycle management for LTO, 3592,
and DS8000 turbo drives. Tivoli Key Lifecycle Manager is a WebSphere Application Server
application that runs within the System Services Runtime Environment. Like its predecessor,
IBM Encryption Key Manager, Tivoli Key Lifecycle Manager provides a key server for serving
keys to LTO, 3592, and DS8000 turbo drives.

Tivoli Key Lifecycle Manager for z/OS comes as an SMP/E installable package in SMP/E
RELFILE format on magnetic tape. Samples are included with the SMP/E package to aid with
completing the SMP/E install. The result of the SMP/E install will be a tar file, tklm.tar, located
in the recommended directory of /usr/lpp/tklm/.

Once the SMP/E work is complete and the tklm.tar file is available in the filesystem, several
post SMP/E steps are required to configure Tivoli Key Lifecycle Manager and deploy the
product within System Services Runtime Environment. These steps include copying the
tklm.tar to another working directory containing the necessary ownership values and
permission bits required by Tivoli Key Lifecycle Manager. After the tklm.tar is copied to an
appropriate working space, the contents must be extracted. The extract files consist of all the
binaries, post SMP/E installation scripts, and samples needed to configure Tivoli Key
Lifecycle Manager, deploy it within the System Services Runtime Environment configuration
HFS, and create the Tivoli Key Lifecycle Manager database within DB2. The scripts and
samples provide a relatively easy process for setting up Tivoli Key Lifecycle Manager on a
standalone system or within a Parallel Sysplex.

Once deployed within the System Services Runtime Environment, Tivoli Key Lifecycle
Manager includes a graphical user interface and a command line interface, which provide for
a simplified configuration and key management experience.

Tivoli Key Lifecycle Manager for z/OS stores key material and certificates in a Java-based
keystore. Depending on your choice of keystore, JCEKS, JCECCAKS, JCERACFKS, or
JCECCARACFKS, the actual key material might be additionally protected by ICSF, RACF, or
both. Tivoli Key Lifecycle Manager for z/OS uses DB2 for z/OS to store metadata about key
material and certificates, devices configured with Tivoli Key Lifecycle Manager, and other
configuration data used by Tivoli Key Lifecycle Manager. Tivoli Key Lifecycle Manager's
integration with RACF, ICSF, and DB2 requires the Tivoli Key Lifecycle Manager
administrator to work closely with their respective RACF, ICSF, and DB2 administrators
during the System Services Runtime Environment and Tivoli Key Lifecycle Manager
installation and configuration, and any future maintenance needed.

Tivoli Key Lifecycle Manager for z/OS contains a backup and restore function that is used to
capture configuration data and key material at a given point in time. While taking backups of
Tivoli Key Lifecycle Manager it is important for the Tivoli Key Lifecycle Manager administrator
to coordinate with their RACF, ICSF, and DB2 administrators to ensure a full snapshot is

Chapter 3. Tivoli Key Lifecycle Manager installation 45

taken of both Tivoli Key Lifecycle Manager's configuration data, any key material protected
optionally by RACF and ICSF, and the Tivoli Key Lifecycle Manager database within DB2. If a
problem occurs within Tivoli Key Lifecycle Manager that requires a restore, the Tivoli Key
Lifecycle Manager administrator will need to coordinate with their DB2 administrator to
ensure they have a DB2 backup that matches with the Tivoli Key Lifecycle Manager backup
they intend to use. In a case were the restore requires a backup of key material that is
protected by RACF or ICSF, or both, coordination with the RACF or ICSF administrator will
also be required.

The Tivoli Key Lifecycle Manager backup and restore function is also used for synchronizing
Tivoli Key Lifecycle Managers deployed across LPARs within a Parallel Sysplex on z/OS. By
leveraging DB2, ICSF, and RACF in datasharing mode within your Parallel Sysplex, Tivoli
Key Lifecycle Manager updates made to each of these components in your first LPAR
housing Tivoli Key Lifecycle Manager will be reflected to the other members of your Parallel
Sysplex. Part of the Tivoli Key Lifecycle Manager configuration does remain in the System
Services Runtime Environment configuration HFS, and must be propagated to all subsequent
LPARs housing Tivoli Key Lifecycle Manager in order to synchronize them with your initial
LPAR where you have fully installed and configured Tivoli Key Lifecycle Manager. The
backup and restore functions provide a simplified way of propagating this Tivoli Key Lifecycle
Manager configuration data to all members of the Parallel Sysplex. All subsequent updates to
Tivoli Key Lifecycle Manager will require the backup and restore function to be executed
again in order to propagate updates to the Tivoli Key Lifecycle Manager configuration to all
members of the Parallel Sysplex.

Migration from the IBM Encryption Key Manager product to Tivoli Key Lifecycle Manager is
allowed during the Tivoli Key Lifecycle Manager installation step or optionally before a master
keystore is defined for Tivoli Key Lifecycle Manager. At the end of the post SMP/E Tivoli Key
Lifecycle Manager installation script, <POST_SMPE_TKLM_HOME>/bin/installTKLM.sh, the
user will be prompted to migrate their IBM Encryption Key Manager. If the user is not ready to
migrate at this point they can select no, and choose to run the migration at a later time using
the post SMP/E Tivoli Key Lifecycle Manager migration script,
<POST_SMPE_TKLM_HOME>/bin/migrateEKM.sh as long as the user does not configure
Tivoli Key Lifecycle Manager with a master keystore. There are certain restrictions when
migrating specific key material from IBM Encryption Key Manager to Tivoli Key Lifecycle
Manager. Careful planning should take place before attempting to perform the migration.

Like IBM Encryption Key Manager, Tivoli Key Lifecycle Manager writes audit records to
record successful and unsuccessful operations. However, unlike IBM Encryption Key
Manager, Tivoli Key Lifecycle Manager by default writes audit records to SMF type 83
sub-type 6 records. These audit records may be extracted into a report using the RACF SMF
Unload Utility. For utility support of the Tivoli Key Lifecycle Manager SMF type 83 sub-type 6
Audit records in z/OS Release 9 and 10, you must apply service for APAR OA26653.
Optionally, Tivoli Key Lifecycle Manager for z/OS can be configured to write audit records to
the file system much like IBM Encryption Key Manager.

3.2.2 IBM DB2 for z/OS
DB2 is used to store Tivoli Key Lifecycle Manager configuration data along with metadata
associated with the keystore and key material. Tivoli Key Lifecycle Manager makes use of
DB2 through a JDBC connection set up for the user during installation. DB2 is not installed as
part of the Tivoli Key Lifecycle Manager installation process, and must be installed
separately. You must work with your DB2 administrator to properly create the Tivoli Key
Lifecycle Manager tables, JDBC, stored procedures, and so forth; to ensure that the
SSREGRP ID has the appropriate permissions; and to establish appropriate backup


procedures. Refer to Installing Tivoli Key Lifecycle Manager Installation and Configuration
Guide for details.

Note: DB2 must reside on the same z/OS system on which the IBM Tivoli Key Lifecycle
Manager server runs. You must also install the DB2 JDBC driver, which is an optional
FMID that comes bundled with the base DB2 package.

3.2.3 IBM System Services Runtime Environment for z/OS, Resource
Recovery Service, and Integrated Solutions Console
System Services Runtime Environment is an entitled version of WebSphere Application
Server for z/OS v6.1 that is leveraged by Tivoli Key Lifecycle Manager. It provides simplified
installation and configuration for Web services, helping to reduce the cost and complexity to
the customer by optimizing resource utilization. Only specific, entitled z/OS products are
allowed to be deployed within the System Services Runtime Environment, and it cannot be
used by a customer’s WebSphere Application Server applications. System Services Runtime
Environment can coexist with full installations of WebSphere Application Server for z/OS.

Only specific entitled z/OS applications are allowed to be deployed within the System
Services Runtime Environment. System Services Runtime Environment controls which
applications are allowed to be deployed by using a fencing system. The fencing system
requires entitled applications to be signed by a System Services Runtime Environment
certificate. This signature is verified when an application attempts to deploy itself within the
System Services Runtime Environment. Only applications with a valid signature that is
verified during deployment will be able to install and run within the System Services Runtime
Environment.

z/OS UNIX System Services must be up in full function mode when installing System
Services Runtime Environment. System Services Runtime Environment V1.1 with Fixpack 1
includes the IBM Integrated Solutions Console, which provides a common administration
console for several IBM products. Tivoli Key Lifecycle Manager makes use of many System
Services Runtime Environment services for items such as serving the GUI to the user,
transactions with DB2, providing security between the browser and Tivoli Key Lifecycle
Manager GUI, and many other tasks.

Resource Recovery Service (RRS) is a z/OS component that can do transaction
management for systems such as DB2. Tivoli Key Lifecycle Manager makes use of DB2 and
System Services Runtime Environment, which make use of RRS to handle DB2 and
WebSphere transactions on the system.

System Services Runtime Environment and WebSphere for z/OS
WebSphere Application Server for z/OS is a priced product. System Services Runtime
Environment is an entitled product that can be ordered and used with entitled products for
z/OS, such as Tivoli Key Lifecycle Manager. In addition to being a free product, System
Services Runtime Environment also has cost savings with regard to mips consumption.

The System Services Runtime Environment contains a registrar file within its product
filesystem that registers the product as a different product from WebSphere Application
Server. This prevents customers from being charged WebSphere Application Server mips for
the usage of System Services Runtime Environment. CPU usage data is recorded within
SMF type 89-1 records for System Services Runtime Environment; however, again, the
customer will not be charged for this usage.


IBM System Services Runtime Environment is based on WebSphere Application Server for
z/OS and Java. If an IBM zAAP processor is installed on the system, some of the System
Services Runtime Environment workload may be eligible for offload. Depending on the
amount of Java application code being executed by the zAAPs, the processor savings will
vary. For more details on zAAP implementation and usage see zSeries Application Assist
Processor (zAAP) Implementation, SG24-6386. Additional information is also available from
the System z Application Assist Processor (zAAP) Web page:

http://www-03.ibm.com/systems/z/advantages/zaap/index.html

Tivoli Key Lifecycle Manager for z/OS must be run within System Services Runtime
Environment V1.1 with Fixpack 1 (APAR PK72726) and is not supported within the priced
WebSphere Application Server for z/OS product.

This level of System Services Runtime Environment is based on WebSphere Application
Server for z/OS 6.1.0.19. System Services Runtime Environment Fixpack 1 contains an
upgraded ISC 7.1, level which is required by the Tivoli Key Lifecycle Manager GUI.

System Services Runtime Environment Fixpack 1 also contains an upgraded Java 5 SR 8
level that contains fixes specific to Tivoli Key Lifecycle Manager. This level of Java is only
available for use with System Services Runtime Environment. Both Tivoli Key Lifecycle
Manager for z/OS and System Services Runtime environment should not be configured to
point to a Java level other then the one that is embedded within System Services Runtime
Environment because this would be considered an unsupported configuration.

3.2.4 RACF/SAF
Resource Access Control Facility (RACF)/System Authorization Facility (SAF) is used by
Tivoli Key Lifecycle Manager to optionally store key and certificate material. Any given
security backend can be used by Tivoli Key Lifecycle Manager provided that it supports the
IRRSDL00 API. The IRRSDL00 API is used to read and write certificate information to the
SAF security backend.

3.2.5 ICSF
ICSF is used by Tivoli Key Lifecycle Manager to optionally store key material, when hardware
encryption is desired or required. It requires that a Crypto Express2 Coprocessor be installed
on the System z server. Asymmetric keys are stored in the ICSF PKDS dataset. Symmetric
keys are stored in the ICSF CKDS dataset as DATA keys.

3.2.6 SMF
SMF is a z/OS component that is used for storing audit records. By default Tivoli Key
Lifecycle Manager on z/OS makes use of SMF audit records to store all audit information
created by Tivoli Key Lifecycle Manager, in type 83, sub-type 6 records. An SMFPRMxx
member must be created in SYS1.PARMLIB allowing for the recording of type 83 records,
using standard z/OS commands.

The Tivoli Key Lifecycle Manager audit function writes audit records in text format to
files/datasets as various auditable events occur during request processing. Tivoli Key
Lifecycle Manager provides three possible values for the audit setting:
1. Low – Stores minimal audit records.
2. Medium – Stores an intermediate amount of audit records.
3. High – Stores the maximum amount of audit records.


The default audit setting for Tivoli Key Lifecycle Manager for z/OS is “High.” This means audit
records will be recorded for all events, and both successes and failures will be recorded.

Note: Remember that when using an audit setting of “High,” configuration management
events are also recorded. By default, on a z/OS system, all auditable events including
configuration management events are recorded in the active SMF dataset.

You can format Tivoli Key Lifecycle Manager audit data using the RACF SMF data unload
utility. For information about how to run the RACF SMF data unload utility, see z/OS Security
Server RACF Auditor’s Guide.

For utility support of the Tivoli Key Lifecycle Manager SMF type 83 sub-type 6 audit records in
z/OS Release 9 and 10, you must apply service for APAR OA26653.

For more information about the Tivoli Key Lifecycle Manager SMF Record format, refer to the
Tivoli Key Lifecycle Manager information center. The information center is available at this
Web site:

http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/topic/com.ibm.tklm.doc/welcome.htm

3.3 z/OS System Services Runtime Environment installation
and configuration
This section describes the z/OS-related configuration tasks required for the IBM System
Services Runtime Environment V1.1.0. We provide an overview of the configuration of
System Services Runtime Environment on our system, which we performed following the
instructions detailed in the IBM System Services Runtime Environment for z/OS
Configuration Guide, with annotations where relevant.

The configuration tasks that follow assume that System Services Runtime Environment
(FMID HSSR110) is installed on your z/OS system as described in the Program Directory.
Refer to Program Directory for Systems Services Runtime Environment for z/OS V1.1 for the
latest product requirements.

You can download System Services Runtime Environment from ShopzSeries at:
https://www14.software.ibm.com/webapp/ShopzSeries/ShopzSeries.jsp

The System Services Runtime Environment product documentation and program directory
can be downloaded from:
http://www-03.ibm.com/servers/eserver/zseries/software/ssre/

z/OS V1 Release 9 or later is required to run z/OS System Services Runtime Environment.

You should also review the current Preventive Service Planning (PSP) information and
acquire all the necessary maintenance available for the product. Table 3-1 lists the PSP
bucket for IBM System Services Runtime Environment.

Table 3-1 PSP upgrade, subset, fmid, and compid
Upgrade ID Subset ID FMID COMPID

SSREZOS HSSR110 HSSR110 5655S26EW
5655S26SS


3.3.1 System Services Runtime Environment installation and configuration
overview
This section describes, at a high level, the steps used to prepare for, perform, and verify the
configuration of the System Services Runtime Environment. This sequence follows closely
the configuration path defined in IBM System Services Runtime Environment for z/OS
Configuration Guide, GA76-0404. The procedure to install and configuration the System
Services Runtime Environment is as follows:
1. SMP/E installation of System Services Runtime Environment
This is not covered in this book; it is fully documented in Program Directory for Systems
Services Runtime Environment for z/OS V1.1.
2. Prepare the host system
Several changes must be made to the host systems where System Services Runtime
Environment will run. The specific changes are:
– Update BPXPRMxx
– Catalog the System Services Runtime Environment load library (BBA.SBBALOAD)
– Review APF authorization
– Ensure that required Language Environment® data sets are in the linklist
– Ensure that BBORTS61 is in LPA
– Prepare to collect SMF records
– Reserve TCP/IP ports
– Ensure RRS is active
– Set up system logger for RRS error log logstream
– Update CTIBBOxx
– Update BLSCUSER
3. Create System Services Runtime Environment configuration file
This phase consists of creating a System Services Runtime Environment configuration file
either manually or through the configSSRE.sh shell script.
4. Create System Services Runtime Environment instance
This phase creates a configuration file system. This is the runtime instance of System
Services Runtime Environment. The instance is created through execution of the
createSSRE.sh script and uses the configuration file previously created.
5. Modify System Services Runtime Environment configuration
After creation of a System Services Runtime Environment instance, additional changes
must be made to the host system and the System Services Runtime Environment instance
configuration file system. This is done by executing the modifySSRE.sh script.
6. Verify System Services Runtime Environment configuration
After all system changes are made and the instance created, you can now start System
Services Runtime Environment and verify that you can log in to the System Services
Runtime Environment console.


3.3.2 Preparing the host system
Our configuration process begins after System Services Runtime Environment has been
SMP/E installed using the product’s program directory. During the SMP/E installation,
datasets were created and UNIX System Services directory paths were created. Information
from the SMP/E installation regarding dataset names and path definitions should be provided
to whomever performs the configuration for System Services Runtime Environment and Tivoli
Key Lifecycle Manager. Table 3-2 lists the default dataset and configuration values and our
installation values. Where possible we kept the default values. The variables for now are
reference only, but are defined in the System Services Runtime Environment configuration file
or defined and used by the System Services Runtime Environment configuration scripts. We
indicate where we could not use the default values.

Table 3-2 Configuration variables for System Services Runtime Environment product binaries
Variable name Default value Our values Description

_SSRE_CODE_FS BBA.SBBAZFS BBA.SBBAZFS HFS or zFS created during SMP/E install
job BBAISHFS or BBAISZFS containing
System Services Runtime Environment
file system

_SSRE_CODE_FS_TYPE ZFS™ ZFS Type of filesystem of _SSRE_CODE_FS

_SSRE_CODE_DIRECTORY /usr/lpp/ssre/V1R1 /usr/lpp/ssre/V1R1 Mount point for _SSRE_CODE_FS
created during SMP/E install job
BBAISMKD

_SSRE_PDSE BBA.SBBALOAD BBA.SBBALOAD System Services Runtime Environment
load library name as defined in
BBAALLOC SMP/E install job for target
load library

Host system changes
You will need to make changes to the z/OS system where System Services Runtime
Environment will run. This section describes the changes that are required. These changes
are covered in greater detail in the IBM System Services Runtime Environment for z/OS
Configuration Guide.

BPXPRMxx
The System Services Runtime Environment product zFS or HFS must be mounted in the
UNIX System Service file system. For the System Services Runtime Environment product
zFS we created the following entry in our BPXPRMxx member (Example 3-1).

Example 3-1 Mount statement for System Services Runtime Environment product zFS
MOUNT FILESYSTEM('BBA.SBBAZFS')
MOUNTPOINT('/usr/lpp/ssre/V1R1')
TYPE(ZFS)
MODE(READ)

The dataset must be mounted for the configuration process, so if not already mounted, you
can mount it using the TSO MOUNT command. In our case the command would be:
MOUNT FILESYSTEM('BBA.SBBAZFS) MOUNTPOINT('/usr/lpp/ssre/V1R1') TYPE(ZFS)
MODE(READ)


Note: It is critical that the System Services Runtime Environment product zFS be mounted
READ only, otherwise the Tivoli Key Lifecycle Manager installation will be corrupted with
invalid directory attributes.

Note: We accept the default value of AUTOMOVE for the mount statement since we want
this filesystem to be shared across the Sysplex. If deploying on a single LPAR that is not
part of a Sysplex, specify UNMOUNT. If you only want to run System Services Runtime
Environment and Tivoli Key Lifecycle Manager on a single LPAR in the SYPLEX, you can
specify the SYSNAME and UNMOUNT options. Optionally, use AUTOMOVE with either
an INCLUDE or EXCLUDE list to limit filesystem ownership in a Sysplex.

Catalog the BBA.SBBALOAD dataset
Make sure the BBA.SBBALOAD module is catalogued to the z/OS LPAR where System
Services Runtime Environment will run.

APF
Ensure that the following data sets are APF-authorized:
BBA.SBBALOAD
CEE.SCEERUN
CEE.SCEERUN2

These data sets should be added to your applicable PROGxx member to ensure they are
APF authorized after each IPL. The actual dataset names for the Language Environment
Runtime libraries, SCEERUN and SCEERUN2, may be different in your environment.

We verified that the that these were in the APF list by issuing the following MVS command:
D PROG,APF

We only needed to APF authorize the System Services Runtime Environment dataset, so we
added the following statement to our PROGxx member (Example 3-2).

Example 3-2 PROGxx update for APF
APF ADD
DSNAME(SSRE.SBBALOAD)
VOLUME(OP1TSE)

You can dynamically authorize the library through the MVS SETPROG command. In our case
we issued the following command:
SETPROG APF,ADD,DSNAME=BBA.SBBALOAD,VOLUME=(OP1TSE)

LINKLIST
Ensure that the following required Language Environment data sets are in the link list:
CEE.SCEERUN
CEE.SCEERUN2

These data sets should be added to your applicable PROGxx member to ensure they are in
the LINKLIST after each IPL. The actual dataset names for the Language Environment
Runtime libraries, SCEERUN and SCEERUN2, may be different in your environment.


We verified that these datasets were linklisted by issuing the following MVS command:
D PROG,LNKLST

LPA
Ensure that module BBORTS61is in LPA. This module is used by System Services Runtime
Environment for component trace support (CTRACE) and must be in the LPA. To place
BBORTS61 in LPA we loaded it dynamically and then added a statement to have it loaded at
IPL time.
Enter the following command to load it dynamically:
SETPROG LPA,ADD,MODNAME(BBORTS61),DSNAME(BBA.SBBALOAD)
We placed the following statement in a PROGxx member of PARMLIB:
LPA ADD MODNAME(BBORTS61) DSNAME(BBA.SBBALOAD)

SMF recording (optional)
You can optionally collect SMF records created by the runtime servers. To do this you can
update SMFPRMxx as follows if you want to collect SMF type 120 records created by the
runtime servers:
– Update the SYS or SUBSYS(STC,…) statement for started tasks to include the type
120 record.

Optionally, specify subtypes 1 through 6 as show in the following example:
– SUBSYS(STC,INTERVAL(SMF,SYNC),TYPE(0,30,70:79,88:90,101,110,120(1:6))

We added SMF record type 120 to our SYS statement of BPXPRMxx as follows
(Example 3-3).

Example 3-3 SMFPRMxx changes to enable SMF type 120
SYS(TYPE(30,70:79,80:83,103,108,120,130),EXITS(IEFU83,IEFU84,IEFU85,
IEFACTRT,IEFUJV,IEFUSI,IEFUJP,IEFUSO,IEFUTL,IEFUAV),
INTERVAL(SMF,SYNC),NODETAIL)

Note: SMF recording must also be enabled via the WebSphere Application Server
administrative console.

Reserve TCP/IP ports
System Services Runtime Environment requires a range of 16 consecutive ports. You define
what System Services Runtime Environment is to use later on when you create the
configuration file by indicating the starting port address. The relevant configuration variables
and default starting port are shown in Table 3-3.

Table 3-3 Configuration variables
Variable Default Our value

_SSRE_PORT_BASE 32200 32200

_SSRE_PROC_PREFIX SSRE SSRE

You must update your TCP/IP profile dataset to reserve the ports. The
_SSRE_PROC_PREFIX is used during the configuration to indicate the name of the System
Services Runtime Environment started task and is required to be known ahead of time in
order to create the proper reserve statement in your TCP/IP profile dataset.


Note: The process, which is named _SSRE_PROC_PREFIX+S, will use port 3801 and
441. These should also be reserved for use.

We added the following statements to the PORTS section of our TCP/IP profile dataset
(Example 3-4).

Example 3-4 Port statements
32200 TCP SSRE
32201 TCP SSRE
32202 TCP SSRE NODELAYACKS
32203 TCP SSRE
32205 TCP SSRE
32206 TCP SSRED
32207 TCP SSRED NODELAYACKS
32208 TCP SSRE
32210 TCP SSRE
32212 TCP SSREA
32213 TCP SSREA NODELAYACKS
32214 TCP SSRE

To enable this change immediately you can create a dataset with the statement and issue a
vary TCP command with the OBEY parameter. To do so we created a member called OBEY
in a PDS called TCP.SC59.TCPPARMS and issued the following MVS operator command:
VARY TCPIP,,O,TCP.SC59.TCPPARMS(OBEY)

The contents of our obey file are shown in Example 3-5.

Example 3-5 Obey file
PORT
32200 TCP SSRE
32201 TCP SSRE
32203 TCP SSRE
32205 TCP SSRE
32206 TCP SSRED
32207 TCP SSRED NODELAYACKS
32208 TCP SSRE
32210 TCP SSRE
32212 TCP SSREA
32213 TCP SSREA NODELAYACKS
32214 TCP SSRE


You should have a TCP/IP hosts file defined if DNS is not available. Your network
administrator must define a HFS hosts file, /<system>/etc/host, where <system> is the name
of your z/OS system. Example 3-6 is an example of the hosts file.

Example 3-6 hosts file
127.0.0.1 localhost
9.57.2.230 ALPS1229
9.57.2.230 ALPS1229.pok.ibm.com

For more information contact your network administrator or see z/OS Communications Server
IP Configuration Guide, SC31-8776-12.

System Logger logstream (optional) for error log
If you wish to define a logstream for the System Services Runtime Environment error log then
ensure that LOGGER couple data sets have been defined and system logger is active.

For more information on defining couple data sets, defining log streams and configuring
system logger, see z/OS MVS Setting Up a Sysplex, SA22-7625-14.

Example 3-7 is a sample JCL that can be used to define the log stream. You need to adjust
the fields in bold to meet your installation standards. The LOGSTREAM name cannot be
changed at this time.

Example 3-7 System Services Runtime Environment log stream definition
//BBORCLGS EXEC PGM=IXCMIAPU
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DATA TYPE(LOGR)
DEFINE LOGSTREAM NAME(XDOMAIN.ERROR.LOG)
DASDONLY(YES)
HLQ(IXGLOGR)
LS_SIZE(3000)
STG_SIZE(3000)
MAXBUFSIZE(4096)
AUTODELETE(YES)
RETPD(1)
LS_DATACLAS(STANDARD)

CTIBBOxx (optional)
To configure IBM System Services Runtime Environment for z/OS to use component trace
(CTRACE), you must ensure the CTIBBOxx PARMLIB member is set up, as shown in
Example 3-8.

Example 3-8 CTIBBOxx sample
/* */
/* FUNCTION: This is a sample parmlib member used to establish */
/* defaults for SSRE's use of Component Trace. */
/* */
/* TO USE: Copy this member into a data set in the system parmlib */
/* concatenation and rename the copy to CTIBBOxx, where */
/* "xx" is a value of your choice. This value must */
/* match the component trace suffix you specify in the */
/* customization Dialog. */


/* */
/* Change "procname" to the name of the external writer */
/* cataloged procedure that will be used to write trace */
/* data for SSRE. */
/* */
/* If you want the external writer to start whenever */
/* SSRE does, remove the comments around the WTRSTART */
/* parameter. */
/* */
/* DEFAULT CTIBBOxx MEMBER */
/* ================================================================ */
TRACEOPTS
/* ---------------------------------------------------------------- */
/* -Start a ctrace writer. Remove comments to start the PROC */
/* during SSRE initialization. */
/* ---------------------------------------------------------------- */
/* WTRSTART(procname) */
/* */
/* ---------------------------------------------------------------- */
/* -Indicate that tracing is active: */
/* ---------------------------------------------------------------- */
ON
/* ---------------------------------------------------------------- */
/* -Connect to ctrace external writer: */
/* (Note that the WTR value must match the value for the started */
/* ctrace external writer, wtrstart.) */
/* ---------------------------------------------------------------- */
WTR(procname)

Make sure these statements are added to your CTIBBOxx member.

Note: If you have WebSphere Application Server installed, you should be able to find a
sample in the BBOCTI00 member of your SBBOJCL data set.

BLSCUSER (optional)
To use the IPCS support provided by IBM System Services Runtime Environment for z/OS,
you must append the statement in Example 3-9 to the contents of the BLSCUSER member in
your IPCSPARM or PARMLIB.

Example 3-9 BLSCUSER sample
/* ----------------------------------------------------------------*/
/* Sample BLSCUSER member. */
* ================================================================ */
/* */
/*******************************************************************/
/* ----------------------------------------------------------------*/
/* IPCS Verb Exits */
/* ----------------------------------------------------------------*/
EXIT EP(BBORDATA) VERB(CBDATA)
/* ----------------------------------------------------------------*/
/* IPCS Data Struct */
/* ----------------------------------------------------------------*/
DATA STRUCTURE(ACRW) MODEL(BBORACRW) ENVIRONMENT(ALL); /* acrw */
DATA STRUCTURE(ASR) MODEL(BBORASR) ENVIRONMENT(ALL); /* asr */


DATA STRUCTURE(ASRE) MODEL(BBORASRE) ENVIRONMENT(ALL); /* asre */
DATA STRUCTURE(BACB) MODEL(BBORBACB) ENVIRONMENT(ALL); /* bacb */
DATA STRUCTURE(BGVT) MODEL(BBORBGVT) ENVIRONMENT(ALL); /* bgvt */
DATA STRUCTURE(BOAM) MODEL(BBORBOAM) ENVIRONMENT(ALL); /* boam */
DATA STRUCTURE(BOAX) MODEL(BBORBOAX) ENVIRONMENT(ALL); /* boax */
DATA STRUCTURE(BOBK) MODEL(BBORBOBK) ENVIRONMENT(ALL); /* bobk */
DATA STRUCTURE(BTCB) MODEL(BBORBTCB) ENVIRONMENT(ALL); /* btcb */
DATA STRUCTURE(DAUE) MODEL(BBORDAUE) ENVIRONMENT(ALL); /* daue */
DATA STRUCTURE(LSCB) MODEL(BBORLSCB) ENVIRONMENT(ALL); /* lscb */
DATA STRUCTURE(LSCP) MODEL(BBORLSCP) ENVIRONMENT(ALL); /* lscp */
DATA STRUCTURE(LSPC) MODEL(BBORLSPC) ENVIRONMENT(ALL); /* lspc */
DATA STRUCTURE(LSVT) MODEL(BBORLSVT) ENVIRONMENT(ALL); /* lsvt */
DATA STRUCTURE(TBUFSET) MODEL(BBORRTBS) ENVIRONMENT(ALL);
DATA STRUCTURE(TBUF) MODEL(BBORRTBF) ENVIRONMENT(ALL); /* tbuf */
DATA STRUCTURE(TRC) MODEL(BBORRTRC) ENVIRONMENT(ALL); /* trc */
DATA STRUCTURE(SCOX) MODEL(BBORSCOX) ENVIRONMENT(ALL); /* scox */
DATA STRUCTURE(TMP) MODEL(BBORTMP) ENVIRONMENT(ALL); /* tmp */

RRS
Ensure that RRS is active by issuing the following z/OS command:
D A,RRS

The response should inform you if RRS is active. For additional information on RRS, see
z/OS MVS Programming: Resource Recovery, SA22-7616-07.

3.3.3 Create System Services Runtime Environment configuration file
This section describes how to update and run the System Services Runtime Environment
configuration scripts. This procedure assumes you used the default directories when you
installed System Services Runtime Environment. The default directories and permissions
defined and used during our installation are identified in Table 3-4. We used zFS data sets for
our installation, but you can also use HFS data sets.

The product data sets and the /etc and /var file systems should have been created as part of
the job during the installation. If they were not, they should be created with the names
described in Table 3-4.

Your security administrator might be required to run some of the configuration utilities
because RACF administrator access is required. You might also need to be in “superuser”
mode in OMVS for configuration activities. This requirement will be identified with the tasks.

The configuration of System Services Runtime Environment will be done using UNIX System
Services; these commands should be run from the OMVS command interface, not ISHELL.

Table 3-4 Default directory names and permission settings
Directory Permission bit HFS data set

/var/ssreconf 775 BBA.SSRECONF.ZFSa

/usr/lpp/ssre 755 BBA.SBBAZFSb

/usr/lpp/ssre/V1R1 755 BBA.SBBAZFS

/usr/lpp/ssre/V1R1/bin 755 BBA.SBBAZFS

/usr/lpp/ssre/V1R1/defaults 755 BBA.SBBAZFS


Directory Permission bit HFS data set

/usr/lpp/ssre/V1R1/IBM 755 BBA.SBBAZFS

/etc/ssre 755 System ETC file

/etc/ssre/V1R1 755 System ETC file

/etc/ssre/V1R1/conf 755 System ETC file

/etc/ssre/V1R1/ssre_default 755 System ETC file

/var/ssre 755 System VAR file

/var/ssre/V1R1 755 System VAR file

/var/ssre/V1R1/ssre_default 755 System VAR file

/var/ssre/V1R1/ssre_default/logs 755 System VAR file
a. This file is created by the createSSRE.sh script and is the value specified by the
_SSRE_CONFIG_FS_ variable in step 2 on page 59.
b. These are System Services Runtime Environment product files delivered by IBM.

Configuring the environment for System Services Runtime Environment
The following steps are required for configuring the environment:
1. Copy the environment file.
Copy the default ssre_env.sh environmental file from the installation directory to your
/etc/ssre/V1R1/ssre_default directory for customization.

Note: If you did not use the default installation values, you must update this table
with your installation values.

The contents of the environment file are shown in Table 3-5.

Table 3-5 Contents of the ssre_env.sh file
Environment variables set Description Default value
Our config value

SSRE_HOME Install root for the System Services Runtime /usr/lpp/ssre/V1R1
Environment product directory /usr/lpp/ssre/V1R1

SSRE_CFG_ROOT Location of System Services Runtime /etc/ssre/V1R1/conf
Environment configuration files - used by /etc/ssre/V1R1/conf
createSSRE.sh

SSRE_APPSERVER_ROOT Location where System Services Runtime /ssreconf
Environment instance will be created /var/ssreconf

SSRE_DB_ROOT Location of database server none
none

SSRE_LOGFILE_ROOT Directory location where log files are stored /var/ssre/V1R1/ssre_default/logs
/var/ssre/V1R1/ssre_default/logs

SSRE_USERDATA_HOME Directory location where user data for this none
instance is stored none


Copy the System Services Runtime Environment environment file from the installation
directory to the etc directory. Superuser authority is required.
cp /usr/lpp/ssre/V1R1/defaults/ssre_env.sh /etc/ssre/V1R1/ssre_default/
2. Update the System Services Runtime Environment configuration file.
The ssre_env.sh file that you just copied from the installation directory is used as input for
this process, and step 1 of this procedure must be have completed successfully for the
configuration to work.
Update the values in the configuration file using the configSSREscript.
The configSSRE.sh shell command reads the default configuration values shipped in the
/usr/lpp/ssre/V1R1/defaults/SSREdflt.cfg. We recommend that you copy this file to the
/etc/ssre/V1R1/conf directory using the following command:
cp /usr/lpp/ssre/V1R1/defaults/SSREdflt.cfg /etc/ssre/V1R1/conf/SSREdflt.cfg
Go to the install directory and run the following z/OS UNIX shell command:
/usr/lpp/ssre/V1R1/bin/configSSRE.sh -cfg /etc/ssre/V1R1/conf/SSREdflt.cfg
You then are prompted to supply the configuration parameters. You can reply by pressing
Return to accept the defaults or reply with new site-specific values. If you omit the -cfg
parameter, the script prompts you with the output file name and creates it as a result of the
script execution.

Example 3-10 provides an example of the configuration utility.

Example 3-10 Sample of configSSRE.sh execution
BBA0100I: IBM System Services Runtime Environment for z/OS V1R1 configuration
request is being processed

BBA0101I: configSSRE started -
Wed Mar 18 14:37:24 EDT 2009

BBA0547I: Running with ssre_env.sh data
SSRE_HOME=/ssre/V1R1
SSRE_CFG_ROOT=/etc/ssre/V1R1/conf
SSRE_APPSERVER_ROOT=/var/ssreconf
SSRE_DB_ROOT=
SSRE_LOGFILE_ROOT=/var/ssre/V1R1/ssre_default/logs
SSRE_USERDATA_HOME=
BBA0524I: Input option -cfg: /etc/ssre/V1R1/conf/SSREdflt.cfg
BBA0525I: Input option -v: true
BBA0551I: Configuration file save directory set to: /etc/ssre/V1R1/conf
BBA0544I: IBM System Services Runtime Environment for z/OS invoke path set to :
BBA0503I: Found configuration format 1.4 file /etc/ssre/V1R1/conf/SSREdflt.cfg ;
script is now continuing
BBA0299I: Accepted parameter : BBA.SBBAZFS
BBA0299I: Accepted parameter : ZFS
BBA0299I: Accepted parameter : /ssre/V1R1
BBA0299I: Accepted parameter : BBA.SSRECONF.ZFS
BBA0299I: Accepted parameter : ZFS
BBA0299I: Accepted parameter : /var/ssreconf
BBA0250W: User already exists
BBA0299I: Accepted parameter : SSREADM
1234
BBA0299I: Accepted parameter : 1234


BBA0299I: Accepted parameter : SSREGRP
4321
BBA0299I: Accepted parameter : @SYSNAME
BBA0299I: Accepted parameter : @PLEXNAME
BBA0299I: Accepted parameter : @HOSTNAME
32200
32200
BBA0299I: Accepted parameter : SSRE
BBA0299I: Accepted parameter : BBA.SBBALOAD
BBA0299I: Accepted parameter : BBA.PROCLIB
BBA0299I: Accepted parameter : SSRE
BBA0299I: Accepted parameter : OP1TSD
BBA0260I: Writing format 1.4 configuration data to
/etc/ssre/V1R1/conf/SSREdflt.cfg ; script is now exiting
BBA0700I Write of configuration file completed; script is now exiting

If the configSSRE.sh command completes successfully, you should see message
BBA0260I, indicating that the updated file has been saved.

Note: Log files are created for each run of configSSRE.sh and are written to the location
pointed to by the SSRE_LOGFILE_ROOT variable. Each log file is named as follows:
configSSRE_ddmmyy_hhmmss.log

The contents of the default configuration file are listed in Table 3-6.

Table 3-6 Contents of the zmaSSRE.cfg file
Variable name Variable description Default value
Our config value

_SSRE_CODE_FS_ This variable contains the name of the data set in BBA.SBBAZFS
which IBM System Services Runtime Environment for BBA.SBBAZFS
z/OS is initially installed.

_SSRE_CODE_FS_TYPE_ This variable contains the name of the file system type ZFS
for the product file system. ZFS

_SSRE_CODE_DIRECTORY_ This variable contains the path of the system mount /usr/lpp/ssre/V1R1
point product file. /usr/lpp/ssre/V1R1

_SSRE_CONFIG_FS_ This variable contains the name of configuration file BBA.SSRECONF.ZFS
system data set. This data set is allocated during BBA.SSRECONF.ZFS
script execution. If you need to reuse an existing data
set, you must specify the -force option at script
execution.

_SSRE_CONFIG_FS_TYPE_ This variable contains the name of the file system type ZFS
of the configuration file system. ZFS

_SSRE_CONFIG_DIRECTORY_ This variable contains the path of the configuration file /ssreconf
system mount point. The file specified at /var/ssreconf
_SSRE_CONFIG_FS_ is mounted at this mount point
during script execution.

_SSRE_USERID_ This variable contains the user ID for the IBM System SSREADM
Services Runtime Environment for z/OS instance. SSREADM


Variable name Variable description Default value
Our config value

_SSRE_UID_ This variable contains the numeric z/OS UNIX UID to 1234
be assigned to the user ID in the _SSRE_USERID_ 1234
variable.

_SSRE_GROUP_ This variable contains the security group associated SSREGRP
with the IBM System Services Runtime Environment SSREGRP
for z/OS profile.

_SSRE_GID_ This variable contains the numeric z/OS UNIX GID to 4321
assign to the group defined in _SSRE_GROUP_. 4321

_SSRE_SYSTEM_NAME_ This variable contains the name of the system. @SYSNAME
@SYSNAME

_SSRE_SYSPLEX_NAME_ This variable contains the name of the Sysplex. @PLEXNAME
@PLEXNAME

_SSRE_SYSTEM_IPNAME_ This variable contains the DNS host name for TCP/IP. @HOSTNAME
@HOSTNAME

_SSRE_PORT_BASE_ This variable contains the starting port base. 32200
32200

_SSRE_CELL_NAME_ This variable contains the IBM System Services SSRE
Runtime Environment for z/OS cell name. SSRE

_SSRE_PDSE_ This variable contains the IBM System Services BBA.SBBALOAD
Runtime Environment for z/OS program PDSE. BBA.SBBALOAD

_SSRE_PROCLIB_ This variable contains the PROCLIB PDS into which BBA.PROCLIB
the started task PROCS is copied. BBA.PROCLIB

_SSRE_PROC_PREFIX_ This variable contains the member name prefix to use SSRE
for the IBM System Services Runtime Environment SSRE
for z/OS started task PROCS.

_SSRE_VOLSER_ This variable contains the name of the VOLSER. This BBAVOL
volser is used for zFS and HFS file allocation. If the OP1TSD
environment is SMS managed, the volser will not be
used, although a real volser still must be represented.

_SSRE_CONFIG_FORMAT_ Format of configuration file. 1.4
1.4

Attention: Make sure that the HFS or zFS file addressed by
_SSRE_CONFIG_DIRECTORY_ is not handled by automount, which might cause the
script createSSRE to fail.

3.3.4 Creating a System Services Runtime Environment instance
This section describes how to create an instance of IBM System Services Runtime
Environment for z/OS. The configuration file created earlier is used as input for this step. The
createSSRE.sh utility is used to create the configuration.

To start the configuration run the following shell command:
/usr/lpp/ssre/V1R1/bin/createSSRE.sh –cfg /etc/ssre/V1R1/conf/SSREdflt.cfg


To see the options available for the script, refer to the next section, “Options available with
createSSRE.sh”.

Note: While the shell command is running, you might see “Input” at the bottom right of your
OMVS session. If you see this, press the Enter key for a null reply. No input is needed.

After the shell command completes, you should see the following message:
BBA0600I /createSSRE.sh: success

This message indicates that the script ran successfully. Your output on your final panel
should look similar to Example 3-11.

Example 3-11 Sample createSSRE.sh output
BBA0528I: Copying IBM System Services Runtime Environment for z/OS procs
BBA0700I: Inside copy_procs
BBA0700I: Updating zWASPostInstaller.properties file
BBA0529I: Updating setupCmdLine.sh file
BBA0505I: Setup of IBM System Services Runtime Environment for z/OS security may
be required
BBA0506I: Run /usr/lpp/ssre/V1R1/bin/modifySSRE.sh -cfg /etc/ssre/V1R1/conf/zmaS
SRE.cfg or equivalent.
BBA0600I: ./createSSRE.sh: success

Note: Log files are created for each rung of createSSRE.sh and are written to the
/var/ssre/V1R1/ssre_default/logs/ directory by default. Each log file has the name
configSSRE_ddmmyy_hhmmss.log.

Options available with createSSRE.sh
The following options can be used with createSSRE.sh.
-cfg xxxxxx.cfg (required)
Precedes the name of the config file to be used for the create.
-cellname xxxxxx (not required)
Precedes the name of the cell to be used for the create. When specified, -cellname
overrides the cell name specified in the config file.
-cellname and the cell name are required only if @CELLNAME is in the config file.
-force (not required)
Indicates that the configuration file system should be removed or reused if it exists prior to
allocating a new one. The configuration file system is the HFS or zFS that is specified by
the _SSRE_CODE_FS_ variable.
If -force is not specified and the configuration file system exists, createSSRE.sh will fail.
Consider using the -force option in either of these cases:
– Your HFS or zFS specified by the _SSRE_CONFIG_FS_ already exists. The utility
allocates a new data set when it first runs, and if it finds an existing data set, it will fail.
– You are using symbolic links for /usr/lpp/ssre/V1R1 _SSRE_CODE_DIRECTORY_.
Alternately you can specify the true name of the install path without symbolics.
-v (not required)
Indicates verbose messages are to be displayed. This option assists in displaying issues


with the createSSRE.sh utility. If -v is not specified, informational messages are not
displayed.

Modify System Services Runtime Environment configuration
The modifySSRE.sh script must be run to complete the installation. This script sets file and
directory ownership and permissions, performs some customization not handled by the
configuration file, and runs RACF commands stored in
_SSRE_CONFIG_DIRECTORY/misc/RACFMSTR.
1. Review the contents of the modifySSRE script and the RACFMSTR commands.
This job sets up various groups, IDs, classes, and digital certificates for System Services
Runtime Environment. Make sure your SBBALOAD library is cataloged and that you have
not changed the _SSRE_PDSE_ variable after running the createSSRE script. Any of
these conditions will cause the modifySSRE script to fail. To fix the problem, rerun
createSSRE.sh with the new variable set and then run the modifySSRE.sh script.

Note: You need Superuser and RACF security administrator authority to run this job.

2. Run the following OMVS shell commands:
/usr/lpp/ssre/V1R1/bin/modifySSRE.sh -cfg /etc/ssre/V1R1/conf/SSREdflt.cfg
You can specify the option -noracf if your installation does not use RACF.
Upon successful completion, you should see the following message:
BBA0601I: /usr/lpp/ssre/V1R1/bin/modifySSRE.sh: completed

3.3.5 Verify the System Services Runtime Environment configuration

Starting System Services Runtime Environment
To start System Services Runtime Environment, perform the following actions:
1. Activate the PARMLIB changes (APF, LINKLIST, LPA, SMF, and so on), mentioned in
“Preparing the host system” on page 51.
2. Ensure the members from the library pointed to by the _SSRE_PROCLIB variable are in a
system-defined PROCLIB (for example, SYS1.PROCLIB).
3. Start System Services Runtime Environment. The generic start command is:
START appserver_proc_name,JOBNAME=server_short_name,
ENV=cell_short_name.node_short_name.server
where appserver_proc_name and cell_short_name are specified in the configuration file.
node_short_name is always NODE1. For example, if you used the default settings, the
start command should look like:
S SSRE,JOBNAME=SSRE,ENV=SSRE.NODE1.SSRE

Verifying deployment of System Services Runtime Environment
This section describes how to verify the deployment of System Services Runtime
Environment for z/OS.

To verify that System Services Runtime Environment for z/OS has been successfully
deployed:
1. Start IBM System Services Runtime Environment for z/OS. You can skip this step if it has
already been started.


2. Verify the following started tasks are running: SSRED, SSRES, and SSRE.
3. Open a Web browser and direct it to https://<host_name>:32211/ibm/console, where
<host_name> is your fully qualified system host name. You should now see a logon
window. An example of the logon screen is shown in Figure 3-1.

Figure 3-1 Integrated Solutions Console logon screen

Enter the logon ID, which in our case is SSRECFG, and password. This completes the
verification process.

Stopping System Services Runtime Environment
To stop System Services Runtime Environment, issue the following MVS command:
P SSRE
P SSRED

System Services Runtime Environment configuration is now complete. You can continue with
installing Tivoli Key Lifecycle Manager as described in the next section.

3.4 Tivoli Key Lifecycle Manager installation
This section describes the z/OS related configuration tasks required for the IBM Tivoli Key
Lifecycle Manager for z/OS V1.0.0. We provide an overview of the configuration of Tivoli Key
Lifecycle Manager in our environment, following the instructions detailed in the Tivoli Key
Lifecycle Manager Installation and Configuration Guide, SC23-9977, with annotations where
relevant.

The configuration tasks that follow assume that Tivoli Key Lifecycle Manager (FMID
HCKL100) is installed on your z/OS system as described in the Program Directory. In
addition, we recommend that you install APAR OA28422, which is Tivoli Key Lifecycle
Manager V1.0 Fix Pack 1.


The complete system requirements for the SMP/E installation are supplied in the Program
Directory. Refer to Program Directory for IBM Tivoli Key Lifecycle Manager z/OS V1.0.0 for
the latest product requirements.

The primary requirements are:
IBM System Services Runtime Environment for z/OS with APAR PK72726
z/OS DB2 for V8 or later
z/OS V1.9 or later

You should also review the current Preventive Service Planning (PSP) information and
acquire all the necessary maintenance available for the product. Table 3-7 lists the PSP
bucket for Tivoli Key Lifecycle Manager.

Table 3-7 PSP upgrade, subset, fmid, and compid
Upgrade ID Subset ID FMID COMPID

ITKLM4ZOS HCKL100 HCKL100 5698B3500

Note: On 5/22/09 Tivoli Key Lifecycle Manager for z/OS Fixpack 1 was released (APAR
OA28422). New customers who have not yet installed Tivoli Key Lifecycle Manager at the
GA level will find it beneficial to do a fresh install at the fixpack level. Current customers
should move to the fixpack level in order to pick up the latest level of service. Instructions
for both new users and current users can be found in the following README file:
ftp://ftp.software.ibm.com/eserver/zseries/zos/tklm/pdf/oa28422.pdf

3.4.1 Tivoli Key Lifecycle Manager installation overview
This section describes, at a high level, the steps used to prepare for, perform, and verify the
configuration of the Tivoli Key Lifecycle Manager. This sequence follows closely the
configuration path defined in the Tivoli Key Lifecycle Manager Installation and Configuration
Guide.

Note: We do not perform a IBM Encryption Key Manager to Tivoli Key Lifecycle Manager
migration during this installation process.

The procedure to install and configuration the Tivoli Key Lifecycle Manager is as follows:
1. SMP/E installation of Tivoli Key Lifecycle Manager.
This is not covered in this book; it is fully documented in the Program Directory for IBM
Tivoli Key Lifecycle Manager z/OS V1.0.0.
2. SMP/E installation of APAR OA28422, Tivoli Key Lifecycle Manager V1.0 Fix Pack 1.
This is not covered in this book; it is fully documented in the IBM Tivoli Key Lifecycle
Manager Version 1.0 z/OS Fix Pack 1 README file located at:
ftp://ftp.software.ibm.com/eserver/zseries/zos/tklm/pdf/oa28422.pdf
3. Host system requirements:
– Verify that the DB2 JDBC is installed.
– Verify that the required stored procedures are installed.
– Define DSNR, SERVER and STARTED class SAF profiles.


– Configure System Management Facilities (SMF) auditing.
4. System Services Runtime Environment configuration changes.
Several changes must be made to the hosting System Services Runtime Environment
system:
– Change the time zone setting in System Services Runtime Environment.
– Optionally, enable the IBMJCECCA provider if you plan on using keystore types
JCECCARACFKS or JCECCAKS.
5. Install Tivoli Key Lifecycle Manager product tar file created during the SMP/E install.
6. Run DB2 SPUFI scripts.
7. Create the Tivoli Key Lifecycle Manager response file by running the
createResponseFile.sh script.
8. Install Tivoli Key Lifecycle Manager into System Services Runtime Environment by
running the installTKLM.sh script.
9. Perform post installation steps:
– Optionally, configure file-based auditing.
– Configure System Services Runtime Environment to use available authentication data
when an unprotected URI is accessed.
10.Stop and Restart System Services Runtime Environment.
11.Verify installation.

3.4.2 SMP/E install Tivoli Key Lifecycle Manager and SMP/E install Tivoli Key
Lifecycle Manager Fix Pack 1
This process is not covered in this book; it is fully documented in the Program Directory for
IBM Tivoli Key Lifecycle Manager z/OS V1.0.0. For Fix Pack 1 review the README file
mentioned previously.

First install the base level of Tivoli Key Lifecycle Manager, then install the Fix Pack 1 APAR.

After the SMP/E install you will have a TAR file containing the Tivoli Key Lifecycle Manager
product installed in a zFS or HFS. The default installation path is /usr/lpp/tklm.

3.4.3 Host system requirements

Verify the installation of the JDBC drivers
The appropriate JDBC drivers for DB2 must be installed and available prior to beginning the
Tivoli Key Lifecycle Manager installation. Take note of the path of the JDBC drivers because
this information is required when creating the Tivoli Key Lifecycle Manager response file. The
typical locations for these drivers are:
DB2 v8.1 /usr/lpp/db2810/jcc/classes
DB2 v9.1 /usr/lpp/db2910/db2910_jdbc/classes
Our value /usr/lpp/db2/db9g/db2910_jdbc/classes/

Verify that Stored Procedures are available
The Tivoli Key Lifecycle Manager installation guide specifies that the DSNTIJSG job must be
run to create the required Stored Procedures. In addition, the DSNTIJMS job must be run to


bind the packages for JDBC and CLI metadata. The following Stored Procedures are
required:
DB2_INSTALL_JAR
DB2_REMOVE_JAR
DB2_REPLACE_JAR
DB2_UPDATEJARINFO
DSNACCOR
DSNACICS
DSNLEUSR
DSNTBIND
DSNTJSPP
DSNTPSMP
DSNUTILS
DSNUTILU
DSNWSPM
DSNWZP
INSTALL_JAR
REMOVE_JAR
REPLACE_JAR

Furthermore, your WLM application environment must be defined and active for the Java
Stored Procedures in the list. This WLM application environment must be defined separately
from the WLM application environment defined for the base SQL stored procedures.

Verify that the DESCSTAT subsystem parameter is set to YES
Tivoli Key Lifecycle Manager requires that the DESCRIBE FOR STATIC (DESCSTAT)
parameter be set to YES during the installation of the JDBC drivers. Enabling the DESCSTAT
parameter allows retrieval of column names and metadata from the catalog. If the
DESCSTAT parameter is not turned on failures might occur during the Tivoli Key Lifecycle
Manager installation.

More information about the reasons for enabling the DESCSTAT parameter can be found at:

http://www-01.ibm.com/support/docview.wss?uid=swg21157678

Define DSNR, SERVER and STARTED class profiles
Create access for the SSREGRP group ID so that it has a connection to the generic DSNR
class profile for the DB2 subsystem on which Tivoli Key Lifecycle Manager will run. See the
Tivoli Key Lifecycle Manager Installation and Configuration Guide for additional details. For
our test systems we create very permissive profiles that might not be suitable for a production
environment.
RDEF DSNR DB9*.* UACC(READ) OWNER(WELLIE2)

For your WLM Application environment additional profiles might have to be created. We
created the following to cover all requirements:
RDEF SERVER DB2.*.** UACC(READ) OWNER(WELLIE2)

For started tasks initiated by WLM we created the following started profile:
RDEF STARTED ** OWNER(WELLIE2) STDATA(USER(SYS1) GROUP(SYS1))

Note: Work with your DB2 and Security administrators to determine if the appropriate
profiles are already in place; if not, review the requirements in Tivoli Key Lifecycle Manager
Installation and Configuration Guide to determine what is required for your installation.


Configure Systems Management Facilities (SMF) auditing
By default Tivoli Key Lifecycle Manager is set to cut SMF type 83 sub-type 6 records. To
enable Tivoli Key Lifecycle Manager to cut these SMF records give Tivoli Key Lifecycle
Manager permission to IRR.RAUDITX profile of the FACILITY CLASS:
PE IRR.RAUDITX CL(FACILITY) ID(SSREADM) ACCESS(READ)
SETR RACLIST(FACILITY) REFRESH

Make sure your SMFPRMxx member of parmlib is set up to collect SMF type 83 sub-type 6
records. Our systems were set as shown in Example 3-12.

Example 3-12 SMFPRMxx
SYS(TYPE(30,70:79,80:83,103,108,120,130),EXITS(IEFU83,
IEFU84,IEFU85,IEFACTRT,
IEFUJV,IEFUSI,IEFUJP,IEFUSO,IEFUTL,IEFUAV),
INTERVAL(SMF,SYNC),NODETAIL)

3.4.4 System Services Runtime Environment configuration changes
The Tivoli Key Lifecycle Manager Installation and Configuration Guide details the following
changes that should be made to the System Services Runtime Environment.

Change the time zone setting in System Services Runtime Environment
We performed the following steps to change the time zone setting to an appropriate value for
our environment:
1. Log on to the System Services Runtime Environment Integrated Solutions Console.
2. Expand the Environment option menu and click the WebSphere Variables option, as
shown in Figure 3-2 on page 68.

Figure 3-2 Environment options menu


3. The WebSphere variables panel is displayed. From the Scope selection list, select the
Cell= option, as shown in Figure 3-3.

Figure 3-3 Scope selection


4. Click the New button to create a new variable (Figure 3-4).

Figure 3-4 Create a New variable

5. At the configuration screen for the new variable, enter TZ as the Name value and an
appropriate value for your time zone; we entered EST5EDT (Figure 3-5). Click Apply.

Figure 3-5 Enter properties


6. A message will appear asking whether to save or review. Click the highlighted Save
option in the message body to save to the master configuration (Figure 3-6).

Figure 3-6 Confirmation

The new variable TZ will now appear under the variable list for the CELL scope.

Enable the IBMJCECCA provider (optional)
If you plan on using keystore types JCECCARACFKS or JCECCAKS, you must enable the
IBMJCECCA provider. In addition, ICSF must be available. In our case, we had ICSF at level
HCR7751 available and operational.

The modifications in this section are made to the Java Runtime Environment installed as part
of the System Services Runtime Environment instance. This instance directory is referred to
in the various documents as the _SSRE_CONFIG_DIRECTORY. In our example it is
/var/ssreconf.

The Java environment to be modified is located at
_SSRE_CONFIG_DIRECTORY/AppServer/java, which in our case resolves to
/var/ssreconf/AppServer/java.

Hereafter, we refer to the path _SSRE_CONFIG_DIRECTORY/AppServer/java as
SSRE_JAVA_HOME.


The following steps describe what we did to enable the IBMJCECCA provider:
1. Log on to an OMVS session with access to the filesystem containing the System Services
Runtime Environment instance to be modified.
2. Change directory to the SSRE_JAVA_HOME/lib/security directory. For example:
cd /var/ssreconf/AppServer/java/lib/security
3. The files to be modified are local_policy.jar and US_export_policy.jar. If you display the
contents of the directory with the command ls -l, you see that these two files are
currently symbolic links. Since you will be copying two identically named files to this
location you must remove these symbolic links. Issue the following commands:
unlink local_policy.jar
unlink US_export_policy.jar
4. Copy the local_policy.jar and US_export_policy.jar file from the
SSRE_JAVA_HOME/demo/jce/policy-files/unrestricted directory to the
SSRE_JAVA_HOME/lib/security directory. Assuming your current working directory is
SSRE_JAVA_HOME/lib/security, use the following command:
cp ../../demo/jce/policy-files/unrestricted/*
5. Compare the files you just copied to the originals to make sure they are the same. We
issued an ls -ltr command on both sets and compared the values to make sure they
matched.
6. Modify the java.security file. This step adds the IBMJCECCA provider to the security
provider list.
– Change your working directory to SSRE_JAVA_HOME/lib/security, for example:
cd /var/ssreconf/AppServer/java/lib/security
– Back up the current java.security file:
cp ./java.security ./java.security.backup
– The java.security file is encoded in ASCII. This file must be converted to EBCDIC
before you can edit it. The following command converts this ASCII and places the
converted text into a file called java.security.converted.
iconv -f iso8859-1 -t ibm-1047 java.security > java.security.converted
– Edit the java.security.converted file. The default provider list looks as shown in
Example 3-13.

Example 3-13 List of providers
# List of providers and their preference orders (see above):
#
#security.provider.1=com.ibm.crypto.hdwrCCA.provider.IBMJCECCA
#security.provider.1=com.ibm.crypto.fips.provider.IBMJCEFIPS
security.provider.1=com.ibm.crypto.provider.IBMJCE
security.provider.2=com.ibm.jsse.IBMJSSEProvider
security.provider.3=com.ibm.jsse2.IBMJSSEProvider2
security.provider.4=com.ibm.security.jgss.IBMJGSSProvider
security.provider.5=com.ibm.security.cert.IBMCertPath
security.provider.6=com.ibm.security.sasl.IBMSASL
security.provider.7=com.ibm.security.cmskeystore.CMSProvider
security.provider.8=com.ibm.security.jgss.mech.spnego.IBMSPNEGO


To enable the IBMJCECCA provider, move or copy the line
Place it before or after the line
In Example 3-14, we have copied and uncommented the IBMJCECCA provider and
placed it before the IBMJCE provider.

Example 3-14 Updated List of providers
#
security.provider.1=com.ibm.crypto.hdwrCCA.provider.IBMJCECCA

Note: The precedence order of the IBMJCECCA provider should always be higher
than the IBMJCE provider. If this is not the case, any request for Java cryptographic
services that do not explicitly specify the provider will always be routed to the
IBMJCE provider because the IBMJCE provider supports a superset of the
algorithms supported by the IBMJCECCA provider. If the IBMJCECCA provider is
inserted below the IBMJCE provider, System Services Runtime Environment and
Tivoli Key Lifecycle Manager will leverage the z/OS cryptographic hardware only
when necessary.

Notice in Example 3-14, that each provider is prefixed with a security.provider.<#>
statement. Now that you have added a provider the numbers must be updated. The
numbers must be updated to start at 1 and have no duplicate numbers. The
renumbered list is shown in Example 3-15.

Example 3-15 Renumbered provider list
#


7. Convert the updated java.security.converted file from EBCDIC to ASCII and save it as
java.security using the following command:
iconv -f ibm-1047-f -t iso8859-1 java.security.converted > java.security
8. Compare the policy files you copied. Check the file sizes to make sure they match. We
issued the following commands:
ls -ltr /var/ssreconf/AppServer/java/demo/jce/policy-files/unrestricted
ls -ltr /var/ssreconf/AppServer/java/lib/security

Note: Make sure the local_policy.jar and US_export_policy.jar files are owned by the
System Services Runtime Environment started task ID. Issue a chown command if they
are not. For example, in our case the commands would be:
chown SSREADM:SSREGRP /var/ssreconf/AppServer/java/lib/security/local_policy.jar
chown SSREADM:SSREGRP /var/ssreconf/AppServer/java/lib/security/US_export_policy.jar

3.4.5 Install Tivoli Key Lifecycle Manager product tar file created during the
SMP/E install
We did the following to install the Tivoli Key Lifecycle Manager product:
1. Create a zFS for the Tivoli Key Lifecycle Manager product data.
2. Create a mount point for the Tivoli Key Lifecycle Manager zFS.
3. Mount the Tivoli Key Lifecycle Manager zFS.
4. Make SSRECFG the user owner and make SSREGRP the group owner of the previously
created directory.
5. Give SSRECFG and SSREGRP read, write, and execute access to the previously created
directory.
6. Switch user (su command) to SSRECFG.
7. Copy the Tivoli Key Lifecycle Manager tar file to the Tivoli Key Lifecycle Manager product
directory.
8. Run a tar command to extract the Tivoli Key Lifecycle Manager packages.

The Tivoli Key Lifecycle Manager SMP/E installation created a TAR file at the chosen
installation path, which by default is /usr/lpp/tklm. This TAR file contains the Tivoli Key
Lifecycle Manager installation packages, which must be extracted.

Note: It is recommended that each instance of Tivoli Key Lifecycle Manager have its own
product data for its exclusive use, so when deploying in a multi-LPAR configuration each
LPAR should have its own Tivoli Key Lifecycle Manager product directory.

We are deploying in a Sysplex with a shared UNIX System Services filesystem. The TAR file
was placed into /usr/lpp/tklm. This happens to be a zFS filesystem and directory that is
shared by all systems in the Sysplex; we cannot extract the Tivoli Key Lifecycle Manager TAR
file into this directory for use by more than one system because it is recommended that each
system have it own Tivoli Key Lifecycle Manager product data. Instead we chose /var/tklm as
the mount point for the Tivoli Key Lifecycle Manager zFS filesystem that will contain the Tivoli
Key Lifecycle Manager product files. The /var mount point is a system-specific mount point
and the underlying filesystems are not shared Sysplex wide.


Create a zFS for the Tivoli Key Lifecycle Manager product data
We created a zFS for the Tivoli Key Lifecycle Manager product data called
OMVS.TKLM.ZFS. We needed at least 22580 1024k blocks. After extraction and a few
installs and uninstalls of Tivoli Key Lifecycle Manager, a df -KvP command showed the
utilization for the Tivoli Key Lifecycle Manager zFS as in Figure 3-7.

Filesystem 1024-blocks Used Available Capacity Mounted on
OMVS.TKLM.ZFS 67680 22580 45100 34% /var/tklm
ZFS, Read/Write, Device:84, ACLS=Y
Filetag : T=off codeset=0
Figure 3-7 zFS space

Create a mount point for the Tivoli Key Lifecycle Manager zFS
We decided to create a mount point called /var/tklm for each system. The /var filesystem is
not a shared filesystem. In our installation each LPAR is the owner of a zFS mounted at /var.

We created the /var/tklm directory to be used as a mount point for our Tivoli Key Lifecycle
Manager zFS:
mkdir /var/tklm

Mount the Tivoli Key Lifecycle Manager zFS
We used the ISHELL File_systems Mount menu option to mount the zFS at /var/tklm.

Make SSRECFG the user owner and make SSREGRP the group owner
We issued the following command to make SSRECFG the user owner and SSREGRP the
group owner of the Tivoli Key Lifecycle Manager product directory:
chown SSRECFG:SSREGRP /var/tklm

Give SSRECFG and SSREGRP read, write, and execute access
We issued the following command to give SSRECFG and SSREGRP read, write, and
execute access to the Tivoli Key Lifecycle Manager product directory:
chmod 770 /var/tklm

Switch user to SSRECFG
It is critical to run the extract command under the SSRECFG user ID, so switch to the
SSRECFG user ID. We did so by issuing the following command and providing the password
when prompted:
su SSRECFG

Copy the Tivoli Key Lifecycle Manager tar file to the product directory
We copied the TAR file created during the SMP/E installation to our Tivoli Key Lifecycle
Manager product directory using the following commands:
cd /var/tklm
cp /usr/lpp/tklm/tklm.tar


Run a tar command to extract the packages
We executed the following TAR command to extract the Tivoli Key Lifecycle Manager
packages:
cd /var/tklm
tar oxvfp tklm.tar

We verified that all the extracted files and directories were SSRECFG and SSREGRP by
issuing the following command:
ls -alR /var/tklm

3.4.6 Run DB2 SPUFI scripts
After extracting the Tivoli Key Lifecycle Manager packages, the DB2 setup for Tivoli Key
Lifecycle Manager must be completed before continuing the Tivoli Key Lifecycle Manager
installation. Tivoli Key Lifecycle Manager Fix Pack 1 ships three SPUFI scripts: one to install,
one to uninstall the required DB2 data, and one to migrate an existing Tivoli Key Lifecycle
Manager instance to Fix Pack 1 level. Since this is a new install we do not need the migrate
SPUFI script.

These scripts can be found in the samples directory of the Tivoli Key Lifecycle Manager
product installation directory. In our case these are located at:
/var/tklm/samples/tklmsql_zos_install.db2
/var/tklm/samples/tklmsql_zos_uninstall.db2

You must copy these to a z/OS dataset, modify them for your installation and then run the
script.

First we created a z/OS PDS to contain the SPUFI scripts. The dataset must be fixed block
and have a record length of 80.

Then we issued the following commands to copy the samples to the z/OS datasets:
cp -T /var/tklm/samples/tklmsql_zos_install.db2 "//'TKLM.SPUFI(tklmdb2i)'"
cp -T /var/tklm/samples/tklmsql_zos_uninstall.db2 "//'TKLM.SPUFI(tklmdb2u)'"

The SPUFI script for installation must be edited and placeholders replaced with values
appropriate for your environment. Table 3-8 lists the placeholders in the installation script.

Table 3-8 SPUFI script placeholders
Placeholder Description Our value

-DB_OWNER Owner of the Tivoli Key Lifecycle Manager database. SSRECFG

-TS_STORGROUP- Storage group to contain Tivoli Key Lifecycle Manager SYSDEFLT
DB2 tablespaces.

-TS_BP- Buffer pool name for Tivoli Key Lifecycle Manager BP8K0
tablespaces. A minimum buffer pool size of 8K must be
specified.

-INDEX_STOGROUP- Storage group to contain the Tivoli Key Lifecycle SYSDEFLT
Manager indexes.

-INDEX_BP- Buffer pool name for Tivoli Key Lifecycle Manager DB2 BP8K0
indexes. Recommended size is 8k. However, on DB2
V8, this buffer pool must be 4K in size.


Once the script has been customized, it can be run to set up the Tivoli Key Lifecycle Manager
tables in DB2. Verify all SQLCODES are 0. Have your installation’s DB2 administrator run this
script if the Tivoli Key Lifecycle Manager installer does not have the proper DB2
authorization.

If Tivoli Key Lifecycle Manager DB2 data must be removed, run the uninstall script. This
contains commands to drop the Tivoli Key Lifecycle Manager DB2 tablespaces and
database.

3.4.7 Create the Tivoli Key Lifecycle Manager response file by running the
createResponseFile.sh script
Before Tivoli Key Lifecycle Manager can be installed into the hosting System Services
Runtime Environment, a configuration file (called a response file) must be created for the
Tivoli Key Lifecycle Manager install script to use during its installation process. This response
file is created by running the createResponseFile.sh script. It is located in the bin directory of
the Tivoli Key Lifecycle Manager installation directory; for our installation this is:

/var//tklm/bin/createResponseFile.sh

The script is interactive. You are prompted to supply information regarding your environment.
Questions regarding your System Services Runtime Environment, DB2, and TCP/IP settings
are asked. The response file created by the script is called tklmInstall.response.

The script will attempt to detect the values for many of the prompts and display the detected
value. You can press Enter to accept the detected values or type in the correct value.
Table 3-9 lists the configuration information required to complete this step.

Table 3-9 Data required to reply to createResponsFile.sh
Prompt Response file Description Our value
variable

System Services Runtime tklm_was_home Refer to the value of the _SSRE_CONFIG /var/ssreconf
Environment _DIRECTORY parameter in your System
Configuration directory Services Runtime Environment
configuration. This is the path to the System
Services Runtime Environment instance,
configuration file system that was created –
not the System Services Runtime
Environment product directory.

System Services Runtime tklm_was_userid The System Services Runtime Environment SSRECFG
Environment configuration user ID. This is the user ID that
Configuration user ID was created with the RACFMSTR file when
run during the modifySSRE.sh script.

System Services Runtime tklm_was_password System Services Runtime Environment
Environment Configuration user ID password.
Configuration user ID
password

System Services Runtime tklm_was_profile The System Services Runtime Environment default
Environment profile profile into which the Tivoli Key Lifecycle
Manager server module is to be installed.
Generally, there is only one profile and the
Tivoli Key Lifecycle Manager installation
program will automatically detect it.


Prompt Response file Description Our value
variable

System Services Runtime tklm_was_cell The System Services Runtime Environment SSRE
Environment cell cell into which the Tivoli Key Lifecycle
Generally, there is only one cell and the

System Services Runtime tklm_was_node The System Services Runtime Environment node1
Environment node node into which the Tivoli Key Lifecycle
Generally, there is only one node and the

System Services Runtime tklm_was_server The System Services Runtime Environment server1
Environment server server into which the Tivoli Key Lifecycle
Generally, there is only one server and the

Hostname or IP address tklm_db2_hostname The hostname or IP address of the wtsc61.itso.ibm.com
of database server database server.

Database port number tklm_db2_port The TCP port number where the database 37953
server receives incoming requests.

Database location name tklm_db2_location The location name of the database server. DB9I
This value is case sensitive.

Tivoli Key Lifecycle tklm_db2_userid The ID that your DB2 administrator used in SSRECFG
Manager database owner the DB2 Tivoli Key Lifecycle Manager
ID installation SPUFI script.

Tivoli Key Lifecycle tklm_db2_password The password of the Tivoli Key Lifecycle
Manager database owner Manager database owner.
user ID password

Database JDBC driver tklm_db2_jdbc_driver The location of the JDBC drivers. The Tivoli /usr/lpp/db2/db9i/db
path Key Lifecycle Manager installation program 2910_jdbc/classes/
will attempt to automatically detect the
location.

Once you are certain of the values to be used, run the createResponseFile.sh script located
in the Tivoli Key Lifecycle Manager product installation bin directory. Our location for this
script is:

/var/tklm/bin/createResponseFile.sh

The syntax of the command is:
createResponseFile.sh [-v] [-responseFile <response_file>]

All parameters are optional; they are defined as follows:
-v Allows for verbose output.
-responsefile <response_file> Passes an existing response file to the script.


Make sure you run the command as SSRECFG. We issued the following commands to
execute the script:
su SSRECFG
cd /var/tklm/bin
./createResponseFile.sh

Example 3-16 captures the createResponseFile.sh prompts and our responses. Should the
process fail at any point, the error message and reason will be displayed to the screen and
logged. The log can be found in the log directory of the Tivoli Key Lifecycle Manager product
installation directory. For example, our log for this session is located at:

/var/tklm/logs/createResponseFile_042109_110323.log

The log name format is createResponseFile_<mmddyy>_<time>.log. The log file location and
name are displayed during the execution of the createResponseFile.sh script.

Example 3-16 createResponseFile.sh prompts and responses
Log file "/var/tklm/logs/createResponseFile_042109_110323.log" will be used for this run
No response file passed. Defaulting to "/var/tklm/bin/tklmInstall.response"
Using newly created response file "/var/tklm/bin/tklmInstall.response"
Enter the fully qualified path name of the SSRE configuration directory where TKLM will be installed:
/var/ssreconf
Accepted input: ["/var/ssreconf"]
The following SSRE configuration user ids were detected: ["SSRECFG"]
Enter the SSRE configuration user id, or return to accept the detected value [SSRECFG]:
Accepted input: ["SSRECFG"]
Enter the password for user id "SSRECFG", or return to leave blank. (If you do not wish this password to be
a part of the response file, leave this field blank. You will be prompted for the password during the
install in a secure manner):
Please re-enter the password for confirmation:
Accepted input: [*** Password not displayed ***]
The following WebSphere profiles were detected within SSRE: ["default"]
Enter the name of the WebSphere profile where you want to install TKLM, or return to accept the detected
value [default]:
Accepted input: ["default"]
The following WebSphere cells were detected within SSRE: ["SSRE"]
Enter the name of the WebSphere cell where you want to install TKLM, or return to accept the detected value
[SSRE]:
Accepted input: ["SSRE"]
The following WebSphere nodes were detected within SSRE: ["node1"]
Enter the name of the WebSphere node where you want to install TKLM, or return to accept the detected value
[node1]:
Accepted input: ["node1"]
The following WebSphere servers were detected within SSRE: ["server1"]
Enter the name of the WebSphere server where you want to install TKLM, or return to accept the detected
value [server1]:
Accepted input: ["server1"]
Enter the hostname or IP address of your database server: wtsc61.itso.ibm.com
Accepted input: ["wtsc61.itso.ibm.com"]
Enter the TCP port number of the database server: 37953
Accepted input: ["37953"]
Enter the location name of the database server where the TKLM database will be created: db9g
Accepted input: ["db9i"]
Enter the user id of the TKLM database owner: ssrecfg
Accepted input: ["ssrecfg"]
Enter the password for user id "ssrecfg", or return to leave blank. (If you do not wish this password to be
a part of the response file, leave this field blank. You will be prompted for the password during the
install in a secure manner):
Please re-enter the password for confirmation:


Accepted input: [*** Password not displayed ***]
Unable to detect any DB2 JDBC drivers on this system. If you plan to install TKLM on this system, first
ensure that the DB2 JDBC drivers available.
Enter the fully qualified path name of where the JDBC driver is located:
/usr/lpp/db2/db9i/db2910_jdbc/classes
Accepted input: ["/usr/lpp/db2/db9i/db2910_jdbc/classes"]
Writing response data to file /var/tklm/bin/tklmInstall.response
...
Script completed with exit code: 0(SUCCESS)

The resulting file, tklmInstall.response, is created in the Tivoli Key Lifecycle Manager product
installation bin directory. The location is also displayed during the execution of the
createResponseFile.sh script. Figure 3-8 is the tklmInstall.response file created after our
execution of the createResponsFile.sh script.

#TKLM install response file. Do NOT edit. Last modified Tue Apr 21 11:05:16 EDT
2009.
tklm_version=1.0
tklm_was_home=/var/ssreconf
tklm_was_userid=SSRECFG
tklm_was_password=ssrecfg
tklm_was_profile=default
tklm_was_cell=SSRE
tklm_was_node=node1
tklm_was_server=server1
tklm_db2_userid=ssrecfg
tklm_db2_password=ssrecfg
tklm_db2_hostname=wtsc61.itso.ibm.com
tklm_db2_port=37953
tklm_db2_location_name=db9i
tklm_db2_jdbc_driver=/usr/lpp/db2/db9i/db2910_jdbc/classes
Figure 3-8 tklminstall.response file

3.4.8 Install Tivoli Key Lifecycle Manager by running the installTKLM.sh script
After the successful creation of a response file, Tivoli Key Lifecycle Manager can now be
installed into System Services Runtime Environment. This is accomplished by running the
installTKLM.sh script located in the Tivoli Key Lifecycle Manager installation bin directory. In
our case this is located at:

/var/tklm/bin/installTKLM.sh

The syntax of the command is:
installTKLM.sh [-responseFile <response_file>] [-wasPassword <was_password>]
[-dbPassword <db_password>] [-v]

All parameters are optional and are described as follows:
-responseFile <response_file> Name of an existing response file that was created by
createResponseFile.sh
<was_password> Password of the WebSphere user ID
<db_password> Password of the Database user ID
-v Send verbose output to the console (verbose output always goes
to the log)


Note: During the installation process certain files and directories are created or copied for
which chmod commands are not issued.

These files and directories will be affected by the umask setting of the user issuing the
installTKLM.sh command. The umask setting must give the UNIX System Services group
owner read and execute (lookup) access for directories and at least read access for files.

Issue the umask -S command to check your settings. Issue umask g=rx to give the group
access to files and directories.

By default, the script will uses the response file found in the Tivoli Key Lifecycle Manager
installation bin directory. Make sure you run the command as SSRECFG. We issued the
following commands to execute the script:
su SSRECFG
cd /var/tklm/bin
./installTKLM.sh

You might be prompted to enter passwords if you did not do so during the
createResponseFile.sh script execution.

Should the process fail at any point, the error message and reason will be displayed to the
screen and logged. The log can be found in the log directory of the Tivoli Key Lifecycle
Manager product installation directory. An example file name of a log file is:

/var/tklm/logs/installTKLM_041309_154612.log

The log name format is installTKLM_<mmddyy>_<time>.log. The log file location and name
are displayed during the execution of the installTKLM.sh script.

The execution of the script installs several Tivoli Key Lifecycle Manager components into
System Services Runtime Environment. Tivoli Key Lifecycle Manager logs these component
installs in a file called .installedComponents located in the Tivoli Key Lifecycle Manager
installation bin directory. After the successful installation of Tivoli Key Lifecycle Manager into
System Services Runtime Environment, the contents of the .installedComponents file on our
system is as shown in Figure 3-9.

Home Directory
Plugin
Properties
Database Configuration:Home
Database Configuration:JDBC Driver
Database Configuration:J2C Alias
Database Configuration:Non-XA JDBC Provider
Database Configuration:Non-XA Datasource
Database Configuration:XA JDBC Provider
Database Configuration:XA Datasource
Database Configuration:Scheduler
Console:Module
Console:Directory
Server
Figure 3-9 Contents of .installedComponents


If the script encounters no error conditions you will see the following message:
TKLM install is complete

Immediately following the installation complete message you will be prompted with the
message shown in Example 3-17.

Example 3-17 Migration prompt
If you have EKM installed, you may migrate EKM to TKLM. If you choose not to
migrate now, you can migrate EKM separately by running the migrateEKM.sh script
but you MUST do so before configuring TKLM.
Do you want to migrate EKM to TKLM now [y/n]?

We replied no to this prompt because we are not migrating an IBM Encryption Key Manager
configuration to Tivoli Key Lifecycle Manager.

Uninstalling Tivoli Key Lifecycle Manager
If the installTKLM.sh script fails to complete, and you must correct some issue, the failed
Tivoli Key Lifecycle Manager installation must first be uninstalled before you can attempt to
install again.

Appendix A, “Troubleshooting” on page 107 contains information to assist in diagnosing
problems during the installation.

To uninstall Tivoli Key Lifecycle Manager run the uninstallTKLM.sh script found in the Tivoli
Key Lifecycle Manager installation bin directory. The syntax to execute this script is:
uninstallTKLM.sh [-wasPassword <was_password>] [-dbPassword <db_password>] [-v]

All parameters are optional and are described as follows:
<was_password> Password of the WebSphere user ID
<db_password> Password of the Database user ID
-v Send verbose output to the console (verbose output always goes
to the log)

The uninstallTKLM.sh script uses a file called tklmUninstall.response, which is created by the
installTKLM.sh script.

Should the uninstall process fail at any point, the error message and reason will be displayed
to the screen and logged. The log can be found in the log directory of the Tivoli Key Lifecycle
Manager product installation directory. An example file name of a log file is:
/var/tklm/logs/uninstallTKLM_031809_155439.log

The log name format is uninstallTKLM_<mmddyy>_<time>.log. The log file location and
name are displayed during the execution of the uninstallTKLM.sh script.

Made sure you run the command as SSRECFG. Note that you may be prompted to enter
passwords. As a test, we executed the uninstallTKLM.sh script with the following commands:
su SSRECFG
cd /var/tklm/bin
./uninstallTKLM.sh

Success will be indicated by the following message:
TKLM Uninstallation has succeeded.


3.4.9 Perform post installation steps
After Tivoli Key Lifecycle Manager has been installed into System Services Runtime
Environment, there are a few post install steps recommended by the Tivoli Key Lifecycle
Manager Installation and Configuration Guide.

Configure file-based auditing (optional)
You can optionally configure Tivoli Key Lifecycle Manager to use file-based auditing rather
than the default of sending all audit records to SMF. We chose to use the default SMF audit
records. See the Tivoli Key Lifecycle Manager Installation and Configuration Guide for the
procedure to enable filed-based auditing.

Use available authentication data when an unprotected URI is accessed
The Tivoli Key Lifecycle Manager Installation and Configuration Guide instructs you to
configure System Services Runtime Environment to use available authentication data when
an unprotected URI is accessed. To configure this setting perform the following steps.
1. Log on to the System Services Runtime Environment Integrated Solutions Console.
2. Expand the Security drop-down menu on the left and select the Secure administration,
applications, and infrastructure option Figure 3-10.

Figure 3-10 Security menu


3. From the Secure administration, applications, and infrastructure screen, expand the
Web Security menu located on the right side under Authentication, and select the
General settings option, as shown in Figure 3-11.

Figure 3-11 Web Security

4. From the General settings screen click the check box next to Use available
authentication data when an unprotected URI is accessed, then click the Apply button
as shown in Figure 3-12.

Figure 3-12 General Properties


5. A confirmation window will appear. Click Save to save directly to the master configuration
(Figure 3-13).

Figure 3-13 Confirmation

3.4.10 Stop and restart System Services Runtime Environment
System Services Runtime Environment must be restarted to pick up the configuration
changes. Stop System Services Runtime Environment as detailed in “Stopping System
Services Runtime Environment” on page 64. Then start System Services Runtime
Environment as detailed in “Starting System Services Runtime Environment” on page 63.

3.4.11 Verify installation
To verify the installation of Tivoli Key Lifecycle Manager perform the following steps:
Log on to the Integrated Solutions Console.
A new menu item called Tivoli Key Lifecycle Manager should appear in the left menu list
and as a product selection on the Welcome screen, as shown in Figure 3-14.


Figure 3-14 Verify installation

3.5 Defining a master keystore
After verifying Tivoli Key Lifecycle Manager is installed, you can now proceed to configure a
keystore. For our test environment we chose to define a JCERACFKS keystore as the master
keystore.

3.5.1 Create RACF profiles for JCERACFKS or JCECCARACFKS keystores
Prior to creating the RACF keyring associated with the JCERACFKS keystore, additional
RACF profiles must be created or modified.

For more information see the section entitled “Configuring Tivoli Key Lifecycle Manager for
z/OS to use RACF keyrings” in the Tivoli Key Lifecycle Manager Installation and
Configuration Guide.

The required commands can be found in a sample Rexx exec called racfpermissions.rexx.
This exec is located in the samples directory of the Tivoli Key Lifecycle Manager installation
directory. In our case it was located at:
/var/tklm/samples

The script contains documentation about the purpose of the commands and what must be
modified. Prior to executing the script or the equivalent commands, you must decide on the
keystore owner and the keyring name. In our case, the master keystore will be owned by
SSRECFG. The name of the keyring will be TKLMKeyRing.

The script sets four variables as shown in Table 3-10 on page 87. Replace these values with
the correct ones for your installation.


Table 3-10 Script variables
Variable Our value Description

groupid SSREGRP By default, the permissions in this sample script are granted at
the group level (that is, SSREGRP). This value can be any
RACF ID (either user ID or groupid) that needs access to the
Keyring.

user ID SSREADM This user ID is only used once in this script. The user ID should
be the System Services Runtime Environment Started Task
user ID (default: SSREADM).

ownerid SSRECFG The user ID of the owner of Master Keystore Keyring.

keyring TKLMKeyRing The keyring that the System Services Runtime Environment
group is being granted access to.

The racfpermissions.rexx exec issues the commands to define several FACILITY class
profiles if they do not already exist, as shown in Example 3-18.

Example 3-18 FACILITY profiles
RDEFINE FACILITY IRR.DIGTCERT.LISTRING UACC(NONE)
RDEFINE FACILITY IRR.DIGTCERT.LIST UACC(NONE)
RDEFINE FACILITY IRR.DIGTCERT.ADD UACC(NONE)
RDEFINE FACILITY IRR.DIGTCERT.DELETE UACC(NONE)
RDEFINE FACILITY IRR.DIGTCERT.ADDRING UACC(NONE)
RDEFINE FACILITY IRR.DIGTCERT.CONNECT UACC(NONE)
RDEFINE FACILITY IRR.DIGTCERT.GENCERT UACC(NONE)
RDEFINE FACILITY IRR.DIGTCERT.GENREQ UACC(NONE)

Since the owner of the keyring is an ID other than the started task ID, additional profiles are
created in the RDATALIB class, as shown in Example 3-19. These will allow other IDs to
access and update the keyring.

Example 3-19 RDATALIB profiles
RDEFINE RDATALIB "ownerid"."keyring".LST UACC(NONE)
RDEFINE RDATALIB "ownerid"."keyring".UPD UACC(NONE)

Group access is then granted to the FACILITY class profiles defined in Example 3-18. These
commands are shown in Example 3-20.

Example 3-20 Access to FACILITY class profiles
PERMIT IRR.DIGTCERT.LISTRING CLASS(FACILITY) ID("groupid") ACC(UPDATE)
PERMIT IRR.DIGTCERT.LIST CLASS(FACILITY) ID("groupid") ACC(UPDATE)
PERMIT IRR.DIGTCERT.ADD CLASS(FACILITY) ID("groupid") ACC(CONTROL)
PERMIT IRR.DIGTCERT.DELETE CLASS(FACILITY) ID("groupid") ACC(CONTROL)
PERMIT IRR.DIGTCERT.ADDRING CLASS(FACILITY) ID("groupid") ACC(CONTROL)
PERMIT IRR.DIGTCERT.CONNECT CLASS(FACILITY) ID("groupid") ACC(CONTROL)
PERMIT IRR.DIGTCERT.GENCERT CLASS(FACILITY) ID("groupid") ACC(CONTROL)
PERMIT IRR.DIGTCERT.GENREQ CLASS(FACILITY) ID("groupid") ACC(CONTROL)

Group access is then granted to the RDATALIB class profiles defined in Example 3-19. These
commands are shown in Example 3-21 on page 88.


Example 3-21 Access to RDATALIB profiles
PERMIT "ownerid"."keyring".LST CLASS(RDATALIB) ID("groupid") ACCESS(CONTROL)
PERMIT "ownerid"."keyring".UPD CLASS(RDATALIB) ID("groupid") ACCESS(CONTROL)

The System Services Runtime Environment started task ID is given access to the DIGTCERT
class as shown in Example 3-22.

Example 3-22 DIGTCERT access
ALTUSER "userid" CLAUTH(DIGTCERT)

Finally, the DIGTCERT class is made active if it is not already active, and the FACILITY and
RDATALIB classes are refreshed as shown in Example 3-23.

Example 3-23 Additional commands
SETR CLASSACT(RDATALIB)
SETROPTS RACLIST(FACILITY)
SETROPTS RACLIST(FACILITY) REFRESH
SETROPTS RACLIST(RDATALIB)
SETROPTS RACLIST(RDATALIB) REFRESH

3.5.2 Define the keystore
For more information on defining keystores and administering the Tivoli Key Lifecycle
Manager environment see the Tivoli Key Lifecycle Manager product documentation and Tivoli
Key Lifecycle Manager information center documents. In addition, the following documents
contain more detailed information about defining Tivoli Key Lifecycle Manager keystores:
IBM System Storage DS8000: Disk Encryption Implementation and Usage Guidelines,
REDP-4500-00
IBM System Storage Tape Encryption Solutions, SG24-7320-02

To create the keystore we performed the following steps:
1. Log in to the System Services Runtime Environment Integrated Solutions console.
2. Expand the Tivoli Key Lifecycle Manager menu.
3. Select Keystore and define the master keystore.
4. Select Configuration and define an SSL certificate.
5. You can now select Key Administration and define keys for your particular encryption
capable hardware.

3.6 Deploying additional Tivoli Key Lifecycle Manager servers
in a Sysplex
Once the initial Tivoli Key Lifecycle Manager server has been installed and configured
(keystores defined for encryption-enable tape or disk drives), additional Tivoli Key Lifecycle
Manager servers can be deployed to other LPARS in a Sysplex.


Review the guidelines and instructions in the section entitled “Installing Tivoli Key Lifecycle
Manager on z/OS Parallel Sysplex systems” of the Tivoli Key Lifecycle Manager Installation
and Configuration Guide.

To deploy our second Tivoli Key Lifecycle Manager server on an additional LPAR in our
Sysplex we performed the following steps:
1. Install System Services Runtime Environment on the second LPAR.
2. Install Tivoli Key Lifecycle Manager on the second LPAR.
3. Back up the primary Tivoli Key Lifecycle Manager server.
4. Restore the primary Tivoli Key Lifecycle Manager backup to the second Tivoli Key
Lifecycle Manager server.

3.6.1 Install System Services Runtime Environment on a second LPAR
As mentioned previously, the System Services Runtime Environment configuration file
system for each System Services Runtime Environment instance must be exclusive to that
System Services Runtime Environment instance. However, the System Services Runtime
Environment installation file system can be shared.

We installed the additional System Services Runtime Environment instance on system SC62.
We used /var/ssreconf as the mount point for the System Services Runtime Environment
configuration filesystem. Note that although this is the same directory name and path as our
first system, it is unique to this second instance due to the way our symbolics are resolved in
our share UNIX System Services directory structure: essentially, /var is a system-specific
directory.

When creating multiple System Services Runtime Environment instances in a Sysplex while
using different hostnames, the installation variables identified in Table 3-11 must be unique.

Table 3-11 System Services Runtime Environment configuration changes
Variable Our first instance Our second instance

_SSRE_CELL_NAME_ SSRE SSRE62

_SSRE_PROC_PREFIX_ SSRE SSRE62

_SSRE_CONFIG_FS_ BBA.SSRECONF.ZFS BBA.SSRECONF.SC62.ZFS

_SSRE_CONFIG_DIRECTORY_ /var/ssreconf /var/ssreconfa

_SSRE_SYSTEM_NAME_ wtsc61.itso.ibm.com wtsc62.itso.ibm.com
a. Although these directories are named the same they resolve to unique zFS filesystems.

See the section “Creating multiple instances of IBM System Services Runtime Environment
for z/OS” in the IBM System Services Runtime Environment for z/OS Configuration Guide for
additional information.

We followed the steps detailed in “System Services Runtime Environment installation and
configuration overview” on page 50 to install this second instance of System Services
Runtime Environment.

Because we are sharing a RACF database, many of the commands issued by the
modifySSRE.sh script were redundant; however, there are several unique profiles and
certificates created for this instance, so we ran the modifySSRE.sh script again.


Once installed, verify that you can access the Integrated Solutions Console on the newly
created System Services Runtime Environment instance.

3.6.2 Install Tivoli Key Lifecycle Manager on the second LPAR
You must create a unique Tivoli Key Lifecycle Manager installation product file system. We
created another zFS for the second Tivoli Key Lifecycle Manager and uncompressed (via tar
command) the Tivoli Key Lifecycle Manager product installation packages into that new zFS.

We followed the same steps as detailed in “Tivoli Key Lifecycle Manager installation” on
page 64, with the following exception:
Since the DB2 subsystems of the first and second Tivoli Key Lifecycle Manager servers
are in a datasharing group, we did not run the commands in the SPUFI files to create the
Tivoli Key Lifecycle Manager database, because it has already been created.

Verify that Tivoli Key Lifecycle Manager has been installed into the second instance.
Attempting to access the Tivoli Key Lifecycle Manager configuration will result in an error; you
must synchronize with the primary Tivoli Key Lifecycle Manager before this instance
becomes usable.

3.6.3 Back up the primary Tivoli Key Lifecycle Manager server
We backed up the required Tivoli Key Lifecycle Manager data of our primary Tivoli Key
Lifecycle Manager (SC61) using the procedure detailed in 4.2, “Backup procedures” on
page 95.

Since our RACF database is shared, there is no need to transfer any key materials. Likewise,
the DB2s are in a data sharing group, so no DB2 data must be transferred.

3.6.4 Restore the primary Tivoli Key Lifecycle Manager backup to the second
Tivoli Key Lifecycle Manager server
We restored the backup of our primary Tivoli Key Lifecycle Manager to the second Tivoli Key
Lifecycle Manager instance (SC62) using the procedures detailed in 4.3, “Restore
procedures” on page 100.

3.6.5 Shut down and restart the second Tivoli Key Lifecycle Manager server
We performed a shutdown and a restart of the second Tivoli Key Lifecycle Manager instance
(SC62). The second Tivoli Key Lifecycle Manager server then became functional.

3.7 Managing the SSRECFG user ID password
If you use a password which expires or must be changed periodically for the SSRECFG user
ID, the new password must be updated in the System Services Runtime Environment JAAS -
J2C authentication data panels or else Tivoli Key Lifecycle Manager will be unable to connect
to DB2.


Additional information on changing the administrator password and DB2 password on z/OS
systems can be found at:
lm.doc/install/cpt/cpt_insguide_config_passwordissues_zos.html

Perform the following steps to change the password for the SSRECFG user ID within the
System Services Runtime Environment:
1. Log in to the ISC console.
2. Click Security → Secure administration, applications, and infrastructure.
3. From the Secure administration, applications, and infrastructure panel, under
Authentication click Java Authentication and Authorization Services → J2C
authentication data.
4. From the JAAS - J2C authentication data panel click on each alias (tklm_db and tklmdb)
and update the password in the General Properties panel for each alias entry.

In addition, all the tklm response files created during installation, migration, and update will
have a copy of your password as well. If you want to use the same response files for future
service these files need to updated with the new password.


4

Chapter 4. Tivoli Key Lifecycle Manager
backup and restore
This chapter discusses procedures for backing up and restoring Tivoli Key Lifecycle Manager
data on z/OS. This is useful for backing up a Tivoli Key Lifecycle Manager environment,
consisting of:
Tivoli Key Lifecycle Manager configuration data
Tivoli Key Lifecycle Manager DB2 tables
Keystores
Keyrings and certificates

Backups might be done for disaster recover purposes, or to clone one Tivoli Key Lifecycle
Manager server to another Tivoli Key Lifecycle Manager server.

This chapter also covers procedures for exporting and importing keys and related data from
one Tivoli Key Lifecycle Manager server to another Tivoli Key Lifecycle Manager server.


4.1 Backup and restore of Tivoli Key Lifecycle Manager data
Each installation should establish procedures for backing up and restoring their Tivoli Key
Lifecycle Manager environment. Backing up the environment means capturing all the data
required to recreate that instance of Tivoli Key Lifecycle Manager.

Backing up Tivoli Key Lifecycle Manager data consists of capturing configuration information,
metadata stored in the associated database files, and the encryption key material. The
processes required to create the backup information are highly dependent on the keystore
type in use.

Table 4-1 summarizes what is required for backup and restore based on the type of keystore
that is implemented. The data being backed up might be Tivoli Key Lifecycle Manager
configuration data, DB2 tables, keyrings and key material residing in ICSF datasets.

Table 4-1 Backup and restore requirements
Keystore Backup and restore requirements

JCEKS Back up and restore Tivoli Key Lifecycle Manager configuration data
using Tivoli Key Lifecycle Manager GUI
Keystore is backed up as part of Tivoli Key Lifecycle Manager GUI
backup/restore
Back up DB2 Tables using DB2 backup procedures

JCERACFKS Back up and restore Tivoli Key Lifecycle Manager configuration data
Back up and restore DB2 Tables using DB2 backup and restore
procedures
Back up and restore SAF based keyrings, keys and certificates

JCECCAKS Back up and restore Tivoli Key Lifecycle Manager configuration data
procedures
Back up and restore ICSF CKDS and PKDS data

JCECCARACFKS Back up and restore Tivoli Key Lifecycle Manager configuration data
procedures
Back up and restore SAF based keyrings, keys and certificates
Back up and restore ICSF PKDS data

These backups can be integrated into your existing disaster recovery backup procedures. For
example, if you are taking point in time backups (whether full volume dumps or using some
form of DASD mirroring), make sure you back up:
HFS or zFS file systems containing Tivoli Key Lifecycle Manager configuration data
RACF database
DB2 unloads or image copies of Tivoli Key Lifecycle Manager DB2 tables
ICSF CKDS and PKDS datasets

However, if you need to clone or synchronize data between two Tivoli Key Lifecycle Manager
for z/OS instances which have discrete environments the following procedures are a useful
starting point.


4.2 Backup procedures
This section covers the procedures followed to perform backups.

4.2.1 Backing up Tivoli Key Lifecycle Manager configuration data
In all cases, the Tivoli Key Lifecycle Manager configuration data must be backed up. This
data is internal to Tivoli Key Lifecycle Manager. The Tivoli Key Lifecycle Manager GUI is used
to generate the backup files for the configuration data. From the Integrated Solutions Console
(ISC), select Backup and Restore from the Tivoli Key Lifecycle Manager Settings Tasklist,
as shown in Figure 4-1.

Figure 4-1 Tivoli Key Lifecycle Manager tasklist from ISC

Select Create Backup from the Backup and Restore panel as shown in Figure 4-2 on
page 96.

Chapter 4. Tivoli Key Lifecycle Manager backup and restore 95

Figure 4-2 Backup and Restore panel

Fill in the Create Backup panel with the location, password, and description of the backup
being created. Click the Create Backup button to generate the backup. An example is shown
in Figure 4-3 on page 97.


Figure 4-3 Create backup

At this point, the configuration data is saved in a .jar file on the z/OS image at the location
defined in the parameters panel. Message CTGKM0241I is displayed as shown in Figure 4-4.

Figure 4-4 CTGKM0241I message

The new backup will be displayed on the Backup and Restore panel as shown in Figure 4-2.

4.2.2 Backing up DB2 tables
Backing up the tables used to store Tivoli Key Lifecycle Manager for z/OS metadata in DB2
should be done through standard DB2 backup procedures using DB2 utilities such as image
copy and unload at the database or tablespace level.


4.2.3 Backing up a JCEKS keystore
To extract key material from the Tivoli Key Lifecycle Manager keystore, use the
AdminTask.tklmKeyExport command from the Tivoli Key Lifecycle Manager command line
interface.

The following script (Example 4-1) executes the wsadmin.sh shell script and passes a python
file containing the CLI commands to be executed. In this example, wsadmin.sh is called from
zTKLMExport.sh script.

Example 4-1 zTKLMExport.sh example shell script
/usr/lpp/ssre/ssreconf/AppServer/bin/wsadmin.sh
-username <Authorized User ID>
-password <Authorized user ID's password>
-lang jython
-f exportKey.py

Example 4-2 shows are the contents of the exportKey.py file containing the commands to
export key material. The key material is placed into a password protected PKCS#12
formatted file. Multiple keys can be exported to multiple files as shown.

Example 4-2 exportKey.py file
print "Exporting key file test1.p12"
print AdminTask.tklmKeyExport('[-fileName test1.p12 -alias test.key.1 -type privatekey
-password "password" -keyStoreName "Tivoli Key Lifecycle Manager Keystore”]')
print “------------------------------------------------------------------------------------”
print "Exporting key file test2.p12"
print AdminTask.tklmKeyExport('[-fileName test2.p12 -alias test.key.2 -type privatekey
-password "password" -keyStoreName "Tivoli Key Lifecycle Manager Keystore”]')

To further understand this method, consider the AdminTask.tklmKeyExport command as
described in the Tivoli Key Lifecycle Manager information center:

http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/topic/com.ibm.tklm.doc/ref/
ref_ic_cli_key_export.html

4.2.4 Backing up a JCERACFKS keyring
Tivoli Key Lifecycle Manager for z/OS can store key material in the external security manager
on z/OS. Keys stored in the external security manager are accessible by Tivoli Key Lifecycle
Manager through JCERACFKS.

Tivoli Key Lifecycle Manager requires that key material be imported in the PKCS#12 format.
To export keys from RACF in that format, use the RACDCERT EXPORT command shown in
Example 4-3.

Example 4-3 RACDCERT EXPORT command
RACDCERT ID(EKMSERV) EXPORT( LABEL(‘<key label or alias') ) DSN ’fully qualified
dataset name’) PASSWORD(<<<PASSWORD>>>)


4.2.5 Backing up a JCECCARACFKS keyring
JCECCARACFKS uses a RACF keyring to store certificate information. It is important to back
up the certificate information using RACFDCERT EXPORT as described in the previous
section.

In order to extract the private key material from ICSF's PKDS, use the KEYXFER tool as
shown in Example 4-4.

Example 4-4 KEYXFER command
KEYXFER WRITE, <<key label or alias>>, <<output dataset name>>
:

Note: The output dataset contains key material encrypted using the Asymmetrical Master
Key of the source system. In order to import the keypair to a target z/OS system, the same
Asymmetrical Master Key must be in use at the target system.

4.2.6 Backing up ICSF datasets
Backing up the ICSF CKDS and PKDS datasets should be done using standard VSAM
dataset backup procedures. A sample JCL job is shown in Example 4-5.

Example 4-5 JCL to backup CKDS
//LOAD1 EXEC PGM=IDCAMS
//INDD DD DISP=SHR,DSN=SYS1.SCSFCKDS
//OUTDD DD DISP=SHR,DSN=SYS1.SCSFCKDS.BACKUP
//SYSIN DD *
REPRO INFILE(INDD) OUTFILE(OUTDD)
/*


4.3 Restore procedures
This section describes procedures used to perform restores.

4.3.1 Restoring Tivoli Key Lifecycle Manager configuration data
From the ISC, select Backup and Restore from the Tivoli Key Lifecycle Manager Settings
Tasklist, as shown in Figure 4-5.

Figure 4-5 Tivoli Key Lifecycle Manager tasklist from ISC

Select Create Backup from the Backup and Restore panel as shown in Figure 4-6 on
page 101.


Figure 4-6 Backup and Restore panel

Select the desired backup of the Tivoli Key Lifecycle Manager configuration data by clicking
the checkbox to the left of the backup file name.

Click Restore from Backup. This action brings up the Restoration parameters panel. Enter
the password of the backup file (see Figure 4-7 on page 102).

Click Restore Backup.


Figure 4-7 Restore panel

A restoration confirmation message is displayed as shown in Figure 4-8.

Figure 4-8 Restore confirm message

Click OK to confirm and complete the restore process.

Note: After restoring the configuration data, Tivoli Key Lifecycle Manager must be
restarted.


4.3.2 Restoring DB2 Tables
Backing up the tables used to store Tivoli Key Lifecycle Manager metadata in DB2 should be
done through standard DB2 backup procedures using utilities such as image copy restore
and load at the database or tablespace level.

4.3.3 Restoring a JCEKS keystore
To add key material to the Tivoli Key Lifecycle Manager keystore, use the
AdminTask.tklmKeyImport command from the Tivoli Key Lifecycle Manager command line
interface.

The following script (Example 4-6) executes the wsadmin.sh shell script and passes a python
file containing the CLI commands to be executed. In this example, wsadmin.sh is called from
zTKLMImport.sh script.

Example 4-6 zTKLMImport.sh
/usr/lpp/ssre/ssreconf/AppServer/bin/wsadmin.sh
-username <Authorized User ID>
-password <Authorized user ID's password>
-lang jython
-f importKey.py

Example 4-2 shows the contents of the exportKey.py file containing the commands, including
key alias and type of key, to import key material. Multiple keys can be imported from multiple
files as shown.

Example 4-7
print "Importing key file test1.p12"
print AdminTask.tklmKeyImport('[-fileName test1.p12 -usage DS8K -alias test.key.1 -type
privatekey -password "password" -keyStoreName "Tivoli Key Lifecycle Manager Keystore”]')
print “------------------------------------------------------------------------------------”
print "Importing key file test2.p12"
print AdminTask.tklmKeyImport('[-fileName test2.p12 -usage DS8K -alias test.key.2 -type
privatekey -password "password" -keyStoreName "Tivoli Key Lifecycle Manager Keystore”]')

To further understand this method, consider the AdminTask.tklmKeyImport command as
described in the Tivoli Key Lifecycle Manager information center:

lm.doc/ref/ref_ic_cli_key_import.html

Removing keys from a PKCS#12 file
It is possible that the PKCS#12 file contains multiple keys. Some levels of Tivoli Key Lifecycle
Manager will not accept this as a valid input file. To remove the unwanted keys, perform
these steps:
1. Using keytool, which comes with the JVM™, display the contents of the PKCS#12 file:
keytool -keystore <path and name of pkcs12 file> -storetype pkcs12 -list
You will be prompted for a password. If this key was exported using keytool, use the
password you entered during the export process. If this key was exported using Tivoli Key
Lifecycle Manager CLI then use the password of the Tivoli Key Lifecycle Manager
keystore.


You should see output similar to Figure 4-9.

Keystore provider: IBMJCE
Your keystore contains 2 entries
decp, Dec 31, 1969, keyEntry,
Certificate fingerprint (MD5): 6C:31:31:3B:7E:4F:35:67:D3:AA:30:81:61:03:2A:1C
decd, Dec 31, 1969, trustedCertEntry,
Figure 4-9 List of PKCS#12

2. The entry you want to delete is the trustedCertEntry. The command to do this is:
keytool -keystore <path and name of pkcs12 file> -storetype pkcs12 -delete
-alias <alias of the trustedCertEntry>
keystore.
3. List the contents of the PKCS#12 file again:
keytool -keystore <path and name of pkcs12 file> -storetype pkcs12 -list
keystore. You should see output similar to Figure 4-10.

Keystore type: pkcs12
Keystore provider: IBMJCE
Your keystore contains 1 entry
decp, Dec 31, 1969, keyEntry,
Figure 4-10 Updated list of PKCS#12

4. Use the Admin.tklmKeyImport command to import the key.

4.3.4 Restoring a JCKRACFKS keyring
Tivoli Key Lifecycle Manager requires that key material be imported via the tklmImport
command as described in the previous section.

4.3.5 Restoring a JCECCARACFKS keyring
Tivoli Key Lifecycle Manager requires the key material be imported via the tklmImport
command described previously.

Tivoli Key Lifecycle Manager must be defined to use hardware cryptography on the target
system through the keystore setup panel. The checkbox to enable ICSF encryption of keys
must be selected.


Figure 4-11 Keystore panel

In order to store key material in ICSF, the private key must be in the PKCS#12 file. This is
only possible if the source system is not using hardware cryptography. Keys protected by
ICSF on z/OS cannot be extracted using the tlkmExport command.

4.3.6 Restoring ICSF datasets
Restoring the ICSF CKDS and PKDS datasets should be done using standard VSAM dataset
backup procedures (IDCAMS). An example is shown in Example 4-8.

Example 4-8 IDCAMS example
//LOAD1 EXEC PGM=IDCAMS
//INDD DD DISP=SHR,DSN=SYS1.SCSFCKDS.BACKUP
//OUTDD DD DISP=SHR,DSN=SYS1.SCSFCKDS
//SYSIN DD *
REPRO INFILE(INDD) OUTFILE(OUTDD)
/*


A

Appendix A. Troubleshooting
This appendix provides troubleshooting hints and tips.


A.1 Problems with System Services Runtime Environment
installation and configuration

A.1.1 +BBOJ0095W: JAVA VERSION/LEVEL IS NOT SUPPORTED BY
WEBSPHERE FOR Z/OS
This error message is actually expected for System Services Runtime Environment Fixpack 1
because System Services Runtime Environment Fixpack 1 contains a Java ifix that is specific
to Tivoli Key Lifecycle Manager and is not shipped with regular WebSphere Application
Server for z/OS. Figure A-1 shows the full error message that will be displayed.

+BBOJ0011I: JVM Build is J2RE 1.5.0 IBM J9 2.3 z/OS s390-31 689
j9vmmz3123-20080710 (JIT enabled) J9VM - 20080625_20583_bHdSMr
JIT - 20080620_1845_r8 GC - 200806_19.
+BBOJ0095W: JAVA VERSION/LEVEL IS NOT SUPPORTED BY WEBSPHERE FOR Z/OS
+BBOJ0051I: PROCESS INFORMATION: S0053767/SSRE8KS , ASID=958(0x3be), 691
PID=17367529(0x10901e9)
Figure A-1 Java version/level not supported

Again, this is expected, and should be considered working as designed.

Note: Only the Java ifix level that is embedded within System Services Runtime Environment
Fixpack 1 is supported by System Services Runtime Environment and Tivoli Key Lifecycle
Manager. Pointing System Services Runtime Environment to another version of Java is not a
supported configuration.

A.1.2 Problem starting up System Services Runtime Environment:
INSUFFICIENT AUTHORITY TO OPEN applyPTF.sh
This failure indicates that there may be a corruption within the System Services Runtime
Environment Configuration HFS. Figure A-2 shows the full error message.

ICH408I USER(SSREADM ) GROUP(SSREGRP ) NAME(System Services Runtime Environment
ADMINISTRATOR ) 276
/Ssreconf/SSRE.NODE1.System Services Runtime Environment.HOME/bin/applyPTF.sh
CL(FSOBJ ) FID(00001DCF000000010000000000000000)
INSUFFICIENT AUTHORITY TO OPEN
ACCESS INTENT(--X) ACCESS ALLOWED(OTHER ---)
EFFECTIVE UID(0000001234) EFFECTIVE GID(0000004321)

Figure A-2 Insufficient authority to open

The root of this problem is usually that the System Services Runtime Environment Product
dataset was mounted in read/write mode while the System Services Runtime Environment
configuration scripts created the System Services Runtime Environment configuration HFS.
The System Services Runtime Environment Product dataset must be mounted in read only
mode at all times. If this dataset is mounted in read/write mode the contents of the System
Services Runtime Environment Product dataset will likely become corrupted and System
Services Runtime Environment will need to be re-installed and re-configured.


A.1.3 RACF ICH408I permission messages for SSRECFG and SSREADM
This indicates that the SSRECFG or SSREADM user IDs have not been given the necessary
permissions to use the RACF keyring that is configured with Tivoli Key Lifecycle Manager.

A.1.4 System Services Runtime Environment PDSE is not APF authorized
An ABEND=S047 will be displayed on z/OS console when starting System Services Runtime
Environment. For example:
SY1 IEF450I System Services Runtime Environment - ABEND=S047 U0000
REASON=00000000 TIME=12.22.11

APF authorize the System Services Runtime Environment load library. For example:
SETPROG APF,ADD,DSN= BBA.SBBALOAD,VOL= BBAVOL

A.1.5 System Services Runtime Environment PDSE is not cataloged
In this case when running the configSSRE.sh shell script to configure System Services
Runtime Environment you receive back a failure. The System Services Runtime Environment
log file will contain the message shown in Figure A-3.

java.lang.NoClassDefFoundError: com.ibm.ws.management.util.PlatformHelperImpl
(initialization failure)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:132)
at java.lang.Class.forNameImpl(Native Method)
....
Caused by: java.lang.UnsatisfiedLinkError: bbouph (Not found in
java.library.path)
Figure A-3 Error log entry

Note: This can also happen if you use different values for the _SSRE_PDSE_ variable when
running the createSSRE.sh and modifySSRE.sh shell scripts.

You can use TSO option 3.4 to specify the dataset name and the volume name, then use the
"c" line command to catalog the dataset.

A.1.6 System Services Runtime Environment file system is not mounted or
the path is incorrect
When you run the System Services Runtime Environment shell scripts you will receive back:
FSUM7351 not found message

To correct this you should verify that the _SSRE_CODE_DIRECTORY_ value points to your
System Services Runtime Environment file system mount point. If the System Services
Runtime Environment file system is not mounted, you will need to manually mount the HFS.
For example:
MOUNT FILESYSTEM('BBA.SBBAZFS') TYPE(ZFS) MODE(READ)
MOUNTPOINT('/usr/lpp/SSRE/V1R1')

Appendix A. Troubleshooting 109

A.1.7 System Services Runtime Environment was started but modifySSRE.sh
or equivalent security setup commands were not executed
Figure A-4 is an example of the error messages that will be displayed on the z/OS console.

SY1 IRR813I NO PROFILE WAS FOUND IN THE STARTED CLASS FOR
SERVR0 WITH JOBNAME SERVR0. RACF WILL USE ICHRIN03.
SY1 $HASP100 SERVR0 ON STCINRDR
SY1 $HASP373 SERVR0 STARTED
SY1 ICH408I JOB(SERVR0 ) STEP(SERVR0 ) CL(PROCESS )
OMVS SEGMENT NOT DEFINED
SY1 $HASP395 SERVR0 ENDED
SY1 IEE132I START COMMAND DEVICE ALLOCATION ERROR
Figure A-4 z/OS console messages

To correct this problem you will need to execute the modifySSRE.sh shell script or perform the
equivalent security setup.

A.1.8 Trying to start System Services Runtime Environment but the
Configuration file system is not mounted
Figure A-5 is an example of the error that will be displayed on the z/OS console.

SY1 IRR812I PROFILE SSRE*.* (G) IN THE STARTED CLASS WAS USED TO START System
Services Runtime Environment WITH JOBNAME System Services Runtime Environment.
SY1 $HASP100 System Services Runtime Environment ON STCINRDR
SY1 $HASP373 System Services Runtime Environment STARTED
SY1 $HASP395 System Services Runtime Environment ENDED
SY1 IEE132I START COMMAND DEVICE ALLOCATION ERROR
Figure A-5 z/OS console messages

To correct this problem you will need to manually mount the System Services Runtime
Environment configuration dataset. For example:
MOUNT FILESYSTEM(' BBA.SSRECONF.ZFS ') TYPE(ZFS) MODE(RDWR)
MOUNTPOINT('/ssreconf')

A.1.9 Multiple browsers windows are logged into the same System Services
Runtime Environment instance
When a second browser is used to log on to System Services Runtime Environment's ISC
admin console, the first browser will be logged off.

To avoid this problem the user should only use one browser at a time to log on to the System
Services Runtime Environment ISC admin console.


A.1.10 Unable to resolve the System Services Runtime Environment
hostname and get to the ISC admin console
In this case you are attempting to log on to the ISC admin console but you receive back an
error from the browser indicating that the page cannot be found.

To resolve this problem verify that the hostname you have specified for the
_SSRE_SYSTEM_IPNAME_ value during the System Services Runtime Environment
configuration is valid. If it is valid, attempt to ping this value from a command prompt.

A.1.11 Unable to make updates on the Tivoli Key Lifecycle Manager GUI
When you try to update one of the settings on the Tivoli Key Lifecycle Manager GUI you get
back an error indicating that the server parameters are not initialized. For example, when you
try to create a certificate you get back:
CTGKM0207E Unable to create certificate.
Server Parameters not initialized

The likely cause is that you have not run the sample Rexx RACF permissions script,
samples/racfpermissions.rexx. This script should be executed in order to give the SSRECFG,
SSREADM, and SSREGRP IDs access to your RACF keyring. Tivoli Key Lifecycle Manager
will be unable to load the keyring and will fail to update any settings or create further key
material if this script is not executed.

A.1.12 Security errors from running the System Services Runtime
Environment scripts
For security setup errors when running the System Services Runtime Environment
configuration scripts, verify your customizations to the RACFMSTR setup exec. The
RACFMSTR setup exec is updated when you execute the createSSRE.sh script with the
values from your System Services Runtime Environment configuration file. The RACFMSTR
setup exec is then executed when you run the modifySSRE.sh shell script. All messages from
the RACFMSTR will be directed to the output file created by the modifySSRE.sh shell script.
If further customizations or corrections are needed to your RACFMSTR, you can run it
separately from the System Services Runtime Environment configuration shell scripts.

A.1.13 Cell name and port number conflicts with System Services Runtime
Environment
For System Services Runtime Environment configuration errors you should ensure that your
cell name and port numbers do not conflict with another instance of WebSphere Application
Server or any other application running on your system. Remember when performing the
Tivoli Key Lifecycle Manager for z/OS Sysplex install that each instance of System Services
Runtime Environment must use a unique cell name.

A.1.14 System Services Runtime Environment errors, abends, hang
conditions
The messages in syslog and the System Services Runtime Environment joblogs should be
examined when System Services Runtime Environment container errors, abends, hang
conditions, and out of resource problems occur. There are three System Services Runtime


Environment joblogs active when System Services Runtime Environment is started up, one
each for the WebSphere Application Server daemon, Control Region (CR), and Servant
Region (SR). You might want to start with the Servant Region and then work to the others. To
determine errors look for the WebSphere Application Server BBO- messages.

Also examine the WebSphere Application Server First Failure Data Capture (FFDC) reports.
These are created in your SSRE_CONFIG_ROOT/profiles/default/logs/ffdc directory. These
reports may contain Java stack traces that indicate what the problem is and where exactly
within the WebSphere Application Server or Tivoli Key Lifecycle Manager code the problem
exists.

The following link brings you to a Redpaper Collection for WebSphere Application Server
V6.1 Problem Determination:

http://www.redbooks.ibm.com/abstracts/sg247461.html?Open

There is also a Redpaper, WebSphere for z/OS problem determination means and tools,
REDP-6880 that covers this topic, located at:

http://www.redbooks.ibm.com/abstracts/redp6880.html?Open

The following WebSphere Application Server (z/OS), Version 6.1 IBM Information Center
pages are also helpful with problem determination:
Troubleshooting and support:
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.
websphere.zseries.doc/info/zseries/ae/welc6toptroubleshooting.html
Diagnosing problems (using diagnostic tools):
websphere.zseries.doc/info/zseries/ae/ttrb_diagfix.html

Runtime abends
Runtime abends should be sent to the service team for further analysis. A CC3 indicates a
daemon abend, DC3 indicates a control region abend, and EC3 indicates a servant region
abend.

Hangs
If you have a hang condition you can issue the following command to determine if
WebSphere Application Server is responding:
F <SSRE>, DISPLAY

If System Services Runtime Environment responds you should get back something like the
message shown in Figure A-6.

15.29.00 STC00496 BBOO0173I SERVER System Services Runtime Environment/System
Services Runtime Environment ACTIVE ON DCEIMGWZ AT LEVEL
6.1.0.19.
15.29.00 STC00496 BBOO0188I END OF OUTPUT FOR COMMAND DISPLAY
Figure A-6 z/OS console message

Set WebSphere Application Server hang-detection variables with the Admin Console to help
diagnose hang conditions (com.ibm.websphere.threadmonitor.interval, and so forth).


Check for enqueue contention by issuing command:
D GRS,C

If there are not enqueue contentions you should get back something like the message shown
in Figure A-7.

15.29.37 ISG343I 15.29.34 GRS STATUS 599
NO ENQ RESOURCE CONTENTION EXISTS
NO LATCH CONTENTION EXISTS
Figure A-7 z/OS console message

Look in the syslog and the System Services Runtime Environment Control Region and
Servant Region job logs for long gaps of time between message timestamps. This would
indicate that System Services Runtime Environment is hung up on something. Repeated
messages or patterns of messages could also indicate that System Services Runtime
Environment is stuck in a loop or period of excessive activity.

To detect high CPU conditions use RMF, SDSF DA display, Omegamon, and so forth.

Check syslog and System Services Runtime Environment WebSphere Application Server
CR/SR joblogs to look for messages indicating a potential loop or period of excessive activity.

Diagnosing: Increase z/OS internal trace (TRACE ST,999K) and collect console dump
(DUMP COMM=xxxx).

For both conditions, system monitoring tools like Omegamon can also help pinpoint which
address spaces are possibly having a problem (that is, CR, SR, or daemon) and provide a PD
starting point.

See the following for additional information regarding Java diagnostics:
http://publib.boulder.ibm.com/infocenter/javasdk/v5r0/index.jsp?topic=/com.ibm.jav
a.doc.diagnostics.50/diag/welcome.html

To view the service log use:
showlog service_log_filename [output_filename]

A.1.15 Collecting data for IBM support center when opening a PMR
When reporting problems to the IBM support center, a PMR will be opened. When the PMR is
opened the following data should be gathered and forwarded to the service team:
Problem description
Software levels (WebSphere Application Server version Info output)
z/OS version and PUT level
Joblogs for CR and SR
SVCDUMP

Submit to IBM under a PMR if you are unable to determine the error.

For additional details, see:
http://publib.boulder.ibm.com/infocenter/javasdk/v5r0/index.jsp?topic=/com.ibm.
java.doc.diagnostics.50/diag/submitting_problems/bugs.html


A.1.16 Additional diagnostic requests by IBM support center
The IBM support center may additionally request and provide instructions for collecting:
A trace (CTRACE or WebSphere Application Server Java package level trace) as
additional problem documentation activated with z/OS console commands.
An example of the console command to activate a Java package-level trace is:
F <SSRE>,tracejava='com.ibm.ws.webservices.*=all'
To activate a security trace:
F <SSRE>,tracedetail=E, F <SSRE>,tracebasic=E
F <SSRE>, tracejava='com.ibm.ws.security.*=all=enabled'
Important: Depending on the selection, these traces can generate a lot of messages to the
WebSphere Application Server JES joblog.
A SLIP or console dump.

A.1.17 Taking a console dump of System Services Runtime Environment
Always dump System Services Runtime Environment (controller + servant), OMVS plus
related components.

Set up your IEACMDxx in USER.PARMLIB, for example, IEACMDSR

Example A-1 IEACMDxx example
DSPNAME=('OMVS'.*)
JOBNAME=(SSRE*) (put in the name of the Jobs to start System Services Runtime
Environment controller and servant)
DATA=(PSA,CSA,LPA,LSQA,RGN,SQA,SUM,SWA,TRT,ALLNUC,GRSQ),
END

Issue a dump command from the operator console, for example
DUMP COMM=(System Services Runtime Environment server as client hung), PARMLIB=SR

A.1.18 Dynamic tracing with ISC
To enable tracing on a running server using the Integrated Service Console perform the
following steps:
1. Start the ISC.
2. Click Servers → Application Servers → server_name → Troubleshooting →
Diagnostic Trace Service.
3. Select the Runtime tab.
4. Select the Save runtime changes to configuration as well check box if you want to
write your changes back to the server configuration.
5. Change the existing trace state by changing the trace specification to the desired state.
6. Configure the trace output if a change from the existing one is desired.
7. Click Apply.


A.1.19 Dynamic tracing using Modify
Use the modify command from the MVS console to dynamically modify product operations.

You can use the modify command to display status of various server components and
activities, including:
Active controllers
Trace settings
Servants
Sessions
Java Virtual Machine (JVM) Heap
Java trace

Use the following format when entering the modify command:
f <server>, options

Dynamic tracing using Modify - Options
CANCEL
Used to cancel a server. You can specify the following options:
– ARMRESTART
Specify this option if you are using ARM and want an Application Response
Management (ARM) agent to restart the server after it terminates. If you do not specify
the ARMRESTART option on the CANCEL parameter, then ARM does not restart the
server.
– HELP
Get help for the CANCEL syntax.
Avoid trouble: You cannot use the CANCEL parameter to cancel a cluster from the
MVS console. Instead, you must cancel each of the servers that make up the cluster.
TIMEOUTDUMPACTION=n
Used to indicate which of the following actions is performed whenever a timeout occurs for
work that has been dispatched to a servant when the control_region_timeout_delay
property is set to a non-zero value:
– If NONE, or none is specified, no dump is taken.
– If JAVACORE or javacore is specified, a Java core dump is taken.
– If SVCDUMP or svcdump is specified, an SVC dump is taken.
TIMEOUTDUMPACTIONSESSION=n
Used to indicate which of the following actions is performed whenever a timeout occurs for
an HTTP, HTTPS, SIP, or SIPS request that has been dispatched to a servant, and the
corresponding recovery property is set to SESSION:
– If NONE, or none is specified, no dump is taken.
– If JAVACORE or javacore is specified, a Java core dump is taken.
– If SVCDUMP or svcdump is specified, an SVC dump is taken.
Following is a list of the corresponding recovery properties:
– protocol_http_timeout_output_recovery
– protocol_https_timeout_output_recovery
– protocol_sip_timeout_output_recovery
– protocol_sips_timeout_output_recovery


TRACEALL=n
Used to establish a general trace level for the server. The following are valid trace levels.
Typically, you should specify a value of 1.
0: No tracing is performed.
1: Tracing is performed when an exception occurs.
2: Basic tracing is performed.
3: Detailed tracing for all components is performed.
Avoid trouble: Be careful when using a level of 3 because this level of tracing might yield
more data than can be handled reasonably.
TRACEBASIC=n
Used to specify the product components for which you want to switch on a basic level of
tracing. This command has the ability to override a different tracing level established by
TRACEALL for those components. Note: Do not change this variable unless directed by
IBM service personnel. Table A-1 shows the values you can specify for this parameter. You
can specify one or more of these values for either TRACEBASIC or TRACEDETAIL.
Table A-1 TRACEBASIC or TRACEDETAIL settings
Value Product

0 RAS

1 Common Utilities

3 COMM

4 ORB

6 OTS

7 Shasta

9 OS/390® Wrappers

A Daemon

E Security

F Externalization

J JRAS (internal tracing that should only be
used under direction from IBM support)

L J2EE™

TRACEDETAIL=n
Used to specify the product components for which you want to turn on a detailed level of
tracing. This command activates the most detailed tracing for the specified product
components and overrides different settings in TRACEALL. The selected components are
identified by their component IDs, which are the same IDs as are listed for the
TRACEBASIC parameter. Subcomponents, specified by numbers, receive detailed
traces. Other parts of the product receive tracing as specified on the TRACEALL
parameter.
Avoid trouble: Do not change this variable unless directed by IBM service personnel.
TRACESPECIFIC=xxyyyzzz
Used to specify tracing overrides for specific product trace points.


Trace points are specified by 8-digit, hexadecimal numbers. To specify more than one
trace point, use parentheses and separate the numbers with commas. You can also
specify an environment variable name by enclosing the name in single quotes. The value
of the environment variable will be handled as if you had specified that value on the
TRACESPECIFIC parameter.
Avoid trouble: Do not use TRACESPECIFIC unless directed by IBM service personnel.
TRACE_EXCLUDE_SPECIFIC=xxyyyzzz
Used to specify product trace points to exclude. Trace points to exclude are specified by
8-digit, hexadecimal numbers. To specify more than one trace point, use parentheses and
separate the numbers with commas. You can also specify an environment variable name
by enclosing the name in single quotes. The value of the environment variable is handled
as if you specify that value on the TRACE_EXCLUDE_SPECIFIC parameter. You can use
the TRACE_EXCLUDE_SPECIFIC parameter as a mask to turn off otherwise-on traces.
For example, use the TRACESPECIFIC command to turn on tracing for a whole part of
the product, and then use the TRACE_EXCLUDE_SPECIFIC parameter to turn off one
trace within that part of the product.
Avoid trouble: Do not use TRACE_EXCLUDE_SPECIFIC unless directed by IBM service
personnel.
TRACEINIT
Used to reset to the initial trace settings.
TRACENONE
Used to turn off all trace settings.
TRACETOSYSPRINT={YES|NO}
Used to select whether to send the trace to SYSPRINT.
Specifying YES sends the trace to SYSPRINT, and specifying NO stops the sending of
the trace to SYSPRINT.
TRACETOTRCFILE={YES|NO}
Used to select whether to direct the trace to the TRCFILE DD card. Specifying YES sends
the trace to the TRCFILE DD card, and specifying NO stops the sending of the trace to the
TRCFILE DD card.
TRACEJAVA
Used to modify the Java trace string. The product tracing of registered trace components
conforms to the tracing rules as specified in the Sun™ Java™ Trace Specification.
Specifying *=all=enabled enables all types of tracing for all registered trace components.
HELP
Used to display a list of all the keywords that you can use with the modify command. You
can also use the HELP parameter after the CANCEL and DISPLAY parameters to display
lists of all the keywords you can use with either of these parameters.
PAUSELISTENERS
Used to prevent work from being accepted into the server. Use this command if you want
to shut down the communication listeners and purge any pending work in the work
registry.
RESUMELISTENERS
Used to restart the communication listeners after issuing a PAUSELISTENERS
parameter. This parameter also allows work to be accepted into the server.


DISPLAY | DISPLAY
Used to display the name of the server, the system name where the server is running, and
the current code level. You can specify the following options for this parameter:
– SERVERS displays the name of the server at which the command is directed, the
system name, and the code level for each active server in the Sysplex that is in the
same cell.
– SERVANTS displays a list of ASIDs of the servants that are attached to the server
against which you issued the display command.
– TRACE displays trace information for a server controller. You can further modify this
command with one of the following options:
• SRS displays trace information for all servants, one at a time.
• ALL displays trace information for the controller and all servants one at a time.
• JAVA displays the Java trace string settings for a server controller. You can further
modify this command with one of the following options:
- SRS displays Java trace information for all servants, one at a time.
- ALL displays Java trace information for the controller and all servants, one at a
time.
- HELP displays a list of all the keywords you can use with the modify display trace
Java command.
• HELP displays a list of all the keywords you can use with the modify display trace
command.
– JVMHEAP displays the JVM heap information for a server controller. You can further
modify this command with one of the following options:
• SRS displays the JVM heap information for all servants, one at a time.
• ALL displays the JVM heap information for the controller and all servants, one at a
time.
• HELP displays a list of all the keywords you may use with the modify display
Javaheap command.
– LISTENERS displays the connection instance name, associated IP address, and
listening port number. The associated IP address can display an asterisk (*) as a
wildcard.
– CONNECTIONS displays each connection instance name and a count of the number
of connections. Each connection instance is on a separate line. You can further modify
this command with one of the following options:
• NAME='name' displays the number of associated connections for the specified
connection instance 'name'. If the connection name is located but has zero
connections, the command returns a count of zero. If the connection name is not
found, the command returns an error message.
• LIST displays the remote host information for all of the connections of each of the
connection instances. If a connection instance name has no connections, the
command returns only the connection instance name.
• LIST, NAME='name' displays the remote host information for all connections of a
specified connection instance 'name'.
• HELP displays a list of all the keywords you can use with the modify command.

You cannot cancel a cluster from the MVS console. Instead, you must cancel each of the
servers that make up the cluster.


Example 1: The following command will cancel the bbo6acr server:
f bbo6acr,cancel

Example 2: The following command will cancel the bbo6acr server and instruct ARM to
restart it after it terminates:
f bbo6acr,cancel,armrestart

A.2 Additional resources for troubleshooting System Services
Runtime Environment configuration problems

A.2.1 First failure data capture
WebSphere Application Server V6: Diagnostic Data, IBM Redpaper:
http://www.redbooks.ibm.com/redpapers/pdfs/redp4085.pdf
Configuring first failure data capture log file purges, IBM information center:
websphere.express.doc/info/exp/ae/ttrb_ffdcconfig.html

A.2.2 Garbage collection tool
IBM Pattern Modeling and Analysis Tool for Java Garbage Collector:
http://www.alphaworks.ibm.com/tech/pmat

A.2.3 Debugging applications via RAD V7 (prior to deploying on z/OS)
Rational® Application Developer V7 Programming Guide, IBM Redbooks publication:
http://www.redbooks.ibm.com/abstracts/sg247501.html?Open

A.2.4 z/OS Debugging tools
Debug Tool for z/OS:
http://www-306.ibm.com/software/awdtools/debugtool/
IBM WebSphere Developer Debugger for System z, V7:
ftp://ftp.software.ibm.com/software/awdtools/debugtool/tools/wddz/wddz_v7_ds.pdf

A.2.5 Additional diagnostic references
WebSphere Application Server for z/OS eSupport home page to search for known
problems:
http://www.ibm.com/software/webservers/appserv/zos_os390/support/
WebSphere Application Server Information Centers:
http://www.ibm.com/software/webservers/appserv/was/library/


For v610:
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/topic/com.ibm.websphere.z
series.doc/info/welcome_nd.html
For the first time in the v610 information center, there is a specific “Troubleshooting and
support” section. When you first enter the information center, look at the list that is
displayed in the left pane. Here you will see the “Troubleshooting and support” section.
Expand that section to see all the individual sub-topics.

A.3 System Services Runtime Environment runtime logs
The references cited in this section provide information about the runtime logs.

A.3.1 How to view logs in TSO
https://websphere.austin.ibm.com/zos60/MvsViewLogs.html

A.3.2 How to create a data set from logs
https://websphere.austin.ibm.com/zos60/DataSetFromLogs.html

A.3.3 How to retrieve logs via FTP
https://websphere.austin.ibm.com/zos60/RetrieveLogs.html

A.4 System Services Runtime Environment application
deployment problems

A.4.1 Application not correctly signed
If the application is not correctly signed for System Services Runtime Environment, you will
see a message such as that shown in Figure A-8 on the ISC console.

Figure A-8 Error message

In addition, a message similar to the one in Figure A-9 will be present in the servant log.

Message: BBOO0220E: ADMA9006E: The application failed to install. The
/Ssreconf/AppServer/profiles/default/wstemp/1608525887/upload/test.ear
application is not licensed to be installed in this server under the IBM System
Services Runtime Environment for z/OS product.
Figure A-9 Log message


A.5 Java problems

A.5.1 Generating additional trace information
Additional detailed trace information can be obtained by enabling Java package level trace for
the failing application component (that is, com.ibm.zosconsole.*) by issuing the following
z/OS command:
F <SSRE>,tracejava='com.ibm.zosconsole.*=all'

This can also be set up using the admin console.

A.6 Problems during the Tivoli Key Lifecycle Manager post
SMP/E install

A.6.1 Locating Tivoli Key Lifecycle Manager log files
Log files are created in the /tklmProductInstall/logs directory every time one of the Tivoli Key
Lifecycle Manager post SMP/E scripts is executed. The name format of the log files is:
<scriptname>_mmddyy_HHMMSS.log

For example, installTKLM.sh might create a log file named installTKLM_012009_182345.log.
The log files contain detailed information about script execution and failures that will help to
troubleshoot a problem.

A.6.2 Unable to allocate memory
If the user has a small TSO/E region size they may experience the following error while
running the System Services Runtime Environment and/or Tivoli Key Lifecycle Manager
installation and configuration scripts:
Error: unable to allocate 17591296 bytes for GC in
j9vmem_reserve_memory.
JVMJ9VM015W Initialization error for library j9jit23(11): cannot
initialize JIT
Could not create the Java virtual machine.

This problem is caused by a small TSO/E region size. To resolve this problem the user should
increase their TSO/E region size. The maximum TSO/E region size of 2096128 KB can be
used. For more information on TSO/E region sizes see the TSO/E Information Center:

http://publib.boulder.ibm.com/infocenter/zos/v1r9/index.jsp?topic=/com.ibm.zos.r9.ikj/ikj.htm

Once the TSO/E region size is increased, the user will need to run the Tivoli Key Lifecycle
Manager uninstall script to clean up the environment before attempting another install. Note
that the uninstall attempts to clean up the entire environment so it will appear to fail on
components that were not installed. These messages can be safely ignored as long as the
uninstall indicates that it was successful overall.


A.6.3 Out of disk space
To check for available disk space, issue the df, disk free, command from a UNIX System
Services command prompt. To resolve this problem you will need to increase the size of the
filesystem.

A.6.4 Using wrong user ID to execute Tivoli Key Lifecycle Manager post
SMP/E scripts
In this case the user has tried to execute one of the Tivoli Key Lifecycle Manager post SMP/E
scripts using the wrong ID.

The user should log on as SSRECFG and rerun the script. To check which user you are
currently logged on as, you can enter one of the following commands:

Example A-2 id -un command
$ id -un
SSRECFG

Example A-3 whoami command
$ whoami
SSRECFG

If the response back is not SSRECFG, the user should switch to the SSRECFG user ID by
issuing the su, switch user command. For example, from a UNIX System Services command
prompt, issue:

su ssrecfg

A.6.5 Not having the correct permissions set up on the
TKLM_POST_SMPE_INSTALL_HOME directory and its contents
Use the chown command to recursively give the ssrecfg user ID and ssregrp group ID
ownership of the TKLM_POST_SMPE_INSTALL_HOME directory and all of its contents.
For example:
chown -R System ssrecfg:ssregrp /tklmProductInstall

Then use the chmod command to recursively give the ssrecfg user ID and ssregrp group ID
read, write, and execute permissions to the TKLM_POST_SMPE_INSTALL_HOME directory
and all of its contents. For example:
chmod -R 770 /tklmProductInstall

A.6.6 Not having correct permission and ownership values on the System
Services Runtime Environment config hfs container
To determine if this is the cause the user can list out the permissions and ownership values of
the <SSRE_HOME> directory using the ls -al UNIX System Services command:
ls -al /ssreconf/AppServer


The user ID owner of this directory should be SSREADM, the System Services Runtime
Environment started task ID, and the group owner should be SSREGRP. The SSREADM and
SSREGRP IDs should have read, write, and execute permissions, as shown in Example A-4.

Example A-4 list command
$ ls -al /ssreconf/AppServer
total 696
drwxrwx--- 39 SSREADM SSREGRP 8192 Mar 4 08:56 .

The SSREADM ID is the appropriate owner of the files within SSRE_HOME, and since
SSRECFG is part of the SSREGRP group it will have the appropriate permission to read,
write, and execute files needed for Tivoli Key Lifecycle Manager. If your SSRE_HOME does
not contain similar permission and ownership values to the example, your System Services
Runtime Environment config HFS may be corrupt, and you will need to either re-install
System Services Runtime Environment or contact System Services Runtime Environment/
WebSphere Application Server service to resolve the problem.

A.6.7 Tivoli Key Lifecycle Manager post SMP/E install script return codes
All failure return codes are defined in /tklmProductInstall/bin/common.sh. Here is a list of
return codes from the Tivoli Key Lifecycle Manager post SMP/E scripts followed by a detailed
explanation:
RC_SUCCESS=0
RC_UNINSTALL_FAILED=2
RC_CANNOT_CREATE_LOG_FILE=5
RC_LOG_DIR_IS_A_FILE=10
RC_LOG_DIR_DOESNT_EXIST=15;
RC_NO_RESPONSE_FILE=20
RC_CANNOT_CREATE_RESPONSE_FILE=25
RC_CANNOT_UPDATE_RESPONSE_FILE=30
RC_INVALID_INPUT=35
RC_INVALID_RESPONSE_FILE=40
RC_CANNOT_CREATE_SSRE_PROD_DIR=45
RC_CANNOT_CREATE_TKLM_PROD_DIR=50
RC_UI_SERVER_INSTALL_FAILED=55
RC_CANNOT_START_WAS_SERVER=60
RC_CANNOT_STOP_WAS_SERVER=65
RC_DBCONF_FAILURE=70
RC_COPY_FAILURE=75
RC_FAILED_PLUGIN_INIT=80
RC_MAY_BE_INVALID_RESPONSE_FILE=85
RC_ERROR_IN_MIGRATION_API=90
RC_ALREADY_INSTALLED=95
RC_INTERNAL_ERROR=99

RC_SUCCESS=0
The script execution was successful.

RC_UNINSTALL_FAILED=2
There was a failure when uninstalling one or more Tivoli Key Lifecycle Manager components.
This failure may surface while the script is removing the Tivoli Key Lifecycle Manager binaries
from within the System Services Runtime Environment container or if there was a failure
removing the Tivoli Key Lifecycle Manager datasource to DB2 from with System Services


Runtime Environment. Common reasons for the failure would be if the uninstall script is
executed by some user other then the SSRECFG user ID. For instructions on checking which
user ID you are logged on as, and logging on as the SSRECFG user ID, see the discussion at
the beginning of this section.

Another possibility for the failure may be if the script encounters an intermittent WebSphere
problem due to a timing issue. In this case simply rerunning the uninstall script should resolve
the problem.

If the script still fails after rerunning, check the log file for details and WebSphere exceptions.
If a WebSphere exception is found, the System Services Runtime Environment or
WebSphere service teams will need to troubleshoot the problem. Otherwise, check the error
messages to determine which of the remaining uninstalled components is causing the failure.

RC_CANNOT_CREATE_LOG_FILE=5
There was a problem creating a log file associated with the script the customer has attempted
to execute. A potential cause of this problem is that the file system is full. For instructions on
checking file system space, see the discussion at the beginning of this section.

Another potential cause of this failure is that the user has tried to execute one of the Tivoli Key
Lifecycle Manager post SMP/E scripts using the wrong ID. For instructions on checking which
user ID you are logged on as, and logging on as the SSRECFG user ID, see the discussion at
the beginning of this section.

Another potential cause of this failure is that the correct permission and ownership values
have not been set on the TKLM_POST_SMPE_INSTALL directory and its contents. For
instructions on setting up the right ownership and permission values for the
TKLM_POST_SMPE_INSTALL directory and files, see the discussion at the beginning of this
section.

RC_LOG_DIR_IS_A_FILE=10
The /tklmProductInstall/logs folder has been corrupted and appears as a file. In this case you
will need to delete the logs file by issuing the rm, remove command from a UNIX System
Services command shell:
rm logs

Then create a new logs directory and give user ID SSRECFG and group ID SSREGRP
ownership and read, write, and execute permission:
mkdir /tklmProductInstall/logs
chown -R SSRECFG:SSREGRP /tklmProductInstall/logs
chmod -R 770 /tklmProductInstall/logs

Then rerun the script.

RC_LOG_DIR_DOESNT_EXIST=15
The /tklmProductInstall/logs directory was not found in the filesystem. This failure could occur
if the logs directory was deleted, renamed, the wrong user ID is attempting to execute one of
the TKLM_POST_SMP/E_INSTALL scripts, or the permissions on the
TKLM_POST_SMP/E_INSTALL directory and files are incorrect. For instructions on setting
up the right ownership and permission values for the TKLM_POST_SMPE_INSTALL
directory and files, see the discussion at the beginning of this section.

For instructions on creating a new logs directory with the correct ownership and permissions,
see the steps under RC_LOG_DIR_IS_A_FILE=10. For instructions on checking which user


ID you are logged on as, and logging on as the SSRECFG user ID, see the discussion at the
beginning of this section.

RC_NO_RESPONSE_FILE=20
The response file could not be found. The most common causes of this problem are that the
user did not run the createResponseFile.sh shell script yet and is trying to install Tivoli Key
Lifecycle Manager. The createResponseFile.sh shell script must be run prior to the
installTKLM.sh shell script in order to set up the appropriate settings needed to deploy Tivoli
Key Lifecycle Manager within System Services Runtime Environment. To resolve this
problem run the createResponseFile.sh shell script prior to running the installTKLM.sh shell
script.

Another common reason for this is that Tivoli Key Lifecycle Manager is not installed on the
system and the user has attempted to run the uninstallTKLM.sh shell script to uninstall Tivoli
Key Lifecycle Manager. In this case the uninstallTKLM.sh shell script should not be executed
if Tivoli Key Lifecycle Manager is not yet installed.

Another potential reason would be if the user has passed an invalid -responseFile parameter.
The user should check that the response file they are passing in actually exists and that the
SSRECFG user ID has read, write and execute permissions.

If the default response file located in /tklmProductInstall/bin was deleted you will need to rerun
the createResponseFile.sh shell script to recreate it. If it does exist ensure that the
permissions and ownership values are set up correctly. For instructions on setting up the right
ownership and permission values for the TKLM_POST_SMPE_INSTALL directory and files,
see the discussion at the beginning of this section.

RC_CANNOT_CREATE_RESPONSE_FILE=25
There was a problem when trying to create the response file when executing one of the Tivoli
Key Lifecycle Manager post SMP/E scripts. Common causes for this failure would be if the
-responseFile parameter was used but it points to a path where the user has insufficient
permissions to create new files. If the -reponseFile parameter is not specified, the default
location of the response files is in the /tklmProductInstall/bin directory. If permissions are not
setup correctly for the bin directory this error is likely to surface. Another common cause is
when the user is not logged on as the SSRECFG user ID when they execute one of the Tivoli
Key Lifecycle Manager post SMP/E scripts. And finally this failure can occur when the file
system runs out of space. For instructions on how to correct permission problems, logon as
the SSRECFG user ID, and ensure you have enough disk space, see the discussion at the

RC_CANNOT_UPDATE_RESPONSE_FILE=30
The Tivoli Key Lifecycle Manager post SMP/E scripts tried to update the user's response file
but received a failure. Common causes for this failure would be permission and ownership
settings are incorrect, the user is not logged on with the correct user ID, or the file system is
full. For instructions on how to correct permission problems, logon as the SSRECFG user ID,
and ensure you have enough disk space, see the discussion at the beginning of this section.

RC_INVALID_INPUT=35
Invalid input was passed to the script, such as invalid command line arguments. To correct
this problem ensure you are invoking the script with the expected command line arguments
and execute the script again.

RC_INVALID_RESPONSE_FILE=40
The response file is not valid. This failure could surface if one or more of the parameters in
the response file is not valid. To resolve this problem the user should look for messages in the


log file for the createResponse.sh script that indicate an invalid parameter was specified. If it
is not clear which parameter is causing the problem, rerun the createResponseFile.sh shell
script to reset the values in your response file.

RC_CANNOT_CREATE_SSRE_PROD_DIR=45
There was a failure when creating the <SSRE_HOME>/products directory. During the install,
Tivoli Key Lifecycle Manager deploys itself within the System Services Runtime Environment
Config HFS container. Part of that process is to create this product directory to store some
files that are needed by the Tivoli Key Lifecycle Manager product. Common causes of this
failure are that the SSRECFG user ID has insufficient permissions to create the product
directory. For instructions on confirming that the System Services Runtime Environment
Config HFS container permissions are set up correctly, see the discussion at the beginning of
this section.

Another potential cause for this failure would be due to the file system being full or the wrong
user ID was used to run the installTKLM.sh script. For instructions on checking file system
space, checking which user ID you are logged on as, and logging on as the SSRECFG user
ID, see the discussion at the beginning of this section.

RC_CANNOT_CREATE_TKLM_PROD_DIR=50
There was a problem creating the <SSRE_HOME>/products/tklm directory. The reasons for
seeing this problem are the same as listed for RC_CANNOT_CREATE_SSRE_PROD_DIR=45,
the only differences being that this is yet another directory under the product directory. See that
section for information on how to resolve this problem.

RC_UI_SERVER_INSTALL_FAILED=55
There was a failure while trying to install the Tivoli Key Lifecycle Manager UI (tkml-ui.war) or
server (com.ibm.tklm.ear) components within System Services Runtime Environment.
Messages in the log will indicate which of the two has caused the failure.

Potential causes for the failure would be if the SSRECFG has insufficient permissions to the
<SSRE_HOME>/systemApps/isclite.ear directory or <SSRE_HOME>/installableApps
directory. These directories are located within the System Services Runtime Environment
configuration dataset and should be owned by user ID SSREADM, the System Services
Runtime Environment started task ID, and group ID SSREGRP. The permissions on both
directories should give read, write, and execute to both the SSREADM and SSREGRP IDs,
which will allow the SSRECFG, System Services Runtime Environment configuration ID,
access to install the Tivoli Key Lifecycle Manager UI and Server components within System
Services Runtime Environment. For instructions on confirming that the System Services
Runtime Environment Config HFS container permissions are set up correctly, see the
discussion at the beginning of this section.

Another potential cause of this failure would be if a WebSphere API call failed or
installTKLM.sh script encountered a WebSphere exception while trying to install the UI and
server components. In this case the user will need to inspect the Tivoli Key Lifecycle Manager
log file for WebSphere messages to further diagnose the problem. There is a possibility that
the problem is intermittent, in which case the user can try to run uninstall and then run install
again. However, if the same error continues to resurface then it is likely a WebSphere
problem, which would need to be pursued with System Services Runtime
Environment/WebSphere service.

RC_CANNOT_START_WAS_SERVER=60
System Services Runtime Environment failed to start. The most common cause for this
problem is that the SSRECFG user ID was not used to execute one of the Tivoli Key Lifecycle
Manager post SMP/E scripts. For instructions on checking which user ID you are logged on


as, and logging on as the SSRECFG user ID, see the discussion at the beginning of this
section.

Another reason for this failure would be if the SSRECFG user ID was used to execute one of
the Tivoli Key Lifecycle Manager post SMP/E scripts, but the permissions defined for the
<SSRE_HOME>/profiles/default/bin/startServer.sh script do not allow the SSREGRP group
execute permissions. Use the UNIX System Services list command to examine the
ownership and permission settings for the startServer.sh script. For instructions on confirming
that the System Services Runtime Environment Config HFS container permissions are set up
correctly, see the discussion at the beginning of this section.

RC_CANNOT_STOP_WAS_SERVER=65
System Services Runtime Environment failed to stop. The most common cause for this
problem is that the SSRECFG user ID was not used to execute one of the Tivoli Key Lifecycle
Manager post SMP/E scripts. For instructions on checking which user ID you are logged on
as, and logging on as the SSRECFG user ID, see the discussion at the beginning of this
section.

Another reason for this failure would be if the SSRECFG user ID was used to execute one of
the Tivoli Key Lifecycle Manager post SMP/E scripts, but the permissions defined for the
<SSRE_HOME>/profiles/default/bin/stopServer.sh script do not allow the SSREGRP group
execute permissions. Use the UNIX System Services list command to examine the
ownership and permission settings for the stopServer.sh script. For instructions on confirming
that the System Services Runtime Environment Config HFS container permissions are set up
correctly, see the discussion at the beginning of this section.

RC_DBCONF_FAILURE=70
One or more failures occurred while trying to configure the Tivoli Key Lifecycle Manager
database components in System Services Runtime Environment. This problem will surface if
the DB2 JDBC driver is not installed or if the user did not specify the correct path for the DB2
JDBC driver when running the POST_SMPE_TKLM_HOME/bin/createResponseFile.sh shell
script. In this case the following error message may be found in the output log:
--> TKLM_DB_JDBC_DRIVER_PATH not found, creating and setting to
/usr/lpp/db2910/db2910_jdbc/classes *****

The user should ensure that the DB2 JDBC driver is installed and rerun the
POST_SMPE_TKLM_HOME/bin/createResponseFile.sh script to ensure the path is correct.

This problem may also surface if a WebSphere API call failed or one of the Tivoli Key
Lifecycle Manager post SMP/E scripts encountered a WebSphere exception. This problem
could potentially be an intermittent timing issue that can easily be resolved by running the
/tklmProductInstall/bin/uninstallTKLM.sh and then the /tklmProductInstall/bin/installTKLM.sh
again. If the same error continues to surface, the user will need to inspect the WebSphere
messages within the System Services Runtime Environment job logs and pursue the problem
with the System Services Runtime Environment/WebSphere service teams.

RC_COPY_FAILURE=75
There was a problem copying files or folders from within the /tklmProductInstall directory to
the System Services Runtime Environment config HFS.

When the Tivoli Key Lifecycle Manager post SMP/E scripts attempt to deploy Tivoli Key
Lifecycle Manager within the System Services Runtime Environment config HFS container, a
number of binaries, properties files, and configuration files are copied over to the System
Services Runtime Environment config HFS. If the SSRECFG user ID is not used to execute
the Tivoli Key Lifecycle Manager post SMP/E scripts, or the permissions on the files within the


/tklmProductInstall directory are incorrect, this failure is likely to surface. For instructions on
checking which user ID you are logged on as, and logging on as the SSRECFG user ID; and
for instructions on setting up the right ownership and permission values for the
TKLM_POST_SMPE_INSTALL directory and files, see the discussion at the beginning of this
section.

RC_FAILED_PLUGIN_INIT=80
There was a failure to initialize the Tivoli Key Lifecycle Manager plugins that need to be
deployed within System Services Runtime Environment.

When Tivoli Key Lifecycle Manager deploys itself into the System Services Runtime
Environment, it will copy binaries into the System Services Runtime Environment config HFS
that need to be initialized. The initialization is done using the
<SSRE_HOME>/plugins/osgiCfgInit.sh script. This return code will surface if execution of this
script fails.

Potential reason for this failure would be if the wrong user ID was used to execute the Tivoli
Key Lifecycle Manager post SMP/E install script, or if the System Services Runtime
Environment config HFS does not have the correct ownership and permission settings. For
instructions on checking which user ID you are logged on as, and logging on as the
SSRECFG user ID; and for instructions on confirming that the System Services Runtime
Environment Config HFS container permissions are set up correctly, see the discussion at the

RC_MAY_BE_INVALID_RESPONSE_FILE=85
There is a potential problem with the response file being passed to the Tivoli Key Lifecycle
Manager post SMP/E scripts. See RC_INVALID_RESPONSE_FILE=40 for more information.

RC_ERROR_IN_MIGRATION_API=90
The user has attempted an IBM Encryption Key Manager to Tivoli Key Lifecycle Manager
migration and a failure has occurred with the migration steps.

Potential causes for this failure are, the user is attempting to migrate a file-based keystore
which they do not have read, write, or execute permission to. To resolve this problem give the
SSREGRP, System Services Runtime Environment group ID, read, write, and execute access
to the keystore using the UNIX System Services chown and chmod commands. For example:
chown SSRECFG:SSREGRP Your_Keystore_Path_and_Name_Here
chmod 770 Your_Keystore_Path_and_Name_Here

The SSREGRP group ID requires read, write, and execute permissions so that the
SSRECFG user ID can perform the migration, and the SSREADM user ID will be able to load
the keystore when System Services Runtime Environment is restarted.

If using a Hardware keystore, JCECCAKS or JCECCARACFKS, ensure that ICSF is active. If
ICSF is not active when trying to migrate a hardware keystore this error message will be
displayed. In addition to this error message you may find a similar error trace record in the
output log file, as shown in Figure A-10.

CTGKS0117E: Migration failed. Hardware error from call CSNBOWH returnCode
12reasonCode 0
CTGKS0117E: Migration failed.
Figure A-10 output log error messages

To resolve this problem start ICSF and run the migrateEKM.sh.


Another potential cause of this failure is if the user is attempting to migrate a RACF keyring
that does not exist. After confirming that the RACF keyring does exist, make sure that the IBM
Encryption Key Manager config.keystore.file configuration setting is correctly pointing to your
RACF keyring. For example, it should be set using this format:
config.keystore.file = safkeyring://UserID/EKMkeyringName

If the problem continues to persist, ensure that you have given the SSRECFG, SSREADM,
and SSREGRP IDs the appropriate access to use the RACF keyring. For help with setting up
the appropriate access to your RACF keyring see the sample Rexx script located at
/tklmProductInstall/samples/racfpermissions.rexx.

Ensure that the IBM Encryption Key Manager XML files to be migrated are in EBCDIC. If they
are not you will receive this failure and additionally may see the error message in the output
log as shown in Figure A-11.

com.ibm.tklm.common.exception.KLMException: org.xml.sax.SAXParseException:
Content is not allowed in prolog.:
org.xml.sax.SAXParseException: Content is not allowed in prolog.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at javax.xml.parsers.DocumentBuilder.parse(Unknown Source)
Figure A-11 Output log error message

To resolve this problem convert the IBM Encryption Key Manager XML files to EBCDIC and
attempt to open them with a Web browser.

Run the bin/migrateEKM.sh again once the problem is resolved.

RC_ALREADY_INSTALLED=95
The user has attempted to install Tivoli Key Lifecycle Manager; however, one or more of the
Tivoli Key Lifecycle Manager components are already installed and deployed within the
System Services Runtime Environment container. To correct this failure, run the
bin/uninstallTKLM.sh script to successfully uninstall all of the Tivoli Key Lifecycle Manager
components from System Services Runtime Environment. Once the uninstall runs
successfully the user will be able to run the install script again.

Another potential cause of this failure would be if you have successfully uninstalled Tivoli Key
Lifecycle Manager, but a stale version of the /tklmProductInstall/bin/.installedComponents file
is hanging around. This file is used by the Tivoli Key Lifecycle Manager post SMP/E scripts to
determine the state of the Tivoli Key Lifecycle Manager install. To resolve this problem,
examine the .installedComponents file to determine which components are still listed as
being installed within System Services Runtime Environment. Once you have confirmed that
all components are definitely uninstalled, you can rename or delete this file and rerun the
installTKLM.sh script.

RC_INTERNAL_ERROR=99
The Tivoli Key Lifecycle Manager post SMP/E scripts have experienced an unexpected
internal error. In this case the log file will need to be collected and sent to Tivoli Key Lifecycle
Manager Service to resolve the problem.


A.7 General errors resulting from the Tivoli Key Lifecycle
Manager post SMP/E Install

A.7.1 *** SSL SIGNER EXCHANGE PROMPT *** SSL signer from target host
null is not found in trust store safkeyring:///WASKeyring.System
Services Runtime Environment
In this case the Tivoli Key Lifecycle Manager install script will hang because WebSphere has
issued a prompt and is waiting for a response. This problem stems from either using the
wrong user ID to execute the installTKLM.sh script, someone other then SSRECFG, or the
WebSphere Application Server keyring for user SSRECFG was not set up correctly during
the System Services Runtime Environment configuration. To correct the problem ensure you
are logged on as SSRECFG and execute the installTKLM.sh script again. If the problem
persists there is likely a problem with the System Services Runtime Environment
configuration or the WebSphere Application Server keyring that was created for the
SSRECFG user ID. This will need to be pursued with System Services Runtime
Environment/WebSphere Application Server service.

A.7.2 FSUM7343 cannot open "/SYSTEM/tklmProductInstall/logs/.output" for
output: EDC5111I
This problem is a result of a permission error when the Tivoli Key Lifecycle Manager post
SMP/E scripts attempt to write to the logs/.output and logs/.installedComponents files. This
problem could surface if you have previously run the Tivoli Key Lifecycle Manager post SMP/E
scripts under a different user ID or you have changed the UID of the SSRECFG user ID. Use
the UNIX System Services change owner command, chown, to make the SSRECFG user ID
the owner of the logs/.output and logs/.installedComponents files and run the script again.

A.7.3 Attempting to run the bin/migrateEKM.sh, bin/installTKLM.sh or
bin/uninstallTKLM.sh script while System Services Runtime
Environment is already and running
You will see the message shown in Figure A-12 in the output log file.

ADMU3028I: Conflict detected on port 32200. Likely causes: a) An instance of
the server server1 is already running b) some other process is
using port 32200
ADMU3027E: An instance of the server may already be running: server1
ADMU0111E: Program exiting with error:
com.ibm.websphere.management.exception.AdminException: ADMU3027E: An
instance of the server may already be running: server1
ADMU1211I: To obtain a full trace of the failure, use the -trace option.
ADMU0211I: Error details may be seen in the file:

/etc/Ssreconf/AppServer/profiles/default/logs/server1/startServer.log
Figure A-12 Output log error message

This message can be ignored because it only indicates that an instance of System Services
Runtime Environment is already up and running.


A.7.4 Using an unauthorized user to run the Tivoli Key Lifecycle Manager post
SMP/E install scripts
If you use an unauthorized user to execute the Tivoli Key Lifecycle Manager post SMP/E
install scripts you will likely experience the following error message either on your UNIX
System Services login session or within the Tivoli Key Lifecycle Manager log files:
EDC5111I Permission denied.

To correct this situation, ensure you are logged on as the SSRECFG user ID before
executing the Tivoli Key Lifecycle Manager post SMP/E install scripts. To determine which ID
you are currently logged on as, you can issue one of the following commands:

Example A-5 id -un command
$ id -un
SSRECFG

Example A-6 whoami command
$ whoami
SSRECFG

Also ensure that the Tivoli Key Lifecycle Manager post SMP/E install scripts and all files
within your Tivoli Key Lifecycle Manager product install directory are owned by user ID
SSRECFG and group ID SSREGRP, and that the user and group owners have read, write,
and execute permissions. If either the permissions or ownership are incorrect you can correct
this problem by issuing the following sets of commands:
chown -R SSRECFG:SSREGRP /tklmProductInstall
chmod -R 770 /tklmProductInstall

The SSRECFG user ID is created by the RACFMSTR script, which is executed by the
modifySSRE.sh script during the System Services Runtime Environment configuration. If you
continue to have permission problems, ensure that the SSRECFG was correctly created as a
member of the SSREGRP group by issuing the following command from an ISPF command
shell:
lu SSRECFG

If you continue to have permission problems, ensure that the System Services Runtime
Environment configuration dataset has the appropriate permissions set up. List your
SSRE_HOME dir to ensure that SSREADM, the System Services Runtime Environment
started task ID, and SSREGRP have user and group ownership, and both have read, write,
and execute permissions. For example:
ls -al /ssreconf/AppServer
drwxrwx--- 39 SSREADM SSREGRP 8192 Mar 4 08:56 .

If the permissions are not set up correctly it is recommended to reinstall System Services
Runtime Environment because there are a large number of files and sym links within the
System Services Runtime Environment configuration dataset, so it is not advisable to perform
recursive chmod and chowns on the System Services Runtime Environment configuration
dataset.


A.7.5 Tivoli Key Lifecycle Manager product files are not synchronized with
Tivoli Key Lifecycle Manager database in DB2
If the Tivoli Key Lifecycle Manager product files that live in your TKLM_HOME directory are
not synchronized with the Tivoli Key Lifecycle Manager database in DB2, you will receive the
error message shown in Figure A-13, which indicates that Tivoli Key Lifecycle Manager had a
problem loading your master keystore.

CTGKM0103E
CTGKM0103E Unable to get keystore
information.com.ibm.tklm.common.exception.KLMException: Given final block not
properly padded: javax.crypto.BadPaddingException: Given final block not
properly padded at com.ibm.crypto.provider.AESCipher.engineDoFinal

Typically there are 3 ways you can get into this situation:
1. During a Tivoli Key Lifecycle Manager reinstall that will be synchronized up with a certain
backup level. In this case you have uninstalled and then reinstalled the Tivoli Key Lifecycle
Manager product within System Services Runtime Environment and then went directly to
the Tivoli Key Lifecycle Manager welcome page before running the restore function and
recycling System Services Runtime Environment. After an uninstall your Tivoli Key
Lifecycle Manager product files will be deleted. If you have successfully installed and
configured Tivoli Key Lifecycle Manager you should take a backup so that in the future you
will be able to successfully restore the Tivoli Key Lifecycle Manager product files that
contain your configuration settings and potentially your file-based key material. If you need
to uninstall and reinstall Tivoli Key Lifecycle Manager, you will need this backup file to
restore your Tivoli Key Lifecycle Manager back to the configuration you have set up. In this
case, once the restore is performed the database and the Tivoli Key Lifecycle Manager
configuration data within the products directory will be synchronized and the error
message will go away.
2. During a Tivoli Key Lifecycle Manager reinstall to a fresh level of Tivoli Key Lifecycle
Manager and the Tivoli Key Lifecycle Manager database. In this case you have
successfully installed and configured Tivoli Key Lifecycle Manager but no longer want to
keep your current configuration and are sure that nothing needs to be saved for future use,
then you may be reinstalling Tivoli Key Lifecycle Manager in order to completely through
away your current instance of Tivoli Key Lifecycle Manager. If this is the case, and you are
sure your Tivoli Key Lifecycle Manager database contains nothing useful that will be
needed in the future, then you should remember to run the sample SPUFI file provided in
the tklm.tar file to drop the Tivoli Key Lifecycle Manager tablespaces and database. If a
Tivoli Key Lifecycle Manager reinstall is performed and the old Tivoli Key Lifecycle
Manager database is left in DB2, this error will appear because Tivoli Key Lifecycle
Manager will be missing the configuration files in the product directory that was used when
creating the Tivoli Key Lifecycle Manager database. Once the drop Tivoli Key Lifecycle
Manager database sample SPUFI is executed, the create Tivoli Key Lifecycle Manager
database should then be executed in order to make a new fresh instance of the Tivoli Key
Lifecycle Manager database. At this point you will have a fresh instance of Tivoli Key
Lifecycle Manager and the product files that are synchronized with a fresh instance of the
Tivoli Key Lifecycle Manager database, and the error message will go away.
3. During a Sysplex installation. In this case you have successfully installed and configured
Tivoli Key Lifecycle Manager on one LPAR and are in the process of propagating your
backup files to the other LPARs and running the restore function to synchronize each
LPAR with your original Tivoli Key Lifecycle Manager. If all the secondary Tivoli Key


Lifecycle Managers are configured to use DB2 datasharing with the original Tivoli Key
Lifecycle Manager they will be able to see the Tivoli Key Lifecycle Manager database
when they first start up. If you navigate to the welcome page of any of your secondary
Tivoli Key Lifecycle Managers before you perform the restore function, the error message
will appear indicating that the Tivoli Key Lifecycle Manager configuration data within the
Tivoli Key Lifecycle Manager product directory and the Tivoli Key Lifecycle Manager
database are not synchronized. To correct this you should simply go to the Backup page
and restore the backup taken from your original Tivoli Key Lifecycle Manager. After
recycling System Services Runtime Environment, the error message will go away
because your Tivoli Key Lifecycle Manager configuration data in the product directory and
the Tivoli Key Lifecycle Manager database will be synchronized.

A.7.6 Trying to use a hardware keystore but the IBMJCECCA provider not
specified in the java.security file within System Services Runtime
Environment's embedded Java
If you are attempting to configure Tivoli Key Lifecycle Manager with a hardware-based
keystore, IBMJCECCAKS or JCECCARACFKS, you need to enable the JCECCA provider
within the SSRE_APPSERVER_HOME/java/lib/security/java.security file. This file is the
master security properties file of the embedded Java for System Services Runtime
Environment. If the IBMJCECCA provider is not enabled and you attempt to configure Tivoli
Key Lifecycle Manager with a hardware-based keystore, the following error message will
appear on the Tivoli Key Lifecycle Manager panels:
CTGKM0551E Cannot find keystore provider for keystore type JCECCAKS

To correct this problem you must enable the IBMJCECCA provider in the
SSRE_APPSERVER_HOME/java/lib/security/java.security file by following this procedure.
Open the file, and uncomment the following line:

To uncomment this line you must remove the # sign at the beginning. Next, copy this line
either above or below the following line:

If you place the IBMJCECCA hardware provider above the IBMJCE software provider, all
crypto work will be routed to hardware crypto first, including SSL work performed by System
Services Runtime Environment. If you place the IBMJCECCA hardware provider below the
IBMJCE software provider, all crypto work performed by Tivoli Key Lifecycle Manager will be
routed to the hardware provider, but all SSL work performed by System Services Runtime
Environment will be routed to the software provider.

After copying the IBMJCECCA provider line either above or below the IBMJCE line, you then
need to re-order the security.provider.#s to ensure they are in correct sequential order. For
example, if you have selected to put the IBMJCECCA provider above the IBMJCE provider,
then you should end up with the file shown in Example A-7.

Example A-7 Modified java.security file with IBMJCECCA above IBMJCE
#
#



If you have selected to place the IBMJCECCA provider after the IBMJCE provider then you
should end up with the file shown in Example A-8.

Example A-8 Modified java.security file with IBMJCECCA below IBMJCE
#
#

Once the updates are made, save the file and recycle System Services Runtime Environment
in order for the updates to take effect.

A.7.7 Forgot to install the Java unrestricted policy files
If you skip the step for installing the Java unrestricted policy files you will get the error shown
in Figure A-14 when trying to configure certificates for your devices.

R_datalib (IRRSDL00) error: One or more updates could not be completed. Bad
encoding of private key or unsupported algorithm or key size invalid. Function
code: (8) Alias: (ITSO.TKLM.TestCert.March09) Userid: (SSRECFG) Return Codes:
(8, 8, 44)

A.7.8 Attempting to create a file-based keystore in a path that does not exist
If you attempt to create a file-based keystore in a location within UNIX System Services that
does not exist you will get back the error message shown in Figure A-15.

CTGKM0104E Unable to add keystore.CTGKM0504E Error occurred while creating the
keystore: /etc/test/tklm.jceccaks (EDC5129I No such file or directory.
(errno2=0x05620062))


To correct the problem you should specify a valid path within the file system to create your
file-based keystore.

A.7.9 Attempting to create a file-based keystore in a read only directory
If you attempt to create a file-based keystore in a location that is read only within UNIX
System Services you will get back the error message shown in Figure A-16.

keystore: /usr/lpp/jceccaks.keystore (EDC5141I Read-only file system.
(errno2=0x05620060))

To correct this problem you should choose a location within the file system that is not read
only.

A.7.10 Attempting to create a file-based keystore in a directory that the
SSREGRP group does not have authority to write to
If you attempt to create a file based keystore in a location to which the SSREGRP group does
not have write authority, you will get back the error message shown in Figure A-17.

keystore: /etc/jceccaks.keystore (EDC5111I Permission denied.
(errno2=0x5B450002))

To correct this problem, select a location within the file system to which the SSREGRP group
has write access.

A.8 Problems configuring Tivoli Key Lifecycle Manager

A.8.1 Kicked out of ISC console and Tivoli Key Lifecycle Manager panels
because the "Session has become invalid"
IBM System Services Runtime Environment uses Session Cookies to track which users are
logged in from a specific browser. If someone has logged into the ISC console from another
browser using the SSRECFG user ID, your session will become invalid and you will be logged
out of the ISC console.

In addition, concurrent updates to Tivoli Key Lifecycle Manager from multiple browsers is not
supported. Only one user can be logged onto the Tivoli Key Lifecycle Manager panels at a
time to make configuration updates.


Another potential cause of this problem would be if your logon session has timed out. The
default Session timeout value is 30 minutes. Access the panel in which session timeout
values can be configured within the ISC console by selecting:

Servers → Application Servers → server_name → Container Services → Session
management → Session timeout

A.8.2 Tivoli Key Lifecycle Manager panel pops up in a second browser
window
This problem occurs when Tivoli Key Lifecycle Manager first configures a master keystore
through the GUI and the user clicks one of the menu items in the left pane. This is an ISC
problem that is currently under investigation. In the meantime to get around the problem
simply close the pop-up window, log off of the ISC console, and log back in.

A.8.3 DB2 is not active: CODE=-4499, SQLSTATE=08001DSRA0010E: SQL
State = 08001, Error Code = -4,499
If System Services Runtime Environment and Tivoli Key Lifecycle Manager start up before
DB2 is started, or DB2 is not started, this error message may appear in the System Services
Runtime Environment Servant Job (<SSRE_PROC_PREFIX>S).

To resolve this problem start DB2.

A.8.4 CTGKM0597E - Error occurred while generating the secret key
This error indicates that Tivoli Key Lifecycle Manager was unable to generate a key or key
group. When this problem surfaces, the exception shown in Figure A-18 will be found within
your System Services Runtime Environment Servant Job, (<SSRE_PROC_PREFIX>S) and
Tivoli Key Lifecycle Manager debug log.

java.lang.RuntimeException: Hardware error from call CSNBKGN returnCode 8
reasonCode 2093 at
com.ibm.crypto.hdwrCCA.provider.AESKeyGenerator.a(AESKeyGenerator.java:93)

This error indicates that the user is attempting to generate and store AES data keys within
ICSF; however, they may be running an older version than ICSF HCR7751. Versions of ICSF
before HCR7751 do not support generating AES data keys and storing them in ICSF. To
resolve this problem check the z/OS compatibility flag on the Tivoli Key Lifecycle Manager
Configuration GUI panel. This flag will configure Tivoli Key Lifecycle Manager to generate
DES EDE data keys instead of AES keys for use with older version of ICSF.

A.8.5 WebSphere transaction timed out: BBOO0222I: WTRN0006W
Tivoli Key Lifecycle Manager comes with a default transaction timeout value of 600 seconds,
or 10 minutes. If the system is experiencing a very high workload and System Services
Runtime Environment has low priority, Tivoli Key Lifecycle Manager transactions could
potentially take more then 10 minutes and end up timing out. In the case of a timeout, Tivoli
Key Lifecycle Manager will mark all transactions as rollback in order to prevent corruption of
the Tivoli Key Lifecycle Manager database in DB2. An example of the error that will be


displayed in the System Services Runtime Environment Servant Job,
(<SSRE_PROC_PREFIX>S) when a Tivoli Key Lifecycle Manager transaction times out is
shown in Figure A-19.

Trace: 2008/11/17 15:30:24.908 01 t=7BF640 c=UNK key=P8 (13007002) ThreadId:
0000000d FunctionName: com.ibm.ws.Transaction.JTA.TimeoutManager SourceId:
com.ibm.ws.Transaction.JTA.TimeoutManager Category: INFO ExtendedMessage:
BBOO0222I: WTRN0006W: Transaction
1106F385E3957BCF0000000100002002C1F9CF50F50C97B200000100000000010939018F1106F38
5E3957BCF0000000100002002C1F9CF50F50C97B200000100000000010939018F00000001 has
timed out after 600 seconds.
Trace: 2008/11/17 15:32:24.595 01 t=7BD438 c=UNK key=P8 (13007002) ThreadId:
0000001f FunctionName: loadEKMKeyStores SourceId: keymanager Category: ALL
ExtendedMessage: javax.ejb.TransactionRolledbackLocalException: ; nested
exception is: com.ibm.websphere.csi.CSITransactionRolledbackException:
Transaction marked rollbackonly
Figure A-19 Transaction timeout message

To resolve this problem the user can reduce the workload on the system, increase the priority
of the System Services Runtime Environment tasks, or increase the transaction timeout value
to something longer than 10 minutes.

A.8.6 Problems starting System Services Runtime Environment: BBOO0222I:
J2CA0090I when starting System Services Runtime Environment
This problem surfaces when the Tivoli Key Lifecycle Manager datasource has been deleted
from System Services Runtime Environment but there are transactions which System
Services Runtime Environment is trying to recover.

In this case the WebSphere message shown in Figure A-20 will be displayed in the System
Services Runtime Environment Servant Job, (<SSRE_PROC_PREFIX>S).

BBOO0222I: J2CA0090I: This is an English only message: Component-managed
authentication alias tklm_db for connection factory or datasource jdbc/tklmXADS
is invalid. It may be necessary to re-start the server for previous
configuration changes to take effect..
ExtendedMessage: BBOO0220E: J2CA0044E: The ConnectionManager failed to get a
Subject from the security service associated with connection factory
jdbc/tklmXADS. Received exception javax.security.auth.login.LoginException:
Incorrect authDataEntry and alias is: tklm_db

Tivoli Key Lifecycle Manager can get into this situation because RRS still remembers the
datasource and is trying to recover transactions; however, the datasource does not exist in
System Services Runtime Environment anymore because the user has removed it.

To resolve the problem the user will need to delete the URs that were registered for System
Services Runtime Environment in RRS.


From the RRS panel, delete any URs associated with System Services Runtime
Environment. For example:
BBO.System Services Runtime Environment.System Services Runtime
Environment.System Services Runtime Environment.IBM Reset DCEIMGWQ CFCIMGWQ
BBO.System Services Runtime Environment.System Services Runtime
Environment.System Services Runtime EnvironmentD.IBM Reset DCEIMGWQ CFCIMGWQ

After a restart of System Services Runtime Environment the problem should be resolved.

A.8.7 Lexical error when running Tivoli Key Lifecycle Manager CLI commands
from OMVS
After bringing up a WSAdmin command prompt to use the Tivoli Key Lifecycle Manager CLI,
you might receive back Lexical errors when running Tivoli Key Lifecycle Manager CLI
commands from OMVS. For example, the following error message may be displayed after
running a Tivoli Key Lifecycle Manager CLI command from OMVS:
SyntaxError: Lexical error at line 1, column 28. Encountered: "u00dd" (221),
after : ""

This problem may have surfaced because OMVS has incorrectly interpreted the brackets, [
and ], used in your Tivoli Key Lifecycle Manager CLI command.

To resolve this problem start OMVS from the TSO Command Line with the CONVERT option
like this:
OMVS CONVERT((BPXFX111))

A.8.8 IRR.RAUDITX Access Errors due to RACF setup for Tivoli Key Lifecycle
Manager auditing not being performed
Tivoli Key Lifecycle Manager for z/OS leverages the R_AuditX RACF service to create SMF
type 83 subtype 6 audit records. If the Tivoli Key Lifecycle Manager started task ID,
SSREADM, is not given access to the R_AuditX service, the error shown in Figure A-21 will
surface in the System Services Runtime Environment Servant Job log,
(<SSRE_PROC_PREFIX>S).

10.18.19 STC00371 ICH408I USER(SSREADM ) GROUP(SSREGRP ) NAME(System Services
Runtime Environment ADMINISTRATOR ) IRR.RAUDITX CL(FACILITY) INSUFFICIENT
ACCESS AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE ) 10.18.20 STC00371
ICH408I

To resolve this problem, give auditing permissions to the user ID that the System Services
Runtime Environment started task is associated with. By default this is the SSREADM ID. For
example:
PERMIT IRR.RAUDITX CL(FACILITY) ID(SSREADM) ACCESS(READ)
SETR RACLIST(FACILITY) REFRESH

Then update SMF to collect records for Tivoli Key Lifecycle Manager SMF type 83 sub-type 6.
See the System Management Facilities publications for additional details on setting up SMF.


You can format Tivoli Key Lifecycle Manager audit data using the RACF SMF data unload
utility. For information about how to run the RACF SMF Data unload utility, refer to the z/OS
Security Server RACF Auditor's Guide. For utility support of the Tivoli Key Lifecycle Manager
SMF type 83 sub-type 6 audit records in z/OS Release 9 and 10, you must apply service for
APAR OA26653.

For more information on the Tivoli Key Lifecycle Manager SMF Record format, refer to the
Tivoli Key Lifecycle Manager information center at:
http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/topic/com.ibm.tklm.doc/w
elcome.htm

A.8.9 Unable to authenticate to Tivoli Key Lifecycle Manager MBeans:
BBOO0222I: SECJ0305I in the servant job log
In order for the logged on user ID’s credentials to be passed to the Tivoli Key Lifecycle
Manager MBeans, the user must configure System Services Runtime Environment to pass
user credentials for unauthenticated URIs. If this step is missed during the Tivoli Key
Lifecycle Manager installation the error shown in Figure A-22 will surface in the System
Services Runtime Environment Servant Job log, (<SSRE_PROC_PREFIX>S).

BBOO0222I: SECJ0305I: The role-based authorization check failed for admin-authz
operation
KeyStoreServiceMBean:getKeyStores. The user UNAUTHENTICATED (unique ID:
UNAUTHENTICATED) was not granted any of the following required roles:
adminsecuritymanager, operator, deployer, administrator, monitor, configurator.

To correct this problem, the user should configure System Services Runtime Environment to
use available authentication data when an unprotected URI is accessed. Perform the
following steps to do this:
1. Open the System Services Runtime Environment Integrated Solutions Console in a Web
browser and log in with the SSRECFG user ID. The default location for the System
Services Runtime Environment Integrated Solutions Console is:
https://yourhostname:32211/ibm/console/
2. Select Security from the left menu bar.
3. Select Secure administration, applications, and infrastructure from the submenu.
4. Select Web Security on the right side of the screen under Authentication.
5. Select the General Settings submenu under Web Security.
6. Click the checkbox for Use Available authentication data when an unprotected URI is
accessed.
7. Click OK.
8. On the next screen, select Save directly to the master configuration.
9. Select Logout at the top of the Integrated Solutions Console.
10.Recycle System Services Runtime Environment in order for the updates to take affect.

For more information on this setting, refer to the WebSphere Application Server, Version 6.1
Information Center located at:
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp.


A.8.10 DB2's WLM Environment has stopped: SQLCODE: -471,
SQLSTATE: 55023
If DB2's WLM environment has stopped you will receive the following error when Tivoli Key
Lifecycle Manager tries to connect to DB2:
SQLCODE: -471, SQLSTATE: 55023

This error will likely surface in your System Services Runtime Environment Servant job log,
<SSRE_PROC_PREFIX>S. The full error will look something like that shown in Figure A-23.

java.lang.reflect.InvocationTargetException ..... followed by:
com.ibm.tklm.common.exception.KLMException: com.ibm.db2.jcc.c.SqlException: DB2
SQL error: SQLCODE: -471, SQLSTATE: 55023, SQLERRMC: SYSIBM.SQLTABLES;00E7900C:
com.ibm.db2.jcc.c.SqlException: DB2 SQL error: SQLCODE: -471, SQLSTATE: 55023,
SQLERRMC: SYSIBM.SQLTABLES;00E7900C
at com.ibm.db2.jcc.c.mg.d(mg.java:1338)
at com.ibm.db2.jcc.b.db.k(db.java:351)
at com.ibm.db2.jcc.b.db.e(db.java:96)
at com.ibm.db2.jcc.b.t.e(t.java:83)

To resolve this problem, check the status of the application environment and resume it if it
has stopped. The following command can be used to check the application environment:
D WLM,APPLENV=*

The following example shows how to resume the application environment for DB9ENV5:
V WLM,APPLENV=DB9ENV5,RESUME

A.8.11 Unable to import certificates into RACF using the Tivoli Key Lifecycle
Manager import function
If you have configured Tivoli Key Lifecycle Manager with a RACF-based master keystore, and
your are not using hardware protection, the following key size limitations exists:
z/OS 1.9 - RACF database can only import private keys with a maximum size of 1024 bits.
z/OS 1.10 - RACF database can only import private keys with a maximum size of
4096 bits.

If you attempt to import a certificate into RACF that is larger then the maximum size indicated,
the error message in Figure A-24 will surface on the Tivoli Key Lifecycle Manager panels.

R_datalib (IRRSDL00) error: One or more updates could not be completed. Bad
encoding of private key or unsupported algorithm or key size invalid. Function
code: (8) Alias: (ITSO.TKLM.March2009) Userid: (SSRECFG) Return Codes: (8, 8,
44)

Note: Tivoli Key Lifecycle Manager V1 only supports certificates up to 2048 bits. Anything
larger will not work with Tivoli Key Lifecycle Manager.


A.8.12 Tivoli Key Lifecycle Manager has a known problem with SSL
certificates using mixed case alias names
In your Tivoli Key Lifecycle Manager audit log, either SMF dataset or audit log in the file
system if you have configured file-based auditing, you will find this error message:
CTGKS0011E SSL Listener went down.:No available certificate corresponds to the
SSL cipher suites which are enabled.

If debug is turned on you may find the error in Figure A-25 listed within the Tivoli Key Lifecycle
Manager trace records.

ExtendedMessage: javax.net.ssl.SSLException: No available certificate
corresponds to the SSL cipher suites which are enabled.
at com.ibm.jsse2.hc.a(hc.java:76)
at com.ibm.jsse2.hc.accept(hc.java:22)
at
com.ibm.tklm.keyserver.ekm.transport.ssl.SSLListener.run(SSLListener.java:256)
at
com.ibm.tklm.keyserver.ekm.transport.TransportListener.run(TransportListener.ja
va:202)
at java.lang.Thread.run(Thread.java:810)

The work-around for this problem is to fold your SSL certificate alias name to lower case in
the Tivoli Key Lifecycle Manager configuration file and recycle System Services Runtime
Environment.

For example, if your SSL certificate alias name is set to:
config.keystore.ssl.certalias=MixedCase.ssl.cert

change the alias name to lower case:
config.keystore.ssl.certalias=mixedcase.ssl.cert

and recycle System Services Runtime Environment.

A.8.13 Tivoli Key Lifecycle Manager panel pops up and creates 2nd active
windows for the Tivoli Key Lifecycle Manager GUI
There is a known problem with the ISC level within System Services Runtime Environment
that causes the Tivoli Key Lifecycle Manager GUI panels to pop up into a second window
during initial configuration. To resolve this problem, close the pop-up window and log out of
the original window where you started your session. Once you log back into ISC the problem
should not happen again.

A.8.14 Status message on Tivoli Key Lifecycle Manager indicates that I'm
ready to serve keys however my device can't make a connection
The Tivoli Key Lifecycle Manager panels only indicate that you are configured to serve key
material. They do not actually indicate if the key server itself is up and running. To determine
if the key server is actually up and running you can run the netstat command from your
operator’s console to ensure your key server is listening on the TCP and SSL ports.


For example, if the System Services Runtime Environment servant task is named System
Services Runtime EnvironmentS (<SSRE_PROC_PREFIX>S), and the Tivoli Key Lifecycle
Manager keyserver is configured to use port 3801 for TCP and port 441 for SSL, run this
command:
D TCPIP,,NETSTAT,ALLCON

This should display the output shown in Example A-9 that indicates the key server is up on
both ports and listening for incoming requests.

Example A-9 Netstat output
...
System Services Runtime EnvironmentS 00004462 0.0.0.0..441 0.0.0.0..0 LISTEN
System Services Runtime EnvironmentS 00004461 0.0.0.0..3801 0.0.0.0..0 LISTEN
...

If either of these entries is missing, or they are listed in the ESTBLSH state, that would
indicate that there was a problem starting up the key server.

If the key server comes up on both ports then there may be a firewall in between the device
and the Tivoli Key Lifecycle Manager key server. From the device side try to ping the IP or
hostname where the Tivoli Key Lifecycle Manager key server is running to ensure you can
make a connection. For example:
ping my.tklm.server.com

No response back from the ping command would indicate that something is blocking your
connection to the Tivoli Key Lifecycle Manager keyserver. Work with your network
administrators on both ends to determine if a firewall is preventing the connection.

If the Tivoli Key Lifecycle Manager Key Server is not up on both ports, you will need to ensure
that any port reserves are taken off of the ports you intend to use for the TCP and SSL key
server ports that are configured on the Tivoli Key Lifecycle Manager configuration page. You
can either leave the ports open to any task, or the recommended and more secure practice
would be to update your TCPIP settings so that the servant task name is associated with the
Tivoli Key Lifecycle Manager key server ports. You may need to contact your network
administrator for assistance with updating your TCPIP settings. Port reserves are typically
configured in a TCPIP profile dataset, which is defined within the started JCL for TCPIP in
SYS1.PROCLIB. For example, if your servant task name is System Services Runtime
EnvironmentS (<SSRE_PROC_PREFIX>S), and your TCP and SSL ports are 3801 and 441,
you would want to update your TCPIP profile with the lines in Example A-10.

Example A-10 TCPIP profile PORT statement additions for Tivoli Key Lifecycle Manager
3801 TCP System Services Runtime EnvironmentS SHAREPORT ; Key Manager
1443 TCP System Services Runtime EnvironmentS SHAREPORT ; Key Manager

After recycling Tivoli Key Lifecycle Manager the key server should successfully come up on
both ports.

Additionally, if you have migrated your IBM Encryption Key Manager to Tivoli Key Lifecycle
Manager, or if you have IBM Encryption Key Manager up and running on the same system
where you have installed Tivoli Key Lifecycle Manager, ensure that IBM Encryption Key
Manager is not already using the ports you have defined for the Tivoli Key Lifecycle Manager
key server. If IBM Encryption Key Manager is up and running using the same ports that you
have configured for Tivoli Key Lifecycle Manager, the key server will fail to come up on those


ports. To resolve this, either stop IBM Encryption Key Manager, or configure IBM Encryption
Key Manager and Tivoli Key Lifecycle Manager to use a different set of ports.

If the key server again does not come up and you have configured Tivoli Key Lifecycle
Manager to use a RACF keystore, either JCERACFKS or JCECCARACFKS, ensure that you
have customized and run the samples/racfpermissions.rexx script located in the tklm.tar. If
this file is not customized and executed to give the SSREADM user and SSREGRP group
authority to your RACF keyring, Tivoli Key Lifecycle Manager will fail to load the master
keystore when System Services Runtime Environment restarts and the key server will not
come up on the TCP and SSL ports.

If you are using a hardware-based keystore, either the JCECCAKS or JCECCARACFKS
keystore, ensure that ICSF is up and running. If ICSF is not up and running Tivoli Key
Lifecycle Manager will fail to retrieve the needed key material from the master keystore and
the Tivoli Key Lifecycle Manager key server will fail to start up on the TCP and SSL ports.

A.8.15 Unable to update the Tivoli Key Lifecycle Manager configuration after
recycling System Services Runtime Environment
If you have configured Tivoli Key Lifecycle Manager with a RACF-based keystore, either
JCERACFKS or JCECCARACFKS, and you forget to run the samples/racfpermissions.rexx
script located within the tklm.tar file, Tivoli Key Lifecycle Manager will not be able to load the
master keystore after you recycle System Services Runtime Environment. The Tivoli Key
Lifecycle Manager key server will fail to come up and as you navigate through the Tivoli Key
Lifecycle Manager panels and attempt to make updates you will receive failures indicating
that you are unable to update any of the settings.

For example, if you navigate to the Key Administration pages you may get the errors shown in
Figure A-26.

CTGKM0210E Unable to update setting to accept requests from all drives.
CTGKM0601E An error occurred adding/updating value for attribute
ds8k.acceptUnknownDrives

And if you navigate to the Configuration page you might get the errors shown in Figure A-27.

CTGKM0102E Unable to update key serving parameters information.
CTGKM0101E Unable to update audit information.

To correct this problem you need to customize and run the samples/racfpermissions.rexx
script located within the tklm.tar file. At the very top of the sample there are instructions for
updating it. As an example, if the owner of your keyring is SSRECFG, and the name of your
keyring is TKLMKeyring, you should enter the following values in the sample:
groupid = "SSREGRP"
userid = "SSREADM"
ownerid = "SSRECFG"
keyring = "TKLMKeyring"

After running the sample, recycle System Services Runtime Environment and you should be
able to update the Tivoli Key Lifecycle Manager configuration again.


A.8.16 Receiving NOT AUTHORIZED error messages when running the
samples/racfpermissions.rexx script to setup permissions to my RACF
keyring
If you attempt to run this sample with a user who does not have authority to execute the
RACF commands in this sample, like SSRECFG, you will receive many NOT AUTHORIZED
error messages.

To resolve this problem you should switch to a user with authority to execute the RACF
commands listed in this sample.

A.9 Information to gather when Tivoli Key Lifecycle Manager
deployment fails
System Services Runtime Environment configuration files:
– /etc/System Services Runtime Environment/V1R1/SSRE_default/SSRE_env.sh
– /etc/System Services Runtime Environment/V1R1/conf/abcdefh.cfg file used
System Services Runtime Environment log files
– /var/System Services Runtime Environment/V1R1/logs by default
SSRE_APPSERVER_ROOT/profiles/default/bin/rasCollector.sh
Run this script, it gathers up information and packages in a jar file
System Services Runtime Environment Servant, Control, and Daemon job logs
– Servant region is <SSRE_PROC_PREFIX>S in SDSF
– Control region is <SSRE_PROC_PREFIX> in SDSF
– Daemon region is <SSRE_PROC_PREFIX>D in SDSF
Tivoli Key Lifecycle Manager log files found in your /tklmProductInstall/logs, both the most
recent log file and any previous log files if appropriate. For example if the user had
problems running more then one script. Having all the log files will help to determine
where the problem stems from.
Tivoli Key Lifecycle Manager install response file:
/tklmProductInstall/bin/tklmInstall.response
Tivoli Key Lifecycle Manager uninstall response file:
/tklmProductInstall/bin/tklmUninstall.response
Tivoli Key Lifecycle Manager Installed Components file:
/tklmProductInstall/bin/.installedComponents
Tivoli Key Lifecycle Manager debug log: <TKLM_HOME>/logs/
Tivoli Key Lifecycle Manager version which can be obtained by running the
/tklmProductInstall/bin/versionInfo.sh shell script
DB2 version
z/OS version
ICSF version if applicable
DUMP - see A.1.17, “Taking a console dump of System Services Runtime Environment”
on page 114 for details.


System Services Runtime Environment version, which can be obtained on the ISC
console Welcome page for System Services Runtime Environment.
For example, after Tivoli Key Lifecycle Manager is successfully deployed you will see the
ISC welcome page shown in Figure A-28.

Figure A-28 Version information

If the Tivoli Key Lifecycle Manager post SMP/E installation scripts failed to deploy Tivoli
Key Lifecycle Manager within System Services Runtime Environment, the bottom row
"Tivoli Key Lifecycle Manager 1.0" will be missing.

A.10 Enabling System Services Runtime Environment trace
Enabling the trace for the System Services Runtime Environment server process is done
using the ISC while the System Services Runtime Environment is active. You can configure
the System Services Runtime Environment server to start in a trace-enabled state, or change
dynamically by setting the appropriate configuration properties.

When enabling trace at server startup, the diagnostic trace configuration settings for the
System Services Runtime Environment process determines the initial trace state. The
configuration settings are read at System Services Runtime Environment startup and used to
configure the trace service. You can also change many of the trace service properties or
settings while the System Services Runtime Environment is running.

The following are the steps to enable tracing at server startup:
1. LOGON to the ISC console and select the WebSphere Application Server view
2. Go to Troubleshooting → Logs and Trace in the console navigation tree, then click
Server → Change Log Detail Levels
3. Click Configuration.
4. Scroll down to com.ibm.zosconsole.* and click it.
5. Set the trace specification to All messages and Traces.
Note: It is recommended you allow trace and log settings to default unless instructed to
change them by IBM support. Some trace and log settings can cause a performance
degradation on the system.
6. While still on the com.ibm.zconsole.* selection, click Message and Trace Levels, and
select finest option.
7. Click Apply, then OK to save the changed configuration.


8. Re-start the server.

To enable tracing on a running server do the following:
1. LOGON to the ISC console and select the WebSphere Application Server view.
2. Go to Troubleshooting → Logs and Trace in the console navigation tree, then click
Server → Change Log Detail Levels.
3. Click Runtime.
4. Scroll down to com.ibm.zosconsole.* and click on it.
5. Set the trace specification to All messages and Traces.
6. While still on the com.ibm.zconsole.* selection, click Message and Trace Levels, and
select finest option.
7. Click Apply and OK.

You can confirm your runtime changes by viewing the System Services Runtime Environment
control process. You should see a message in the SYSPRINT for the controller region similar
to that shown in Figure A-29.

BBOO0222I: TRAS0018I: The trace state has changed. The new The new trace state
is
*=info:com.ibm.zosconsole.*=all.
Interpreting the trace output
...
Figure A-29 Trace enabled message

A.11 Enabling Tivoli Key Lifecycle Manager trace
When Tivoli Key Lifecycle Manager trace is turned on, Tivoli Key Lifecycle Manager writes to
the Servant log (<SSRE_PROC_PREFIX>S) and debug log. The Servant log will also contain
other traces from WebSphere and System Services Runtime Environment depending on your
System Services Runtime Environment configuration. The Tivoli Key Lifecycle Manager
debug log, however, will only contain Tivoli Key Lifecycle Manager traces. Both logs are
helpful when trying to pinpoint where a problem stems from.

There are two steps to turning on Tivoli Key Lifecycle Manager tracing. First you must turn
debug to all within the Tivoli Key Lifecycle Manager configuration file, and then you must
configure System Services Runtime Environment's tracing filter for the Tivoli Key Lifecycle
Manager traces.

To set debug to all within Tivoli Key Lifecycle Manager's configuration file, you can simply
stop System Services Runtime Environment, edit the
TKLM_HOME/config/TKLMgrConfig.properties file, and change the line:
debug=none

to:
debug=all

Then restart System Services Runtime Environment.

Optionally, you can make this change using the Tivoli Key Lifecycle Manager CLI, which
provides an interface for updating the Tivoli Key Lifecycle Manager configuration file. To


update the debug setting using the Tivoli Key Lifecycle Manager CLI, start a WSAdmin
command prompt, for example with this command:
<SSRE_HOME>/bin/wsadmin.sh -username SSRECFG -password your_password_here -lang
jython

Once WSAdmin displays a prompt, enter the following command to update the Tivoli Key
Lifecycle Manager configuration file to debug=all:
print AdminTask.tklmConfigUpdateEntry('[-name "debug" -value "all"]')

Turn on Tivoli Key Lifecycle Manager tracing within System Services Runtime Environment's
tracing filter by selecting:

Troubleshooting → Logs and Trace → server1 → Change Log Detail Levels → Runtime

Check the box for Save runtime changes to configuration as well and update the Change
Log Detail Levels textbox to say:
*=warning: keymanager=all

Click OK and click the option to save directly to master configuration.

Once the user reproduces the problem, the Tivoli Key Lifecycle Manager debug log (its
location is defined in your Tivoli Key Lifecycle Manager configuration file, by default it is
<TKLM_HOME>/logs) and the System Services Runtime Environment Servant region
(<SSRE_PROC_PREFIX>S) will contain Tivoli Key Lifecycle Manager traces.

Remember to turn off tracing once it is no longer needed. Tracing can be turned off by
starting another WSAdmin prompt and submitting the following Tivoli Key Lifecycle Manager
CLI command:
Print AdminTask.tklmConfigUpdateEntry('[-name "debug" -value "none"]')


B

Appendix B. Basics of cryptography
This appendix introduces some basic cryptographic concepts. The following topics are
covered:
Cryptographic algorithms
– Symmetric key algorithms
– Asymmetric key algorithms
Padding
Encryption modes
Hybrid encryption
Digital signature
Certificates


B.1 Introduction to cryptography
The word cryptography originates from the Greek words “kryptos,” which means hidden and
“gráfo,” which means to write or to speak. Cryptography deals with securing information by
transforming it using cryptographic algorithms. This transformation is performed so that the
real form cannot be revealed without special information. This special information is referred
to as “the key”.

The process of transforming data from clear text into a hidden form is called encryption. And
the reverse transformation, from a hidden form into clear text, is called decryption.

B.2 Cryptographic algorithms
Currently there are two categories of cryptographic algorithms intended for encrypting and
decrypting data: symmetric key algorithms and asymmetric key algorithms.

B.2.1 Symmetric key algorithms
In symmetric key algorithms, the same key is used for both encryption and decryption.
Because a symmetric key can be used to decrypt everything that has been encrypted with it,
it needs to be kept secret. Because of this, it is often referred to as a “secret key”.

If several parties want to share secret information using a symmetric key, they all need to
know the key. This introduces the problem of distributing the key to whomever is authorized
to use it, without exposing it to those which are not authorized to get it.

Figure B-1 shows the flow of encryption and decryption using symmetric key algorithms. As
shown, the same key is used for both encryption and decryption.

Symmetric key

Message in clear Symmetric Encrypted message Symmetric Message in clear
Secret message encryption tQ${3Lo9*s42 decryption Secret message
algorithm algorithm

Figure B-1 Symmetric key algorithm

Symmetric key algorithms perform comparatively faster than asymmetric algorithms, but the
key management stage can increase rapidly in complexity when several parties are involved,
because the secret key needs to be distributed securely to everybody involved.

Today, widely used symmetric algorithms include: DES, Triple-DES, Blowfish, and AES.

Note: Even though DES is still in use, it is now considered to have too short a key for
ensuring proper encryption strength. Currently it is recommended that you migrate to
algorithms with longer keys, such as Triple-DES or AES.


B.2.2 Asymmetric key algorithms
In asymmetric key algorithms, there are two related keys (the “key pair”) involved. When data
is encrypted with one of the keys, only the other key of the pair can be used for decryption.
The owner of the key pair selects one of the keys to become a private key, which needs to be
kept secret. The other key then becomes a public key, and can be freely distributed. The key
construction algorithm is such that it is extremely difficult (that is, impractical) to derive the
value of the private key from the public key value.

Because the public key is freely distributed, anybody can encrypt messages with it, but only
the holder of the private key can decrypt those messages. The benefit of this is that no secret
key needs to be distributed before secure communication can take place (as is the case with
symmetric algorithms). If two parties want to communicate securely using an asymmetric
algorithm, they both use the recipient’s public key for encrypting messages, but use their own
private key for decrypting what they receive.

Figure B-2 shows the flow of encrypting and decryption using asymmetric key algorithms.
Encryption is done using the recipient’s public key, and decryption can then only occur using
the recipient’s private key.

Key pair

Recipient’s
Recipient’s private
public key
key

Message in clear Asymmetric Encrypted message Asymmetric Message in clear
Secret message encryption $9?X+D23-42 decryption Secret message
algorithm algorithm

Figure B-2 Asymmetric key algorithm

Compared to symmetric key algorithms, asymmetric algorithms are extremely heavy
consumers of computing resources. They are usually reserved for encryption of short bursts
of data, and are used in combination with symmetric key algorithms. One of the most widely
used asymmetric algorithms today is RSA (Rivest-Shamir-Adleman).

B.3 Padding
Many encryption algorithms work by encrypting one block of clear text at a time. The size of
these blocks depends on the algorithm used and the size of the keys. If the size of the data
that needs to be encrypted does not reach a whole number of blocks, the data needs to be
“padded.” This padding takes place before the data is encrypted, and it is removed at
decryption time. Several padding schemes are in use today.

B.4 Encryption modes
When encrypting using standard encryption algorithms, two identical data elements that are
encrypted with the same key and algorithm would end up looking the same when encrypted.
Because normal encryption algorithms work in blocks, repeating patterns in the clear data

Appendix B. Basics of cryptography 151

can end up as repeating patterns in the encrypted data, as well. This would make it easier for
someone to break the encryption.

One way to avoid this is by using the output of each block encryption to modify the input of the
next sequential block’s encryption (for example, by an exclusive OR operation). In this way,
all encrypted data blocks contribute to changing the pattern of the next encrypted block. To
start up this process, a randomly generated initial value is used to change the encrypted
output of the first block of data. This value is called an initialization vector, and it must also
be known by the recipient of the encrypted data.

B.5 Hybrid encryption
Both symmetric and asymmetric cryptographic algorithms offer advantages and
disadvantages. Symmetric algorithms are fast, but securely distributing the symmetric keys to
many users may prove to be a very complex and cumbersome process. Conversely,
asymmetric algorithms are slow and extremely demanding of computing resources, but they
solve the key distribution problem because a public key does not require to be secured.

Hybrid encryption attempts to exploit the advantages of both kinds of algorithm classes, while
avoiding their disadvantages.

Figure B-3 on page 153 shows how hybrid encryption works. When the sender of a message
wants to send it encrypted to a recipient, the sender starts up the process by generating a
random symmetric key that is used encrypt the secret message. The symmetric key is then
encrypted using the recipient’s asymmetric public key. The final message that is sent to the
recipient comprises two parts: the encrypted symmetric data key that will be recovered by the
recipient using the recipient’s asymmetric private key, and the recovered symmetric key that
will be used to decrypt the message.

Note that this approach allows the exploitation of the symmetric encryption/decryption
inherent efficiency for the transmitted data part. It also exploits the much simpler key
management processes that asymmetric encryption/decryption offers. The cost of
asymmetric encryption is kept to a minimum because it is expected that the symmetric key
will be considerably shorter than the secret message itself.


Message in clear Message in clear
Secret message Secret message

Sender Receiver

Encrypted message
Symmetric Symmetric
encryption KJ&5#%l@s42 decryption
algorithm algorithm

Encrypted
symmetric key
Asymmetric Asymmetric
encryption decryption
algorithm algorithm
Random Decrypted
symmetric symmetric
key key

Key pair

Recipient’s
Recipient’s private
public key
key

Figure B-3 Hybrid encryption

With hybrid encryption, the use of the random symmetric key is not necessarily limited to only
encrypting one message. Instead, the key can be reused over the course of a communication
session. In that case, the random symmetric key is sometimes referred as a session key.

Hybrid encryption techniques are heavily used; a prominent example is the SSL/TLS
protocol.

B.6 Digital signatures
Digital signature technology aims to provide the same guarantees that you expect from an
handwritten signature, but at the scale, distance, and speed that electronics permit. A
handwritten signature on a document makes it evidential, and a digital signature provides this
same property to a transmitted message.

The algorithms used for making up digital signatures are usually a combination of asymmetric
cryptographic algorithms and one-way hash algorithms.

One-way hash algorithms
One-way hash algorithms produce hash values, that is, fixed-length binary values computed
from an input message. These hash values are usually in the order of a few tens or hundreds
of bits in length. However, the input message can be of any length, and it cannot be retrieved
from the hash value (thus, “one-way hash”).

A hash value is used as a checksum, or “fingerprint”, for the message that produced it,
because changing a single bit in the message would change the hash value. The recipient of
a message can then verify the integrity of a received message by matching the initial hash
value sent with the message and the hash value computed on the final received message.


However, the number of possible combinations produced by the fixed length of the hash
value is much smaller than the combinations that the message length itself is expected to
produce. It may happen that two different messages produce the same hash value. This is
called a “collision” and is unavoidable when using one-way hashes.

Cryptographic techniques are therefore used when designing one-way hash algorithms to
make it extremely difficult for an attacker to find the necessary modifications to a given
message so that it could again, though modified, produce the same hash value.

Several families of algorithms aim to produce one-way hashes: Message Authentication
Code (MAC), Message Detection Code (MDC), Message Digest (MD), Secure Hash
Algorithm (SHA), and so on.

Digital signature generation
Figure B-4 shows the generation of a digital signature for a given message. First, the hash
value of the message is calculated using a chosen one-way hash algorithm. Then the hash
value is encrypted using the signer’s asymmetric private key. This last step implies the signer
is using the unique private key that the signer owns.

Key pair

Signer’s
Signer’s private
public key
key

Digital Signature
Hash code Asymmetric of the message
One-way-hash
Message to sign 5BFA812B42 encryption 8Y%-@5L/42
algorithm
algorithm

Figure B-4 Digital signature generation

Digital signature verification
Figure B-5 on page 155 illustrates how a digital signature is verified. First the signature is
“decrypted” using the signer’s public key, thus yielding the hash value that was computed just
prior to signing the message. A hash value is then calculated against the received message
and the two hash values are compared. If they match, the process has then established that:
1. The message went unmodified to the recipient.
2. The message was actually sent by the declared signer because the digital signature (that
is the initial hash value encrypted with the signer’s private key) could be decrypted with
the signer’s public key.


Signer’s
public
key

Received
digital Signature
Hash code
Asymmetric
8Y%-@5L/42 algorithm 5BFA812B42
Verify if Signature
hashes verification
match ok / not ok
Hash code
One-way-hash
Received message algorithm 5BFA812B42

Figure B-5 Digital signature verification

Digital signatures also play a key role in the generation of digital certificates as a proof of the
integrity and origin of the digital certificate.

Digital signatures are therefore a combination of one-way hash and asymmetric algorithms.
Widely used combinations include MD2 with RSA, MD5 with RSA, SHA1 with RSA, SHA1
with DSA.

B.7 Digital certificates
In large uncontrolled networks, where everybody can publish their own public keys, there is a
requirement to certify that the declared identity is actually the key owner’s identity. The
method widely in use today to achieve this certification is to package one’s public key in a file
called a “digital certificate.” The digital certificate file contains, among other information, the
public key value and the public key owner’s name, and is digitally signed by a Certificate
Authority (CA).

Certificate authorities have an administrative process in place to verify the identity of
requestor and whether this identity, as it will be shown in the certificate, is unique. After this
administrative process is successfully performed, a digital certificate is issued to the
requestor. The content of the certificate is digitally signed by the CA with its private key.

Because the CA makes its public key public in a CA certificate, all recipients of the certificate
can use the CA’s public key to verify the certificate digital signature, thus binding the public
key value in the certificate to the owner’s (the “subject’s”) identity, which is also shown in the
certificate.

Note that a certificate verification is to be performed by software, and the program that
verifies certificates is set up to accept certificates issued by selected CAs. These are the CAs
that are trusted as per the installation trust policy.

The prominent format in use today for digital certificates is X.509 V3 format, which is
promoted by the IETF PKIX group. X.509 is a full standard for a public key infrastructure. It
was originally developed in 1988 as part of the wider X.500 directory standards.


An X.509 V3 certificate contains the following information:
Certificate fields
– Version
– Serial number
– Algorithm ID
– Issuer
– Validity
• Not before
• Not after
– Subject
– Subject public key info
• Public key algorithm
• Subject public key
– Issuer unique identifier
– Subject unique identifier
– Extensions
Certificate signature algorithm
Certificate signature

As mentioned, the digital certificate signature is used at verification time to check the integrity
of a received certificate and its origin; that is, the CA that signed the certificate.


Related publications

The publications listed in this section are considered particularly suitable for a more detailed
discussion of the topics covered in this paper.

IBM Redbooks publications
For information about ordering these publications, see “How to get Redbooks” on page 158.
Note that some of the documents referenced here may be available in softcopy only.
IBM System Storage DS8000: Disk Encryption Implementation and Usage Guidelines,
REDP-4500-00
IBM System Storage DS8000: Architecture and Implementation, SG24-6786-06
IBM System Storage Tape Encryption Solutions, SG24-7320-02

Other publications
These publications are also relevant as further information sources:
IBM System Services Runtime Environment for z/OS Configuration Guide, GA76-0404
MVS Programming: Resource Recovery, SA22-7616
IBM Tivoli Key Lifecycle Manager Installation and Configuration Guide, SC23-9977
IBM Tivoli Key Lifecycle Manager Quick Start Guide, GI11-8738
IBM Tivoli Key Lifecycle Manager Program Directory, GI11-4300
z/OS V1R9.0 Security Server RACF Auditor’s Guide, SA22-7684

Online resources
These Web sites are also relevant as further information sources:
IBM System Services Runtime Environment:
http://www.ibm.com/servers/eserver/zseries/software/ssre/
WebSphere Application Server 6.1 Information Center:
websphere.home.doc/welcome.html
WebSphere Application Server (z/OS) 6.1 Information Center:
websphere.zseries.doc/info/welcome_nd.html
Tivoli Key Lifecycle Manager Information Center:
http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/index.jsp?topic=/com.ibm
.tklm.doc/welcome.htm
Tivoli Software Information Center:


http://publib.boulder.ibm.com/tividd/td/link/tdprodlist.html
Tivoli Software Glossary includes definitions for many of the technical terms related to
Tivoli software
http://publib.boulder.ibm.com/tividd/glossary/tivoliglossarymst.htm
Integrated Cryptographic Services Facility:
http://publib.boulder.ibm.com/infocenter/zos/v1r9/index.jsp
IBM Resource Access Control Facility:
http://www.ibm.com/servers/eserver/zseries/zos/racf/library/
DB2 for z/OS:
http://www.ibm.com/software/data/db2/zos/roadmap.html

How to get Redbooks
You can search for, view, or download Redbooks, Redpapers, Technotes, draft publications
and Additional materials, as well as order hardcopy Redbooks publications, at this Web site:
ibm.com/redbooks

Help from IBM
IBM Support and downloads
ibm.com/support

IBM Global Services
ibm.com/services


Index

Numerics E
EEDK 15
256-bit AES data key 19
Encrypting Data on Disk 24
3494 10, 13, 18, 20, 34, 36, 40
Encryption Key Manager (EKM) 8, 23, 38
3953
Encryption Key Manager instance 38
F05 36
IP address 38
Encryption Key Manager server 39
A encryption policies 10
Advanced Encryption Standard 8 encryption process 15
AES 4, 8–9, 19, 40, 150 environment variable 58
Application Response Management (ARM) 115 Externally Encrypted Data Key 15
application-managed encryption (AME) 10, 17–18, 23,
34
asymmetric algorithm 149
F
Fibre Channel 20
First Failure Data Capture (FFDC) 112
B full disk encryption (FDE) 23
backup information 94
barcode encryption policy 14, 34
BBA.SBBA Load 50–53, 60–61, 109
H
Host system
BEP 14
changes required 51
Blowfish 150
requisites for Tivoli Key Lifecycle Manager 65
hybrid encryption 152
C
certificate 149
Certificate Authority 155
I
I/O supervisor (IOS) 36
checksum 153
IBM DB2 44, 46
configuration file 50, 59, 127, 132, 144
IBM System Services Runtime Environment 49, 55–56,
Configuration Guide 44
60–61
cryptographic algorithm 150
user ID 60
cryptography
IBM System Storage Tape Drive TS1120 2
algorithms 150
IBM tape
asymmetric key algorithms 151
library 11, 24
digital certificate 155
IBM Tivoli Key Lifecycle Manager
digital signature 153
installation 65
encryption modes 151
IBMJCE provider 73, 133
hybrid encryption 152
IBMJCECCA provider 134
introduction 150
IBMJCECCA provider 29, 66, 71, 73, 133
padding 151
precedence order 73
symmetric key algorithms 150
IBMJCEFIPS 8
ICSF 11
D in-band 11
data exchange with business partners 25 initialization vector 152
data exchange within your enterprise 24 installTKLM.sh script 66, 80–82, 126, 129
data key 2, 15, 28, 40 Integrated Cryptographic Services Facility (ICSF) 11, 29,
DB2 table 93–94, 97, 103 44
DB2 unload 94 Integrated Solutions Console (ISC) 31, 47, 68, 83, 85,
decryption process 16 88, 90, 95, 100, 110, 114, 120, 135, 139, 141, 145
DES 150 Internal Label encryption policy (ILEP) 34
device driver 34 IP address 27, 118
digital signature 149
DK 15
DS8000 turbo drive 29–32, 45
J
Java Cryptography Extension 8


Java security keystore 8 Tivoli Key Lifecycle Manager 65
Java Virtual Machine (JVM) 115 private key 15, 25, 40, 99, 105, 134, 140, 151–152,
JCE 8 154–155
JCECCARACFKS (JCECCAKS) 66, 71 Bad encoding 134, 140
JCEKS 29–30, 94, 98, 103 Program Directory
JDBC driver 66, 78, 80 System Services Runtime Environment 49
Tivoli Key Lifecycle Manager 64
public key 15–16, 25, 40, 151–152, 154–156
K
Key Encrypting Key (KEK) 15
key flow 12 R
key material 29–32, 45, 94, 98–99, 103–105, 111, 132, RACF keyring 29, 86, 109, 111, 129, 143–144
141, 143 Redbooks Web site 158
key pair 151 Contact us xii
key ring 29 Resource Access Control Facility (RACF) 5, 27–32, 48,
key server 23, 28, 32, 45, 141–142 94, 98
keystore 8–9, 15–16, 20, 23, 25, 27–30, 32, 38–40, Resource Recovery Service
44–46, 66, 71, 86, 88, 94, 98, 103–105, 128, 132–135, IBM System Services Runtime Environment 47
140–141, 143 Resource Recovery Service (RRS) 47
response file 66, 77–80, 125, 128, 144
default location 125
L successful creation 80
library-managed encryption (LME) 13, 15, 23, 34–35 Rivest-Shamir-Adleman 15
log file RSA 8, 15, 151
configSSRE.sh 60
createSSRE.sh 62
installTKLM.sh 81 S
ssre_env.sh 58 Secure Hash Algorithm (SHA) 154
uninstallTKLM.sh 82 Servant Region (SR) 112–113, 144, 147
LTO 9 SMF record
type 120 53
SMF record collection 53
M Software 11
Message Authentication Code (MAC) 154 SSL port 141–142
Mixed Mode example 20 SSRECFG user ID 122, 124–128, 130–131, 135, 139
Model C10 37 SSREGRP group 122–123, 127–128, 131, 135, 143
Model F05 37 symmetric algorithm 149–150
System Authorization Facility (SAF) 48
O System Management Facilities (SMF) 44, 46–48, 50, 53,
one-way hash 153 63, 66, 68, 83, 138, 141
out-of-band 12 System Services Runtime Environment
Configuration HFS 45
Fixpack 1 48
P Installation 49
PKCS 28, 98, 103–105 system-managed encryption 10–11, 15, 20, 34
planning
checklist 38
database 28 T
EKM to TKLM migration 38 tape drive
Encrypting Data on Tape 24 TS1100 family 29
encryption method comparison 34 tape encryption 3–5, 9, 20
in-band and out-of-band 35 process flow 3
keys and certificates 26 Tivoli Key Lifecycle Manager
performance and capacity 26 deploying in a Sysplex 88
recovery 37 features 7
redundant key servers 27 Installation 64
rekeying 25 introduction 2
selecting the keystore 28 key exchange process 8
sysplex vs monoplex 30 requirements 44
Preventive Service Planning (PSP) supported keystores 29
System Services Runtime Environment 49 supported platforms 6


use of DB2 46
use of ICSF 48
use of RACF/SAF 48
use of SMF 48
Triple-DES 150
TS1120 9, 18
TS1120 Model C06
controller 37
TS1120 Tape Drive 2, 11, 37
TS3100 14
TS3200 14
TS3310 14
TS3400 14, 18, 37, 40
TS3500 10, 13–14, 18, 20, 34, 36, 40
TS7700 11–12, 41

U
unprotected URI 84

V
Virtualization Engine 12, 41

W
Write Once Read Many (WORM) 5

Index 161

Back cover ®

IBM Tivoli Key
Lifecycle Manager
Redpaper
for z/OS
™

Features and benefits This IBM Redbooks publication provides details of a new offering called
IBM Tivoli Key Lifecycle Manager. We introduce the product, provide INTERNATIONAL
planning suggestions, and detail the installation of IBM Tivoli Key TECHNICAL
Planning, installation,
Lifecycle Manager on the z/OS operating system running on a System z SUPPORT
and use server. ORGANIZATION
Tivoli Key Lifecycle Manager is IBM’s latest storage device encryption
Troubleshooting tips
solution. It allows enterprises to create, manage, backup, and
distribute their cryptographic key material from a single control point.
Tivoli Key Lifecycle Manager stems from the existing IBM Encryption
Key Manager solution. Unlike IBM Encryption Key Manager, which only BUILDING TECHNICAL
provided a key server, Tivoli Key Lifecycle Manager provides real key INFORMATION BASED ON
management, security policy capabilities, and a Web-based PRACTICAL EXPERIENCE
user-interface for ease of use. It leverages the existing security
strengths of the z/OS platform by using Integrated Cryptographic IBM Redbooks are developed
Services Facility (ICSF), System Authorization Facility (SAF), and by the IBM International
Java-based keystores to store all the key material.
Technical Support
Organization. Experts from
IBM, Customers and Partners
from around the world create
timely technical information
based on realistic scenarios.
Specific recommendations
are provided to help you
implement IT solutions more
effectively in your
environment.

For more information:
ibm.com/redbooks

REDP-4472-00

Ibm tivoli key lifecycle manager for z os redp4472

More Related Content

What's hot (16)

Similar to Ibm tivoli key lifecycle manager for z os redp4472 (20)

More from Banking at Ho Chi Minh city (20)

Recently uploaded (20)

Ibm tivoli key lifecycle manager for z os redp4472