SlideShare a Scribd company logo
IBM WebSphere MQ for z/OS 
Performance and Accounting 
Alexander Ross (alexross@uk.ibm.com) 
Software Developer
Please Note 
IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal 
without notice at IBM’s sole discretion. Information regarding potential future products is 
intended to outline our general product direction and it should not be relied on in making a 
purchasing decision. 
The information mentioned regarding potential future products is not a commitment, promise, 
or legal obligation to deliver any material, code or functionality. Information about potential 
future products may not be incorporated into any contract. The development, release, and 
timing of any future features or functionality described for our products remains at our sole 
discretion 
Performance is based on measurements and projections using standard IBM benchmarks in a 
controlled environment. The actual throughput or performance that any user will experience 
will vary depending upon many factors, including considerations such as the amount of 
multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and 
the workload processed. Therefore, no assurance can be given that an individual user will 
achieve results similar to those stated here. 
© 2014 IBM Corporation 
2
3 
Agenda 
• Introduction 
• Queue manager accounting and statistics 
• Channel initiator accounting and statistics
5 
Introduction 
• What are you trying to achieve? 
• What you are using? 
• Is your system efficient? 
• Being able to answer these questions allows you to: 
– Administer effectively 
– Prevent outages and degradations in service 
– Plan for growth or change 
– Save MIPS and money!
7 
Monitoring 
• WebSphere MQ provides various monitoring capabilities: 
– DISPLAY/PCF commands, MQ Explorer 
– PCF event messages 
• Queue manager – Start/stop, GET/PUT inhibited, unknown object 
• Configuration – Define/Alter/Delete/Refresh objects 
• Command – Commands issued and by who 
• Performance – Queue depth high/low/full 
• Channel/bridge – Start/stop, SSL 
– Queue manager performance statistics 
• High-level view of activity 
– Queue manager accounting data 
• Detailed information on MQI operations 
• Provides capability to charge users for queue manager usage 
– Channel statistics and accounting
Collecting statistics and accounting data 
• System Management Facility (SMF) records 
– Event messages on distributed 
• Statistics – SMF record type 115 
– START TRACE(STAT) 
– SMFSTAT system parameter (zPARM) 
• Accounting – SMF record type 116 
– START TRACE(ACCTG) 
– SMFACCT system parameter (zPARM) 
• Collection interval can be specified using the STATIME system 
parameter 
– If zero, the SMF global accounting interval is used 
9
10 
Collecting statistics and accounting data 
N 
O 
T 
E 
S 
• On distributed platforms enabling statistics and accounting information 
generates PCF messages, as per events. On z/OS, this information is 
generated as System Management Facility (SMF) records instead. 
• Statistics information is recorded as SMF 115 records. 
• Accounting data is recorded as SMF 116 records. 
• The START TRACE command can be used to start collecting this information. 
A parameter can be provided to specify which type of information should be 
collected. 
• To enable the collection of this information automatically system parameters 
can be set in the zPARM using CSQ6SYSP. When set, data collection will 
commence at queue manager start up. 
• SMF data is collected in memory and written out at periodic intervals. The 
interval MQ should use can be set using the STATIME system parameter. If 
this is set to zero WebSphere MQ uses the SMF global accounting interval.
Interpreting SMF data 
• SMF record layouts fully documented 
• SupportPac MP1B 
– MQSMF displays formatted records 
– Outputs information to various files (DDs) 
– Highlights potential out-of-line conditions 
– Can also output in comma-separated value (CSV) format for importing 
11 
in spreadsheets 
– MQCMD also available 
• Automates issuing of DISPLAY commands - outputs to CSV 
• Useful for monitoring a given channel or queue
13 
Queue manager statistics 
• Provides a lot of activity related information 
– Insignificant overhead so can enable this all the time 
– Keep historical data for trend analysis 
• Easy to see high-level activity 
– How much storage the queue manager is using 
– How many API calls are occurring 
– How much logging is occurring 
– How the buffer pools are performing 
• Problem diagnosis 
– What is using CPU? 
– What is causing delays?
MVS QM Date Time Puts Put1s Gets 
MVSA MQ1A 31/03/2014 00:00:00 53979 54120 306587 
MVSA MQ1A 31/03/2014 00:30:00 27212 42914 227198 
15 
Message Manager statistics 
• Provides counts of MQI requests during SMF interval 
• Includes both successful and failed requests 
– MQGET count includes those where no message was available 
• Written to MSGM and MSGMCSV DD by MQSMF 
• Do you know what your daily/monthly profile looks like? 
• Do you have busy hours each day, or busy days each 
month? 
– What is the trend? 
MVSA MQ1A 31/03/2014 01:00:00 23924 31470 169326
Do you know what your daily profile looks like? 
16 
1600000 
1400000 
1200000 
1000000 
800000 
600000 
400000 
200000 
0 
00:00:00 
01:30:00 
03:00:00 
04:30:00 
06:00:00 
07:30:00 
09:00:00 
10:30:00 
12:00:00 
13:30:00 
15:00:00 
16:30:00 
18:00:00 
19:30:00 
21:00:00 
22:30:00 
API Calls per 30mins 
MQPUT 
MQPUT1 
MQGET 
• The above chart shows potential MQGET inefficiencies 
– Multiple application instances ‘competing’ to get the messages? 
– Need to index one or more queues? 
• Can help to identify growing trends and the need for increased capacity
Log Manager statistics 
• Constraint for persistent messages is normally the logging rate 
• All persistent data changes are logged 
– Mainly MQPUTs and MQGETs of persistent messages 
• Also object changes, etc. 
• Primary reporting is on Control Intervals (CIs) 
– 4KB pages 
– If you’re using dual logging 2 CIs are written per 4KB page 
• Written to LOG and LOGCSV DD by MQSMF 
• Ask the following questions: 
– Do you have I/O issues? 
– Are you checkpointing more than you should? 
– Are you backing out more than you should? 
17
Logging overview 
• Application writes data in to log buffers 
– May have to wait if no log buffers are available 
– Log force requests the log write task to write up to the required RBA 
19 
• Waits for I/O to complete 
• Log write task 
– Waits for previous I/O to complete 
– Looks for highest requested log RBA to write 
– Writes up to and including this RBA, then iterates 
• One log write can include data from many tasks 
• More data written per I/O, the longer the I/O time 
• As more data is logged by the system an individual 
transaction’s commit time may increase
DASD information 
• Writing more data per I/O is more efficient than many short I/Os 
– An I/O has set-up time and data transfer time (connect) – e.g. 
• Set up 0.1 millisecond, connect 0.1 per 4KB page 
• One page per I/O = 0.1 + 0.1 = 0.2 ms = 5000/second = 5000 pages/sec 
• 9 pages per I/O = 0.1 + 0.9 = 1.0 ms = 1000/second = 9000 pages/sec 
– Each larger I/O takes longer but the overall data rate is greater 
• Can stripe the log data set across different volumes 
– Pages 1,5,9 to vol 1; 2,6 to vol 2; 3,7 to vol 3; 4,8 to vol 4 
– Can now perform I/O in parallel - can write 3 pages faster than 9! 
• Expect 140MB/second per log 
20
Logging statistics 
Write_Wait 0, Write_Nowait 538306, Write_Force 193 
Read_Stor 0, Read_Active 0, Read_Archive 0 
BSDS_Reqs 703, CIs_Created 61037, BFWR 7478 
ALW 6, CIs_Offload 60480, Checkpoints 0 
Read delayed 0, Tape Lookahead 0, Lookahead Mount 0 
Write_Susp 7430, Write_Reqs 12007, CI_Writes 64629 
Write_Serl 0, Write_Thrsh 48, Buff_Pagein 0 
WTB 0 
TVC 0 
ALR 0 
_____,__ write requests, CIs, Average I/O , After I/O , pages/IO 
21 
uSec , uSec , 
Log 1, 1 page 7500, 7500, 312, 5, 1 
Log 1,>1 page 4507, 57129, 617, 6, 13 
Standard deviation of first log, 1 page per I/O, response time +- 3 
Log 1, 1 page Longest I/O 3914 at 2014/04/10,08:55:57.429543 UTC 
Log 1, 1 page Longest Request 3918 at 2014/04/10,08:55:57.429543 UTC 
Log 1,>1 page Longest I/O 34352 at 2014/04/10,08:55:57.392493 UTC 
Log 1,>1 page Longest Request 34361 at 2014/04/10,08:55:57.392493 UTC 
Log write rate 2MB/s per copy 
Logger busy: 
Interval: 121 Idle: 115206421 (us)( 95%) LogIOSusp: 5193950 (us)( 4%)
Logging statistics 
•WTB > 0 indicates applications having to wait for buffers to be written first 
– OUTBUFF might need to be increased 
– Alternatively, DASD might be slow 
– MQSMF highlights potential warning conditions: 
22 
• MQQJST09W QJST requests waiting for buffer 99 
• MQQJST11W QJST OK logging rate 65 MB/Sec 
• Log read when transactions back out 
– Log reads also expected during queue manager start-up 
– Data read from buffers – good 
– Data read from active logs – OK 
– Data read from archive logs – BAD – investigate long UOW 
– MQSMF messages: 
• MQQJST00I QJST read log buffers from storage 3433 > 0 
• MQQJST01W QJST read log buffers from active logs 50838 > 0 
• MQQJST02S QJST read log buffers from archive logs 15 > 0
Logging statistics 
•Checkpoints is only incremented when LOGLOAD causes checkpoint 
– Checkpoints due to a log switch are not counted 
•CI_Writes shows 64629 4KB pages have been written 
– 2.1MB/sec average because SMF interval is 2 minutes = VERY LOW! 
• Singular logging because only Log 1 is reported 
• Look at single page I/O time 
– Should be fairly stable – see standard deviation 
• Multiple page I/O sizes vary 
– 312 uSec in example is very good 
• 1000 uSec is OK – if 2000 or higher then attention is warranted 
• Check average number of pages per I/O 
– If 50 or higher this indicates I/O is constrained by DASD speed or trying to 
23 
write too much data 
• Consider splitting the workload over an additional queue manager 
• Longest page I/O took 3.9ms 
– Useful if investigating performance issues, disk issues?
Buffer Manager statistics 
• Are the buffer pools performing well? 
– All private queue messages that are put or got require buffers in a 
buffer pool irrespective of persistence 
– A badly configured buffer pool may have a significant impact on 
performance or waste storage 
– Configuration of buffer pools is dependent on how they are going to be 
used 
• Short-lived messages 
– Keep whole working set of messages in buffer pool 
• Long-lived messages 
– More messages than can be contained in memory 
– Will always hit disk before being got 
• Written to BUFF and BUFFCSV DD by MQSMF 
24
= BPool 1, Size 5000,%full now 80, Highest %full 92, Disk reads 27019 
> 01 Buffs 5000 Low 390 Now 979 Getp 1408819 Getn 137508 
01 Rio 27019 STW 648745 TPW 26587 WIO 2417 IMW 12 
01 DWT 1236 DMC 0 STL 72300 STLA 8 SOS 0 
01 Above the bar PAGECLAS 4KB 
26 
Buffer pool statistics 
• No buffer available (SOS), buffer steal (STLA) and DMC should be 0 
– If not, consider increasing buffer pool size 
• Asynchronous write threshold (DWT) reached 1236 times 
– Number of 'dirty' pages >= 85% total pages 
• Synchronous write threshold (DMC) not reached above 
– Number of 'dirty' pages >= 95% total pages 
• Version 8 reports if the buffer pool is above the bar and whether it is page-fixed 
•MQSMF messages output if buffer pools may be too small: 
– MQQPST04E BP 3 Many (735) pages read from disk 
– MQQPST02S BP 3 Filled many (122) times
27 
Buffer pools for short-lived messages 
• Aim to keep the full working set of messages in memory 
– DWT should be 0 
• Always at least 15% buffers free 
• Pages written (TPW) might be greater than 0 due to checkpoint 
activity 
– However, we shouldn’t be writing because of buffer pool getting full! 
• Read operations (RIO) should be 0 
– If we have everything in memory, we should never need to read from 
the pageset! 
• Unless read from a page set after the queue manager is restarted 
• You are recommended to have sufficient buffers to handle your 
peak message rate
Buffer pools for long-lived messages 
• Expect messages to be written to and read from the page set 
• (RIO+TPW)/statistics interval is the I/O rate to page sets 
• During period that messages are processed (e.g. written during 
day and processed overnight) 
– Number of read I/Os (RIO) should be approximately the total number 
of pages written (TPW). 
• Shows that one page is read for every page written. 
• If RIO is much larger than TPW this means pages are being 
read multiple times. 
– This might indicate applications getting by message or correlation id 
28 
on a queue that is not indexed
Accounting data 
• Understand application activity 
• Can control at queue level using ACCTQ(ON|OFF|QMGR) 
– QMGR means inherit from ACCTQ queue manager attribute 
• Costs 5-10% CPU overhead 
• Heavyweight – multiple records may be cut for each 
transaction, and at SMF intervals for long running units of work 
– Get a lot of useful information about what is being done! 
– However, enabling this has been known to swamp an SMF 
29 
environment 
• Task/thread and queue statistics 
• Even though prolific: 
– Turn on for a few minutes each month to get view of activity 
– Become familiar with the data and the usage patterns
Application activity 
Open name AP01_REP_MQ20_DC_0_0007 Object type:Local Queue 
Base name AP01_REP_MQ20_DC_0_0007 Base type :Queue 
Queue indexed by NONE 
First opened 10-04-2014 08:52:48.62 
Last closed 10-04-2014 08:52:49.69 
Page set ID 4, Buffer pool 1 
Current opens 0, Total requests 158 
Generated messages : 0 
Persistent messages: GETs 37, PUTs 0, PUT1s 0 
Put to waiting getter: PUT 0, PUT1 0 
Put direct to shared queue: 0 
GETs: Valid 78, Max size 9824, Min size 75, Total bytes 151197 
GETs: Dest-S 156, Dest-G 0, Brow-S 0, Brow-G 0, Successful destructive 78 
Time on queue : Max 0.348850, Min 0.118915, Avg 0.231072 seconds 
-MQ call- N ET CT Susp LOGW PSET Epages skip expire 
Open : 1 20 20 0 
Close : 1 6 6 0 
Get : 156 43 38 0 0 0 0 570 0 
31 
Maximum depth encountered 154
33 
A good day versus a bad day 
• Obtain data for a good day to compare with 
– Without this it is much harder to identify the cause of problems 
Metric Good day Bad day 
Get count 16846 3365 
Get elapsed time (average) 329 2735 
Get CPU time (average) 321 966 
Get suspend time 0 1735 
Get pageset count 0 6752 
Get pageset elapsed time 0 1734 
Get valid count 6087 3233 
Get size (maximum) 1043924 1041564 
Get size (minimum) 115196 202600 
Get size (average) 847087 834928
Channel initiator statistics and accounting data 
• Version 8 introduces statistics and accounting data for: 
– Channel initiator address space 
– Channel activity 
• This information can be used for: 
– Monitoring 
– Capacity planning 
– Tuning 
• Previously, many customers have created their own ‘monitoring’ jobs which 
issue periodic DISPLAY CHSTATUS 
– Using this information it is difficult to: 
35 
• Manage historical data 
• Perform capacity planning 
• Investigate channel performance issues 
• Separate controls from queue manager allows 'opt in' 
• Updated MP1B SupportPac will format the data and introduces rule based 
reporting
36 
Channel initiator SMF controls 
• You can start channel initiator statistics (STAT) trace by: 
!MQ08 START TRACE(STAT) CLASS(4) 
CSQW130I !MQ08 'STAT' TRACE STARTED, ASSIGNED TRACE NUMBER 05 
CSQ9022I !MQ08 CSQWVCM1 ' START TRACE' NORMAL COMPLETION 
• You can start channel initiator accounting (ACCTG) trace by: 
!MQ08 START TRACE(ACCTG) CLASS(4) 
CSQW130I !MQ08 ‘ACCTG' TRACE STARTED, ASSIGNED TRACE NUMBER 06 
CSQ9022I !MQ08 CSQWVCM1 ' START TRACE' NORMAL COMPLETION 
• You can display trace by: 
!MQ08 DISPLAY TRACE(*) 
CSQW127I !MQ08 CURRENT TRACE ACTIVITY IS - 
TNO TYPE CLASS DEST USERID RMID 
02 STAT 01 SMF * * 
05 STAT 04 SMF * * 
06 ACCTG 04 SMF * * 
END OF TRACE REPORT 
•ALTER and STOP TRACE commands have also been updated
Channel initiator SMF controls 
• Channel initiator SMF data is collected at the same interval as 
the queue manager 
• Channel initiator statistics (STAT) trace 
– Provides a high level view of activity in the CHINIT address space 
• Channel initiator accounting (ACCTG) trace 
– Provides a detailed view of individual channels 
– STATCHL channel attribute controls granularity 
37 
• Equivalent to ACCTQ for queues 
• Data collected is a superset of that collected in the PCF 
messages on distributed
38 
Channel statistics versus accounting data 
• Statistics 
– Information about number of channels and TCB usage 
– Dispatchers, Adapters, DNS, SSL 
– Do I need more or less dispatchers? 
– Do I need more or less adapters? 
– Do I have spare capacity? 
• Accounting 
– What have the channels done? 
– What are the high use channels?
Channel initiator statistics SMF 115 sub-type 231 
• Channel initiator 
– QSG name 
– Number of current channels 
– Maximum current channels 
– Number of active channels 
– Maximum active channels 
– Maximum TCP/IP channels 
– Maximum LU 6.2 channels 
– Storage usage in MB 
• Dispatcher task 
– Task number (TCB address) 
– Number of requests for task 
– Busy CPU time of task 
– Sum of elapsed time of requests 
– Wait elapsed time of task 
• Adapter task 
– Task number (TCB address) 
– Number of requests for task 
– Busy CPU time of task 
– Sum of elapsed time of requests 
– Wait elapsed time of task 
39 
• DNS task 
– Task number (TCB address) 
– Number of requests for task 
– Busy CPU time of task 
– Sum of elapsed time of requests 
– Wait elapsed time of task 
– Time of day of max DNS request 
– Duration time of max DNS request 
• SSL task 
– Task number (TCB address) 
– Number of requests for task 
– Busy CPU time of task 
– Sum of elapsed time of requests 
– Wait elapsed time of task 
– Time of day of max SSL request 
– Duration of max SSL request
41 
Channel initiator statistics summary 
MV45,MQ20,2014/04/08,20:43:57,VRM:800, 
From 2014/04/08,20:41:54.984681 to 2014/04/08,20:43:57.237939 
duration 122.253258 seconds 
Number of current channels..................... 20 
Number of active channels .... ................ 20 
MAXCHL. Max allowed current channels........... 602 
ACTCHL. Max allowed active channels............ 602 
TCPCHL. Max allowed TCP/IP channels............ 602 
LU62CHL. Max allowed LU62 channels............. 602 
Storage used by Chinit......................... 22MB
Channel initiator statistics summary 
• Number of current and active channels 
– How close are you getting to the maximums? 
• Channel initiator storage usage 
– 31-bit usage - not much currently in 64-bit for the channel initiator 
• Are these trending upwards? 
– Monitor over time 
42 
1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 
3 0 0 0 
2 5 0 0 
2 0 0 0 
1 5 0 0 
1 0 0 0 
5 0 0 
0 
N u m b e r o f c u r r e n t c h a n n e l s 
1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 
5 0 0 
4 5 0 
4 0 0 
3 5 0 
3 0 0 
2 5 0 
2 0 0 
1 5 0 
1 0 0 
5 0 
0 
S T G u s e d i n M B
Dispatcher statistics 
MV45,MQ20,2014/04/08,20:43:57,VRM:800, 
From 2014/04/08,20:41:54.984681 to 2014/04/08,20:43:57.237939 
duration 122.253258 seconds 
Task,Type,Requests,Busy %, CPU used, CPU %,"avg CPU","avg ET" 
, , , , Seconds, , uSeconds,uSeconds 
0,DISP, 46, 0.0, 0.000000, 0.0, 12, 14 
1,DISP, 168912, 1.4, 0.028218, 0.0, 8, 10 
2,DISP, 168656, 1.3, 0.025142, 0.0, 7, 10 
3,DISP, 0, 0.0, 0.000000, 0.0, 0, 0 
4,DISP, 0, 0.0, 0.000000, 0.0, 0, 0 
Summ,DISP, 337614, 0.6, 0.053360, 0.0, 9, 10 
43 
0,DISP, number of channels on this TCB, 0 
1,DISP, number of channels on this TCB, 10 
2,DISP, number of channels on this TCB, 10 
3,DISP, number of channels on this TCB, 0 
4,DISP, number of channels on this TCB, 0 
Summ,DISP, number of channels on all TCBs, 20
Adapter statistics 
MV45,MQ20,2014/04/08,20:43:57,VRM:800, 
From 2014/04/08,20:41:54.984681 to 2014/04/08,20:43:57.237939 
duration 122.253258 seconds 
Task,Type,Requests,Busy %, CPU used, CPU %,"avg CPU","avg ET" 
, , , , Seconds, , uSeconds,uSeconds 
0,ADAP, 127599, 16.5, 0.953615, 0.8, 7, 158 
1,ADAP, 46790, 7.6, 0.309678, 0.3, 7, 199 
2,ADAP, 13702, 3.2, 0.065380, 0.1, 5, 284 
3,ADAP, 2909, 0.7, 0.029541, 0.0, 10, 279 
4,ADAP, 395, 0.1, 0.003179, 0.0, 8, 392 
5,ADAP, 37, 0.0, 0.000241, 0.0, 7, 149 
6,ADAP, 10, 0.0, 0.000175, 0.0, 17, 111 
7,ADAP, 0, 0.0, 0.000000, 0.0, 0, 0 
Summ,ADAP, 191442, 3.5, 1.361809, 0.1, 7, 179 
45
47 
DNS statistics 
MV45,MQ20,2014/04/08,20:41:54,VRM:800, 
From 2014/04/08,20:40:07.101220 to 2014/04/08,20:41:54.984681 
duration 107.883460 seconds 
Task,Type,Requests,Busy %, CPU used, CPU %, 
, , , , Seconds, , 
0,DNS , 24, 0.0, 0.007980, 0.0, 
Summ,DNS , 24, 0.0, 0.007980, 0.0, 
"avg CPU","avg ET",longest ,date ,time 
uSeconds,uSeconds,uSeconds, , 
332, 1031, 24284,2014/04/08,20:41:49.573730 
Summ,332, 1031, 24284,2014/04/08,20:41:49.573730
49 
SSL statistics 
MV45,SS09,2014/04/10,23:22:24,VRM:800, 
From 2014/04/10,22:53:26.883960 to 2014/04/10,23:22:24.204176 duration 
1737.320215 seconds 
Task,Type,Requests,Busy %, CPU used, CPU %,"avg CPU","avg ET" 
, , , , Seconds, , uSeconds,uSeconds 
0,SSL , 109843, 0.3, 0.594580, 0.0, 5, 42, 
1,SSL , 130180, 0.3, 0.713966, 0.0, 5, 41, 
2,SSL , 117544, 0.3, 0.703146, 0.0, 6, 42, 
3,SSL , 145944, 0.4, 0.830535, 0.0, 6, 43, 
4,SSL , 123825, 0.3, 0.679656, 0.0, 5, 43, 
longest ,date ,time 
uSeconds, , 
229638,2014/04/10,22:54:34.264949 
255082,2014/04/10,22:54:54.302855 
230501,2014/04/10,22:54:43.958105 
280241,2014/04/10,22:54:53.499979 
361212,2014/04/10,22:54:53.599940
Channel accounting data 
• Accounting (ACCTG) trace CLASS(4) enables collection of 
individual channel information 
• Channels can be included/excluded from collection 
– Channel STATCHL attribute for predefined channels 
– Queue manager STATACLS attribute for auto-defined cluster channels 
51 
• Controls all auto-defined channels, not individually 
– Queue manager STATCHL attribute for client channels 
• Generally low cost as most channels are long lived 
• Consider if you have a lot of short lived client connections 
• Gives detailed information for each channel 
– What has the channel done? 
– What are my busy channels?
52 
Channel information (1) 
127.0.0.1 MQ89_1 Connection name 127.0.0.1 
127.0.0.1 MQ89_1 Remote qmgr/app MQ89 
127.0.0.1 MQ89_1 Channel disp PRIVATE 
127.0.0.1 MQ89_1 Channel type SENDER 
127.0.0.1 MQ89_1 Channel status RUNNING 
127.0.0.1 MQ89_1 Channel STATCHL HIGH 
127.0.0.1 MQ89_1 Channel started date & time 2014/04/08,19:41:48 
127.0.0.1 MQ89_1 Channel stopped time 
127.0.0.1 MQ89_1 Channel status collect time 2014/04/08,19:43:57 
127.0.0.1 MQ89_1 Last msg time 2014/04/08,19:43:52 
127.0.0.1 MQ89_1 Active for 122 seconds 
127.0.0.1 MQ89_1 Batch size 50 
127.0.0.1 MQ89_1 Messages/batch 38.9 
127.0.0.1 MQ89_1 Number of messages 2,998 
127.0.0.1 MQ89_1 Number of persistent messages 1,506 
127.0.0.1 MQ89_1 Number of batches 77 
127.0.0.1 MQ89_1 Number of full batches 42 
127.0.0.1 MQ89_1 Number of partial batches 35 
127.0.0.1 MQ89_1 Buffers sent 3,319 
127.0.0.1 MQ89_1 Buffers received 109 
127.0.0.1 MQ89_1 Xmitq empty count 13
Channel information (2) 
127.0.0.1 MQ89_1 Message data 17,198,653 16 MB 
127.0.0.1 MQ89_1 Persistent message data 4,251,780 4 MB 
127.0.0.1 MQ89_1 Non persistent message data 12,946,873 12 MB 
127.0.0.1 MQ89_1 Total bytes sent 17,200,221 16 MB 
127.0.0.1 MQ89_1 Total bytes received 3,052 2 KB 
127.0.0.1 MQ89_1 Bytes received/Batch 39 39 B 
127.0.0.1 MQ89_1 Bytes sent/Batch 223,379 218 KB 
127.0.0.1 MQ89_1 Batches/Second 0 
127.0.0.1 MQ89_1 Bytes received/message 1 1 B 
127.0.0.1 MQ89_1 Bytes sent/message 5,737 5 KB 
127.0.0.1 MQ89_1 Bytes received/second 25 25 B/sec 
127.0.0.1 MQ89_1 Bytes sent/second 140,985 137 KB/sec 
127.0.0.1 MQ89_1 Compression rate 0 
127.0.0.1 MQ89_1 Exit time average 0 uSec 
127.0.0.1 MQ89_1 DNS resolution time 0 uSec 
127.0.0.1 MQ89_1 Net time average 312 uSec 
127.0.0.1 MQ89_1 Net time min 43 uSec 
127.0.0.1 MQ89_1 Net time max 4,998 uSec 
127.0.0.1 MQ89_1 Net time max date&time 2014/04/08,19:43:52 
53
55 
Channel accounting SMF 116 sub-type 10 
• For each channel instance 
– Channel name 
– Channel disposition 
– Channel type 
– Channel state 
– STATCHL setting 
– Connection name 
– Date/time of last channel status changed 
– Last message date/time 
– Channel batch size 
– Num of messages 
– Num of persistent messages 
– Num of batches 
– Num of full batches 
– Num of transmission buffers sent 
– Num of transmission buffers received 
– Current shared conversations 
– Num of bytes 
– Num of persistent bytes 
– Number of bytes sent (both control data 
and message data) 
– Number of bytes received (both control 
data and message data) 
– Compression rate 
– Exit time average 
– Exit time min 
– Exit time max 
– Exit time max date/time 
– Net time average 
– Net time min 
– Net time max 
– Net time max date/time 
– Remote queue manager/application name 
– Put retry count 
– Transmission queue empty count
Questions? 
© 2014 IBM Corporation 56
For Additional Information 
IBM Training 
http://www.ibm.com/training 
IBM WebSphere 
http://www.ibm.com/software/websphere/ 
http://www.ibm.com/software/products/ibm-mq 
IBM developerWorks 
http://www.ibm.com/developerworks/websphere 
https://www.ibm.com/developerworks/community/blogs/messaging
IBM MQ Sessions this week 
10:30 - 12:00 13:15 - 14:15 14:30 - 15:30 16:00 - 17:00 17:15 - 18:15 
Tuesday 
Opening General Session- IBM Digital 
Experience and WebSphere Technical 
University 
Session A31: IBM 
MQ CHLAUTH rules 
– with MQ V8 
updates 
Speaker: Morag 
Hughson 
Room 02 
Session A4: WebSphere 
MQ for z/OS: 
Performance and 
Accounting 
Speaker: Alex Ross 
Room 8 
Session I26: 
DataPower-MQ 
Connectivity Deep Dive 
(Theory) 
Speaker: Robin Wiley 
Room 27 
Session Z1: WebSphere 
MQ for z/OS V8: Latest 
Features Deep Dive 
Speaker: Damon Cross 
Room 6 
9:00 - 10:00 10:30 - 11:30 11:45 - 12:45 14:00 - 15:00 15:15 - 16:15 16:45 - 17:45 
Wednesday 
Session Z5: 
WebSphere MQ for 
z/OS: Security 
Speaker: Damon 
Cross 
Room 02 
Session A21: What's 
New in IBM 
Messaging 
Speaker: Morag 
Hughson 
Room 8 
Session C7: 
Messaging in the 
Cloud with IBM MQ 
Light and IBM 
Bluemix 
Speaker: Rob 
Nicholson 
Room 27 
Session A17: Managing 
work-loads, scaling and 
availability with IBM MQ 
clusters 
Speaker: David Ware 
Room 6 
Lab IL5: DataPower-MQ Connectivity Deep Dive 
(Hands-On) 
Speaker: Robin Wiley 
Room 7b 
Session A9: WebSphere 
MQ for z/OS: The Inside 
Story 
Speaker: Damon Cross 
Room 6 
Thursday 
Session A35: How to 
Develop Responsive 
Applications with IBM 
MQ Light 
Speaker: Rob 
Nicholson 
Room 27 
Session A22: New 
IBM MQ V8 Security 
Features 
Speaker: Morag 
Hughson 
Room 01 
Session A3: WebSphere 
MQ for z/OS: Shared 
Queues 
Speaker: Alex Ross 
Room 6 
Session A18: Using 
Publish /Subscribe with 
IBM MQ 
Speaker: David Ware 
Room 27 
Friday 
Lab AL6: Developing a First Application 
with IBM WebSphere MQ Light 
Speakers: Robert Nicholson, Alex Ross 
Room 7b 
Session A16: Using 
IBM MQ Pub/Sub in 
an MQ network 
Speaker: David Ware 
Room 6

More Related Content

PPTX
PDF
IBM Impact 2014 AMC-1877: IBM WebSphere MQ for z/OS: Performance & Accounting
PDF
DHCP Server - pfSense Hangout September 2016
PDF
pfSense 2.3 Preview - pfSense Hangout December 2015
PPTX
PDF
Traffic Shaping Basics with PRIQ - pfSense Hangout February 2016
PDF
Advanced Captive Portal - pfSense Hangout June 2017
PDF
Connectivity Troubleshooting - pfSense Hangout June 2016
IBM Impact 2014 AMC-1877: IBM WebSphere MQ for z/OS: Performance & Accounting
DHCP Server - pfSense Hangout September 2016
pfSense 2.3 Preview - pfSense Hangout December 2015
Traffic Shaping Basics with PRIQ - pfSense Hangout February 2016
Advanced Captive Portal - pfSense Hangout June 2017
Connectivity Troubleshooting - pfSense Hangout June 2016

What's hot (9)

PDF
Server Load Balancing on pfSense 2.4 - pfSense Hangout July 2017
PPT
Module vi
PDF
Multi-WAN on pfSense 2.3 - pfSense Hangout March 2016
PDF
Dynamic Routing with FRR - pfSense Hangout December 2017
PDF
High Availability on pfSense 2.4 - pfSense Hangout March 2017
PPT
16 control unit
PPT
PDF
HBase Coprocessors @ HUG NYC
PPT
03 buses
Server Load Balancing on pfSense 2.4 - pfSense Hangout July 2017
Module vi
Multi-WAN on pfSense 2.3 - pfSense Hangout March 2016
Dynamic Routing with FRR - pfSense Hangout December 2017
High Availability on pfSense 2.4 - pfSense Hangout March 2017
16 control unit
HBase Coprocessors @ HUG NYC
03 buses
Ad

Viewers also liked (19)

PPTX
La contaminación ambiental
PDF
Ecuaciones diferenciales
PPTX
Фотографија, настанак и развој, 4 разред ОШ
PDF
Esclarecer o-habitus
PPTX
8 "vitamine" per far ritrovare la giusta energia al tuo team
PDF
Evo slides fabio_lo_savio
PPTX
England and Ireland Group Project
PPTX
Presentation1 (3)
PPTX
Presentation on Unjust Enrichment
PPTX
Reimaging the Library in the Wake of Technology Anne-Marie Eggleston
PPTX
Website genre
PPTX
Hidrolisis pati ubi kayu menjadi sirup glukosa
PPTX
Sistem reproduksi pada manusia
PPTX
È una questione di fiducia...
PDF
Index IEEE Photonics Journal volume 7 2015
PPTX
Recuerdos de vacaciones
PPTX
Tom mise en scene
PPTX
Font Research
PPTX
What is affiliate marketing and how does it work
La contaminación ambiental
Ecuaciones diferenciales
Фотографија, настанак и развој, 4 разред ОШ
Esclarecer o-habitus
8 "vitamine" per far ritrovare la giusta energia al tuo team
Evo slides fabio_lo_savio
England and Ireland Group Project
Presentation1 (3)
Presentation on Unjust Enrichment
Reimaging the Library in the Wake of Technology Anne-Marie Eggleston
Website genre
Hidrolisis pati ubi kayu menjadi sirup glukosa
Sistem reproduksi pada manusia
È una questione di fiducia...
Index IEEE Photonics Journal volume 7 2015
Recuerdos de vacaciones
Tom mise en scene
Font Research
What is affiliate marketing and how does it work
Ad

Similar to Wtu 2014 ibm web sphere mq for zos - performance and accounting (20)

PDF
IBM Performance and Accounting
PPTX
What's new with MQ on z/OS 9.3 and 9.3.1
ODP
IBM WebSphere MQ for z/OS - The Inside Story
PPTX
Where is my MQ message on z/OS?
PDF
3450 - Writing and optimising applications for performance in a hybrid messag...
PPTX
Hhm 3470 mq v8 and more recent new things for z os
PDF
IBM MQ - better application performance
PDF
3452 - Managing your applications
PDF
HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applicati...
PDF
Websphere MQ (MQSeries) fundamentals
PDF
IBM MQ - Comparing Distributed and z/OS platforms
PDF
IBM MQ - Comparing Distributed and z/OS platforms
PPTX
MQTC 2016: Monitoring and Tracking MQ and Applications
PDF
IBM MQ Appliance - Administration simplified
PDF
Lukasz Lipski - Monitoring Payment Queues
PDF
MQ What's New Beyond V8 - V8003 level
PPTX
Operational and business monitoring with IBM Integration Bus-Sanjay Nagchowdhury
PDF
Ibm tivoli omegamon xe for web sphere mq using ibm tivoli omegamon xe for w...
ODP
Building highly available architectures with WAS and MQ
PDF
What's new in IBM MQ Messaging
IBM Performance and Accounting
What's new with MQ on z/OS 9.3 and 9.3.1
IBM WebSphere MQ for z/OS - The Inside Story
Where is my MQ message on z/OS?
3450 - Writing and optimising applications for performance in a hybrid messag...
Hhm 3470 mq v8 and more recent new things for z os
IBM MQ - better application performance
3452 - Managing your applications
HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applicati...
Websphere MQ (MQSeries) fundamentals
IBM MQ - Comparing Distributed and z/OS platforms
IBM MQ - Comparing Distributed and z/OS platforms
MQTC 2016: Monitoring and Tracking MQ and Applications
IBM MQ Appliance - Administration simplified
Lukasz Lipski - Monitoring Payment Queues
MQ What's New Beyond V8 - V8003 level
Operational and business monitoring with IBM Integration Bus-Sanjay Nagchowdhury
Ibm tivoli omegamon xe for web sphere mq using ibm tivoli omegamon xe for w...
Building highly available architectures with WAS and MQ
What's new in IBM MQ Messaging

Recently uploaded (20)

PDF
Cost to Outsource Software Development in 2025
PDF
Types of Token_ From Utility to Security.pdf
PPTX
assetexplorer- product-overview - presentation
PDF
How Tridens DevSecOps Ensures Compliance, Security, and Agility
PPTX
Trending Python Topics for Data Visualization in 2025
PPTX
Cybersecurity: Protecting the Digital World
PDF
AI/ML Infra Meetup | LLM Agents and Implementation Challenges
PDF
Multiverse AI Review 2025: Access All TOP AI Model-Versions!
PPTX
Introduction to Windows Operating System
PDF
How AI/LLM recommend to you ? GDG meetup 16 Aug by Fariman Guliev
PDF
Wondershare Recoverit Full Crack New Version (Latest 2025)
PPTX
Computer Software and OS of computer science of grade 11.pptx
PPTX
Advanced SystemCare Ultimate Crack + Portable (2025)
PPTX
Oracle Fusion HCM Cloud Demo for Beginners
PPTX
AMADEUS TRAVEL AGENT SOFTWARE | AMADEUS TICKETING SYSTEM
DOCX
How to Use SharePoint as an ISO-Compliant Document Management System
PDF
Microsoft Office 365 Crack Download Free
PDF
Website Design Services for Small Businesses.pdf
PPTX
Weekly report ppt - harsh dattuprasad patel.pptx
PPTX
WiFi Honeypot Detecscfddssdffsedfseztor.pptx
Cost to Outsource Software Development in 2025
Types of Token_ From Utility to Security.pdf
assetexplorer- product-overview - presentation
How Tridens DevSecOps Ensures Compliance, Security, and Agility
Trending Python Topics for Data Visualization in 2025
Cybersecurity: Protecting the Digital World
AI/ML Infra Meetup | LLM Agents and Implementation Challenges
Multiverse AI Review 2025: Access All TOP AI Model-Versions!
Introduction to Windows Operating System
How AI/LLM recommend to you ? GDG meetup 16 Aug by Fariman Guliev
Wondershare Recoverit Full Crack New Version (Latest 2025)
Computer Software and OS of computer science of grade 11.pptx
Advanced SystemCare Ultimate Crack + Portable (2025)
Oracle Fusion HCM Cloud Demo for Beginners
AMADEUS TRAVEL AGENT SOFTWARE | AMADEUS TICKETING SYSTEM
How to Use SharePoint as an ISO-Compliant Document Management System
Microsoft Office 365 Crack Download Free
Website Design Services for Small Businesses.pdf
Weekly report ppt - harsh dattuprasad patel.pptx
WiFi Honeypot Detecscfddssdffsedfseztor.pptx

Wtu 2014 ibm web sphere mq for zos - performance and accounting

  • 1. IBM WebSphere MQ for z/OS Performance and Accounting Alexander Ross (alexross@uk.ibm.com) Software Developer
  • 2. Please Note IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. © 2014 IBM Corporation 2
  • 3. 3 Agenda • Introduction • Queue manager accounting and statistics • Channel initiator accounting and statistics
  • 4. 5 Introduction • What are you trying to achieve? • What you are using? • Is your system efficient? • Being able to answer these questions allows you to: – Administer effectively – Prevent outages and degradations in service – Plan for growth or change – Save MIPS and money!
  • 5. 7 Monitoring • WebSphere MQ provides various monitoring capabilities: – DISPLAY/PCF commands, MQ Explorer – PCF event messages • Queue manager – Start/stop, GET/PUT inhibited, unknown object • Configuration – Define/Alter/Delete/Refresh objects • Command – Commands issued and by who • Performance – Queue depth high/low/full • Channel/bridge – Start/stop, SSL – Queue manager performance statistics • High-level view of activity – Queue manager accounting data • Detailed information on MQI operations • Provides capability to charge users for queue manager usage – Channel statistics and accounting
  • 6. Collecting statistics and accounting data • System Management Facility (SMF) records – Event messages on distributed • Statistics – SMF record type 115 – START TRACE(STAT) – SMFSTAT system parameter (zPARM) • Accounting – SMF record type 116 – START TRACE(ACCTG) – SMFACCT system parameter (zPARM) • Collection interval can be specified using the STATIME system parameter – If zero, the SMF global accounting interval is used 9
  • 7. 10 Collecting statistics and accounting data N O T E S • On distributed platforms enabling statistics and accounting information generates PCF messages, as per events. On z/OS, this information is generated as System Management Facility (SMF) records instead. • Statistics information is recorded as SMF 115 records. • Accounting data is recorded as SMF 116 records. • The START TRACE command can be used to start collecting this information. A parameter can be provided to specify which type of information should be collected. • To enable the collection of this information automatically system parameters can be set in the zPARM using CSQ6SYSP. When set, data collection will commence at queue manager start up. • SMF data is collected in memory and written out at periodic intervals. The interval MQ should use can be set using the STATIME system parameter. If this is set to zero WebSphere MQ uses the SMF global accounting interval.
  • 8. Interpreting SMF data • SMF record layouts fully documented • SupportPac MP1B – MQSMF displays formatted records – Outputs information to various files (DDs) – Highlights potential out-of-line conditions – Can also output in comma-separated value (CSV) format for importing 11 in spreadsheets – MQCMD also available • Automates issuing of DISPLAY commands - outputs to CSV • Useful for monitoring a given channel or queue
  • 9. 13 Queue manager statistics • Provides a lot of activity related information – Insignificant overhead so can enable this all the time – Keep historical data for trend analysis • Easy to see high-level activity – How much storage the queue manager is using – How many API calls are occurring – How much logging is occurring – How the buffer pools are performing • Problem diagnosis – What is using CPU? – What is causing delays?
  • 10. MVS QM Date Time Puts Put1s Gets MVSA MQ1A 31/03/2014 00:00:00 53979 54120 306587 MVSA MQ1A 31/03/2014 00:30:00 27212 42914 227198 15 Message Manager statistics • Provides counts of MQI requests during SMF interval • Includes both successful and failed requests – MQGET count includes those where no message was available • Written to MSGM and MSGMCSV DD by MQSMF • Do you know what your daily/monthly profile looks like? • Do you have busy hours each day, or busy days each month? – What is the trend? MVSA MQ1A 31/03/2014 01:00:00 23924 31470 169326
  • 11. Do you know what your daily profile looks like? 16 1600000 1400000 1200000 1000000 800000 600000 400000 200000 0 00:00:00 01:30:00 03:00:00 04:30:00 06:00:00 07:30:00 09:00:00 10:30:00 12:00:00 13:30:00 15:00:00 16:30:00 18:00:00 19:30:00 21:00:00 22:30:00 API Calls per 30mins MQPUT MQPUT1 MQGET • The above chart shows potential MQGET inefficiencies – Multiple application instances ‘competing’ to get the messages? – Need to index one or more queues? • Can help to identify growing trends and the need for increased capacity
  • 12. Log Manager statistics • Constraint for persistent messages is normally the logging rate • All persistent data changes are logged – Mainly MQPUTs and MQGETs of persistent messages • Also object changes, etc. • Primary reporting is on Control Intervals (CIs) – 4KB pages – If you’re using dual logging 2 CIs are written per 4KB page • Written to LOG and LOGCSV DD by MQSMF • Ask the following questions: – Do you have I/O issues? – Are you checkpointing more than you should? – Are you backing out more than you should? 17
  • 13. Logging overview • Application writes data in to log buffers – May have to wait if no log buffers are available – Log force requests the log write task to write up to the required RBA 19 • Waits for I/O to complete • Log write task – Waits for previous I/O to complete – Looks for highest requested log RBA to write – Writes up to and including this RBA, then iterates • One log write can include data from many tasks • More data written per I/O, the longer the I/O time • As more data is logged by the system an individual transaction’s commit time may increase
  • 14. DASD information • Writing more data per I/O is more efficient than many short I/Os – An I/O has set-up time and data transfer time (connect) – e.g. • Set up 0.1 millisecond, connect 0.1 per 4KB page • One page per I/O = 0.1 + 0.1 = 0.2 ms = 5000/second = 5000 pages/sec • 9 pages per I/O = 0.1 + 0.9 = 1.0 ms = 1000/second = 9000 pages/sec – Each larger I/O takes longer but the overall data rate is greater • Can stripe the log data set across different volumes – Pages 1,5,9 to vol 1; 2,6 to vol 2; 3,7 to vol 3; 4,8 to vol 4 – Can now perform I/O in parallel - can write 3 pages faster than 9! • Expect 140MB/second per log 20
  • 15. Logging statistics Write_Wait 0, Write_Nowait 538306, Write_Force 193 Read_Stor 0, Read_Active 0, Read_Archive 0 BSDS_Reqs 703, CIs_Created 61037, BFWR 7478 ALW 6, CIs_Offload 60480, Checkpoints 0 Read delayed 0, Tape Lookahead 0, Lookahead Mount 0 Write_Susp 7430, Write_Reqs 12007, CI_Writes 64629 Write_Serl 0, Write_Thrsh 48, Buff_Pagein 0 WTB 0 TVC 0 ALR 0 _____,__ write requests, CIs, Average I/O , After I/O , pages/IO 21 uSec , uSec , Log 1, 1 page 7500, 7500, 312, 5, 1 Log 1,>1 page 4507, 57129, 617, 6, 13 Standard deviation of first log, 1 page per I/O, response time +- 3 Log 1, 1 page Longest I/O 3914 at 2014/04/10,08:55:57.429543 UTC Log 1, 1 page Longest Request 3918 at 2014/04/10,08:55:57.429543 UTC Log 1,>1 page Longest I/O 34352 at 2014/04/10,08:55:57.392493 UTC Log 1,>1 page Longest Request 34361 at 2014/04/10,08:55:57.392493 UTC Log write rate 2MB/s per copy Logger busy: Interval: 121 Idle: 115206421 (us)( 95%) LogIOSusp: 5193950 (us)( 4%)
  • 16. Logging statistics •WTB > 0 indicates applications having to wait for buffers to be written first – OUTBUFF might need to be increased – Alternatively, DASD might be slow – MQSMF highlights potential warning conditions: 22 • MQQJST09W QJST requests waiting for buffer 99 • MQQJST11W QJST OK logging rate 65 MB/Sec • Log read when transactions back out – Log reads also expected during queue manager start-up – Data read from buffers – good – Data read from active logs – OK – Data read from archive logs – BAD – investigate long UOW – MQSMF messages: • MQQJST00I QJST read log buffers from storage 3433 > 0 • MQQJST01W QJST read log buffers from active logs 50838 > 0 • MQQJST02S QJST read log buffers from archive logs 15 > 0
  • 17. Logging statistics •Checkpoints is only incremented when LOGLOAD causes checkpoint – Checkpoints due to a log switch are not counted •CI_Writes shows 64629 4KB pages have been written – 2.1MB/sec average because SMF interval is 2 minutes = VERY LOW! • Singular logging because only Log 1 is reported • Look at single page I/O time – Should be fairly stable – see standard deviation • Multiple page I/O sizes vary – 312 uSec in example is very good • 1000 uSec is OK – if 2000 or higher then attention is warranted • Check average number of pages per I/O – If 50 or higher this indicates I/O is constrained by DASD speed or trying to 23 write too much data • Consider splitting the workload over an additional queue manager • Longest page I/O took 3.9ms – Useful if investigating performance issues, disk issues?
  • 18. Buffer Manager statistics • Are the buffer pools performing well? – All private queue messages that are put or got require buffers in a buffer pool irrespective of persistence – A badly configured buffer pool may have a significant impact on performance or waste storage – Configuration of buffer pools is dependent on how they are going to be used • Short-lived messages – Keep whole working set of messages in buffer pool • Long-lived messages – More messages than can be contained in memory – Will always hit disk before being got • Written to BUFF and BUFFCSV DD by MQSMF 24
  • 19. = BPool 1, Size 5000,%full now 80, Highest %full 92, Disk reads 27019 > 01 Buffs 5000 Low 390 Now 979 Getp 1408819 Getn 137508 01 Rio 27019 STW 648745 TPW 26587 WIO 2417 IMW 12 01 DWT 1236 DMC 0 STL 72300 STLA 8 SOS 0 01 Above the bar PAGECLAS 4KB 26 Buffer pool statistics • No buffer available (SOS), buffer steal (STLA) and DMC should be 0 – If not, consider increasing buffer pool size • Asynchronous write threshold (DWT) reached 1236 times – Number of 'dirty' pages >= 85% total pages • Synchronous write threshold (DMC) not reached above – Number of 'dirty' pages >= 95% total pages • Version 8 reports if the buffer pool is above the bar and whether it is page-fixed •MQSMF messages output if buffer pools may be too small: – MQQPST04E BP 3 Many (735) pages read from disk – MQQPST02S BP 3 Filled many (122) times
  • 20. 27 Buffer pools for short-lived messages • Aim to keep the full working set of messages in memory – DWT should be 0 • Always at least 15% buffers free • Pages written (TPW) might be greater than 0 due to checkpoint activity – However, we shouldn’t be writing because of buffer pool getting full! • Read operations (RIO) should be 0 – If we have everything in memory, we should never need to read from the pageset! • Unless read from a page set after the queue manager is restarted • You are recommended to have sufficient buffers to handle your peak message rate
  • 21. Buffer pools for long-lived messages • Expect messages to be written to and read from the page set • (RIO+TPW)/statistics interval is the I/O rate to page sets • During period that messages are processed (e.g. written during day and processed overnight) – Number of read I/Os (RIO) should be approximately the total number of pages written (TPW). • Shows that one page is read for every page written. • If RIO is much larger than TPW this means pages are being read multiple times. – This might indicate applications getting by message or correlation id 28 on a queue that is not indexed
  • 22. Accounting data • Understand application activity • Can control at queue level using ACCTQ(ON|OFF|QMGR) – QMGR means inherit from ACCTQ queue manager attribute • Costs 5-10% CPU overhead • Heavyweight – multiple records may be cut for each transaction, and at SMF intervals for long running units of work – Get a lot of useful information about what is being done! – However, enabling this has been known to swamp an SMF 29 environment • Task/thread and queue statistics • Even though prolific: – Turn on for a few minutes each month to get view of activity – Become familiar with the data and the usage patterns
  • 23. Application activity Open name AP01_REP_MQ20_DC_0_0007 Object type:Local Queue Base name AP01_REP_MQ20_DC_0_0007 Base type :Queue Queue indexed by NONE First opened 10-04-2014 08:52:48.62 Last closed 10-04-2014 08:52:49.69 Page set ID 4, Buffer pool 1 Current opens 0, Total requests 158 Generated messages : 0 Persistent messages: GETs 37, PUTs 0, PUT1s 0 Put to waiting getter: PUT 0, PUT1 0 Put direct to shared queue: 0 GETs: Valid 78, Max size 9824, Min size 75, Total bytes 151197 GETs: Dest-S 156, Dest-G 0, Brow-S 0, Brow-G 0, Successful destructive 78 Time on queue : Max 0.348850, Min 0.118915, Avg 0.231072 seconds -MQ call- N ET CT Susp LOGW PSET Epages skip expire Open : 1 20 20 0 Close : 1 6 6 0 Get : 156 43 38 0 0 0 0 570 0 31 Maximum depth encountered 154
  • 24. 33 A good day versus a bad day • Obtain data for a good day to compare with – Without this it is much harder to identify the cause of problems Metric Good day Bad day Get count 16846 3365 Get elapsed time (average) 329 2735 Get CPU time (average) 321 966 Get suspend time 0 1735 Get pageset count 0 6752 Get pageset elapsed time 0 1734 Get valid count 6087 3233 Get size (maximum) 1043924 1041564 Get size (minimum) 115196 202600 Get size (average) 847087 834928
  • 25. Channel initiator statistics and accounting data • Version 8 introduces statistics and accounting data for: – Channel initiator address space – Channel activity • This information can be used for: – Monitoring – Capacity planning – Tuning • Previously, many customers have created their own ‘monitoring’ jobs which issue periodic DISPLAY CHSTATUS – Using this information it is difficult to: 35 • Manage historical data • Perform capacity planning • Investigate channel performance issues • Separate controls from queue manager allows 'opt in' • Updated MP1B SupportPac will format the data and introduces rule based reporting
  • 26. 36 Channel initiator SMF controls • You can start channel initiator statistics (STAT) trace by: !MQ08 START TRACE(STAT) CLASS(4) CSQW130I !MQ08 'STAT' TRACE STARTED, ASSIGNED TRACE NUMBER 05 CSQ9022I !MQ08 CSQWVCM1 ' START TRACE' NORMAL COMPLETION • You can start channel initiator accounting (ACCTG) trace by: !MQ08 START TRACE(ACCTG) CLASS(4) CSQW130I !MQ08 ‘ACCTG' TRACE STARTED, ASSIGNED TRACE NUMBER 06 CSQ9022I !MQ08 CSQWVCM1 ' START TRACE' NORMAL COMPLETION • You can display trace by: !MQ08 DISPLAY TRACE(*) CSQW127I !MQ08 CURRENT TRACE ACTIVITY IS - TNO TYPE CLASS DEST USERID RMID 02 STAT 01 SMF * * 05 STAT 04 SMF * * 06 ACCTG 04 SMF * * END OF TRACE REPORT •ALTER and STOP TRACE commands have also been updated
  • 27. Channel initiator SMF controls • Channel initiator SMF data is collected at the same interval as the queue manager • Channel initiator statistics (STAT) trace – Provides a high level view of activity in the CHINIT address space • Channel initiator accounting (ACCTG) trace – Provides a detailed view of individual channels – STATCHL channel attribute controls granularity 37 • Equivalent to ACCTQ for queues • Data collected is a superset of that collected in the PCF messages on distributed
  • 28. 38 Channel statistics versus accounting data • Statistics – Information about number of channels and TCB usage – Dispatchers, Adapters, DNS, SSL – Do I need more or less dispatchers? – Do I need more or less adapters? – Do I have spare capacity? • Accounting – What have the channels done? – What are the high use channels?
  • 29. Channel initiator statistics SMF 115 sub-type 231 • Channel initiator – QSG name – Number of current channels – Maximum current channels – Number of active channels – Maximum active channels – Maximum TCP/IP channels – Maximum LU 6.2 channels – Storage usage in MB • Dispatcher task – Task number (TCB address) – Number of requests for task – Busy CPU time of task – Sum of elapsed time of requests – Wait elapsed time of task • Adapter task – Task number (TCB address) – Number of requests for task – Busy CPU time of task – Sum of elapsed time of requests – Wait elapsed time of task 39 • DNS task – Task number (TCB address) – Number of requests for task – Busy CPU time of task – Sum of elapsed time of requests – Wait elapsed time of task – Time of day of max DNS request – Duration time of max DNS request • SSL task – Task number (TCB address) – Number of requests for task – Busy CPU time of task – Sum of elapsed time of requests – Wait elapsed time of task – Time of day of max SSL request – Duration of max SSL request
  • 30. 41 Channel initiator statistics summary MV45,MQ20,2014/04/08,20:43:57,VRM:800, From 2014/04/08,20:41:54.984681 to 2014/04/08,20:43:57.237939 duration 122.253258 seconds Number of current channels..................... 20 Number of active channels .... ................ 20 MAXCHL. Max allowed current channels........... 602 ACTCHL. Max allowed active channels............ 602 TCPCHL. Max allowed TCP/IP channels............ 602 LU62CHL. Max allowed LU62 channels............. 602 Storage used by Chinit......................... 22MB
  • 31. Channel initiator statistics summary • Number of current and active channels – How close are you getting to the maximums? • Channel initiator storage usage – 31-bit usage - not much currently in 64-bit for the channel initiator • Are these trending upwards? – Monitor over time 42 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 3 0 0 0 2 5 0 0 2 0 0 0 1 5 0 0 1 0 0 0 5 0 0 0 N u m b e r o f c u r r e n t c h a n n e l s 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 5 0 0 4 5 0 4 0 0 3 5 0 3 0 0 2 5 0 2 0 0 1 5 0 1 0 0 5 0 0 S T G u s e d i n M B
  • 32. Dispatcher statistics MV45,MQ20,2014/04/08,20:43:57,VRM:800, From 2014/04/08,20:41:54.984681 to 2014/04/08,20:43:57.237939 duration 122.253258 seconds Task,Type,Requests,Busy %, CPU used, CPU %,"avg CPU","avg ET" , , , , Seconds, , uSeconds,uSeconds 0,DISP, 46, 0.0, 0.000000, 0.0, 12, 14 1,DISP, 168912, 1.4, 0.028218, 0.0, 8, 10 2,DISP, 168656, 1.3, 0.025142, 0.0, 7, 10 3,DISP, 0, 0.0, 0.000000, 0.0, 0, 0 4,DISP, 0, 0.0, 0.000000, 0.0, 0, 0 Summ,DISP, 337614, 0.6, 0.053360, 0.0, 9, 10 43 0,DISP, number of channels on this TCB, 0 1,DISP, number of channels on this TCB, 10 2,DISP, number of channels on this TCB, 10 3,DISP, number of channels on this TCB, 0 4,DISP, number of channels on this TCB, 0 Summ,DISP, number of channels on all TCBs, 20
  • 33. Adapter statistics MV45,MQ20,2014/04/08,20:43:57,VRM:800, From 2014/04/08,20:41:54.984681 to 2014/04/08,20:43:57.237939 duration 122.253258 seconds Task,Type,Requests,Busy %, CPU used, CPU %,"avg CPU","avg ET" , , , , Seconds, , uSeconds,uSeconds 0,ADAP, 127599, 16.5, 0.953615, 0.8, 7, 158 1,ADAP, 46790, 7.6, 0.309678, 0.3, 7, 199 2,ADAP, 13702, 3.2, 0.065380, 0.1, 5, 284 3,ADAP, 2909, 0.7, 0.029541, 0.0, 10, 279 4,ADAP, 395, 0.1, 0.003179, 0.0, 8, 392 5,ADAP, 37, 0.0, 0.000241, 0.0, 7, 149 6,ADAP, 10, 0.0, 0.000175, 0.0, 17, 111 7,ADAP, 0, 0.0, 0.000000, 0.0, 0, 0 Summ,ADAP, 191442, 3.5, 1.361809, 0.1, 7, 179 45
  • 34. 47 DNS statistics MV45,MQ20,2014/04/08,20:41:54,VRM:800, From 2014/04/08,20:40:07.101220 to 2014/04/08,20:41:54.984681 duration 107.883460 seconds Task,Type,Requests,Busy %, CPU used, CPU %, , , , , Seconds, , 0,DNS , 24, 0.0, 0.007980, 0.0, Summ,DNS , 24, 0.0, 0.007980, 0.0, "avg CPU","avg ET",longest ,date ,time uSeconds,uSeconds,uSeconds, , 332, 1031, 24284,2014/04/08,20:41:49.573730 Summ,332, 1031, 24284,2014/04/08,20:41:49.573730
  • 35. 49 SSL statistics MV45,SS09,2014/04/10,23:22:24,VRM:800, From 2014/04/10,22:53:26.883960 to 2014/04/10,23:22:24.204176 duration 1737.320215 seconds Task,Type,Requests,Busy %, CPU used, CPU %,"avg CPU","avg ET" , , , , Seconds, , uSeconds,uSeconds 0,SSL , 109843, 0.3, 0.594580, 0.0, 5, 42, 1,SSL , 130180, 0.3, 0.713966, 0.0, 5, 41, 2,SSL , 117544, 0.3, 0.703146, 0.0, 6, 42, 3,SSL , 145944, 0.4, 0.830535, 0.0, 6, 43, 4,SSL , 123825, 0.3, 0.679656, 0.0, 5, 43, longest ,date ,time uSeconds, , 229638,2014/04/10,22:54:34.264949 255082,2014/04/10,22:54:54.302855 230501,2014/04/10,22:54:43.958105 280241,2014/04/10,22:54:53.499979 361212,2014/04/10,22:54:53.599940
  • 36. Channel accounting data • Accounting (ACCTG) trace CLASS(4) enables collection of individual channel information • Channels can be included/excluded from collection – Channel STATCHL attribute for predefined channels – Queue manager STATACLS attribute for auto-defined cluster channels 51 • Controls all auto-defined channels, not individually – Queue manager STATCHL attribute for client channels • Generally low cost as most channels are long lived • Consider if you have a lot of short lived client connections • Gives detailed information for each channel – What has the channel done? – What are my busy channels?
  • 37. 52 Channel information (1) 127.0.0.1 MQ89_1 Connection name 127.0.0.1 127.0.0.1 MQ89_1 Remote qmgr/app MQ89 127.0.0.1 MQ89_1 Channel disp PRIVATE 127.0.0.1 MQ89_1 Channel type SENDER 127.0.0.1 MQ89_1 Channel status RUNNING 127.0.0.1 MQ89_1 Channel STATCHL HIGH 127.0.0.1 MQ89_1 Channel started date & time 2014/04/08,19:41:48 127.0.0.1 MQ89_1 Channel stopped time 127.0.0.1 MQ89_1 Channel status collect time 2014/04/08,19:43:57 127.0.0.1 MQ89_1 Last msg time 2014/04/08,19:43:52 127.0.0.1 MQ89_1 Active for 122 seconds 127.0.0.1 MQ89_1 Batch size 50 127.0.0.1 MQ89_1 Messages/batch 38.9 127.0.0.1 MQ89_1 Number of messages 2,998 127.0.0.1 MQ89_1 Number of persistent messages 1,506 127.0.0.1 MQ89_1 Number of batches 77 127.0.0.1 MQ89_1 Number of full batches 42 127.0.0.1 MQ89_1 Number of partial batches 35 127.0.0.1 MQ89_1 Buffers sent 3,319 127.0.0.1 MQ89_1 Buffers received 109 127.0.0.1 MQ89_1 Xmitq empty count 13
  • 38. Channel information (2) 127.0.0.1 MQ89_1 Message data 17,198,653 16 MB 127.0.0.1 MQ89_1 Persistent message data 4,251,780 4 MB 127.0.0.1 MQ89_1 Non persistent message data 12,946,873 12 MB 127.0.0.1 MQ89_1 Total bytes sent 17,200,221 16 MB 127.0.0.1 MQ89_1 Total bytes received 3,052 2 KB 127.0.0.1 MQ89_1 Bytes received/Batch 39 39 B 127.0.0.1 MQ89_1 Bytes sent/Batch 223,379 218 KB 127.0.0.1 MQ89_1 Batches/Second 0 127.0.0.1 MQ89_1 Bytes received/message 1 1 B 127.0.0.1 MQ89_1 Bytes sent/message 5,737 5 KB 127.0.0.1 MQ89_1 Bytes received/second 25 25 B/sec 127.0.0.1 MQ89_1 Bytes sent/second 140,985 137 KB/sec 127.0.0.1 MQ89_1 Compression rate 0 127.0.0.1 MQ89_1 Exit time average 0 uSec 127.0.0.1 MQ89_1 DNS resolution time 0 uSec 127.0.0.1 MQ89_1 Net time average 312 uSec 127.0.0.1 MQ89_1 Net time min 43 uSec 127.0.0.1 MQ89_1 Net time max 4,998 uSec 127.0.0.1 MQ89_1 Net time max date&time 2014/04/08,19:43:52 53
  • 39. 55 Channel accounting SMF 116 sub-type 10 • For each channel instance – Channel name – Channel disposition – Channel type – Channel state – STATCHL setting – Connection name – Date/time of last channel status changed – Last message date/time – Channel batch size – Num of messages – Num of persistent messages – Num of batches – Num of full batches – Num of transmission buffers sent – Num of transmission buffers received – Current shared conversations – Num of bytes – Num of persistent bytes – Number of bytes sent (both control data and message data) – Number of bytes received (both control data and message data) – Compression rate – Exit time average – Exit time min – Exit time max – Exit time max date/time – Net time average – Net time min – Net time max – Net time max date/time – Remote queue manager/application name – Put retry count – Transmission queue empty count
  • 40. Questions? © 2014 IBM Corporation 56
  • 41. For Additional Information IBM Training http://www.ibm.com/training IBM WebSphere http://www.ibm.com/software/websphere/ http://www.ibm.com/software/products/ibm-mq IBM developerWorks http://www.ibm.com/developerworks/websphere https://www.ibm.com/developerworks/community/blogs/messaging
  • 42. IBM MQ Sessions this week 10:30 - 12:00 13:15 - 14:15 14:30 - 15:30 16:00 - 17:00 17:15 - 18:15 Tuesday Opening General Session- IBM Digital Experience and WebSphere Technical University Session A31: IBM MQ CHLAUTH rules – with MQ V8 updates Speaker: Morag Hughson Room 02 Session A4: WebSphere MQ for z/OS: Performance and Accounting Speaker: Alex Ross Room 8 Session I26: DataPower-MQ Connectivity Deep Dive (Theory) Speaker: Robin Wiley Room 27 Session Z1: WebSphere MQ for z/OS V8: Latest Features Deep Dive Speaker: Damon Cross Room 6 9:00 - 10:00 10:30 - 11:30 11:45 - 12:45 14:00 - 15:00 15:15 - 16:15 16:45 - 17:45 Wednesday Session Z5: WebSphere MQ for z/OS: Security Speaker: Damon Cross Room 02 Session A21: What's New in IBM Messaging Speaker: Morag Hughson Room 8 Session C7: Messaging in the Cloud with IBM MQ Light and IBM Bluemix Speaker: Rob Nicholson Room 27 Session A17: Managing work-loads, scaling and availability with IBM MQ clusters Speaker: David Ware Room 6 Lab IL5: DataPower-MQ Connectivity Deep Dive (Hands-On) Speaker: Robin Wiley Room 7b Session A9: WebSphere MQ for z/OS: The Inside Story Speaker: Damon Cross Room 6 Thursday Session A35: How to Develop Responsive Applications with IBM MQ Light Speaker: Rob Nicholson Room 27 Session A22: New IBM MQ V8 Security Features Speaker: Morag Hughson Room 01 Session A3: WebSphere MQ for z/OS: Shared Queues Speaker: Alex Ross Room 6 Session A18: Using Publish /Subscribe with IBM MQ Speaker: David Ware Room 27 Friday Lab AL6: Developing a First Application with IBM WebSphere MQ Light Speakers: Robert Nicholson, Alex Ross Room 7b Session A16: Using IBM MQ Pub/Sub in an MQ network Speaker: David Ware Room 6