Ds8000 Practical Performance Analysis P04 20060718

Editor's Notes

• #2: The purpose of this presentation is to help storage administrators by providing a high level over view of the DS8000 performance characteristics, storage performance analysis processes, and storage tools.
• #4: The Goal of this Presentation is to provide some practical tips for storage administrators Author intro: End to End Performance Support for Managed Storage Service – 17 DS8000 Systems as of 5/25 Over 125 ESS Model 800 Over 2 Petabytes of Managed Storage Proactive, Reactive support for all customers
• #5: DS8000 not likely to have Backend DA saturation as looped SSA architecture is gone! Backend access is 2 GB/sec switched fibre! The back end DA saturation on the 200MB fiber will happen at rates not attainable by 40MB SSA. Since it is easier to generate disk load than it is to create faster disks, there will still be contention. Disk contention on arrays can and will exist. Port contention can still occur LPAR isolation is limited to model 9A2 New cache algorithm (SARC) is faster but perceived performance improvements will vary depending on actual workload. SARC provides up to a 25% improvement of the I/O response time in comparison to the LRU algorithm. It provides some batch/online isolation and is a platform for more isolation. %SARC improvement is more a description of the talent of the person who ran the measurements than it is a description of the product. People want a number, 25% is okay. I'm glad to have 256GB.
• #7: Various tests were ran with differing block sizes, and same number of I/Os, disk size, disk speed and workload 70/30 50% Read Hit For any given number of I/Os per second, the block size only seems to affect the utilization of the host adapters and the I/O bus. It doesn't appear to affect the utilization of the disk or the processors A mixed size workload (4k, 8k, 16k, 32k,) will perform exactly the same as a fixed size workload of, say all 16k blocks for the same number of IO/s and the same storage configuration. Don't worry about a mixture of large and small block sizes; worry about mixing large and small block sizes. Overall I/O throughput (MB/s) seems to be limited by the number of transactions, not the block size (up to a point!)
• #8: Various tests were ran with the same block size and differing number of I/Os. Disk size, disk speed and same workload 70/30 50% Read Hit Increasing the number of I/Os per second with a fixed block size does impact the utilization of disk together with SMP utilization Side note…other tests showed: Cache hit service time = .5 ms Disk size affects capacity not necessarily performance! Per Siebo, I did 18-36 GB comparisons on DS4000 for various workloads. There is not a difference due to disk size. If they seek the same, spin the same, and transfer the same, you get the same response for any I/O rate.
• #9: Gather Storage Server performance and configuration data Gather SAN fabric configuration and exception data If port saturation then contact SAN design team Analyze storage server configuration and performance data If DS8000 issue exists then recommend corrective actions Why do we use I/O response time? On most systems with virtualized storage with multiple paths the disk utilization numbers are misleading. You might have a device that shows 100% busy but it could have excellent response time. The device is not actually 100% busy because it is not really a device but a path to a logical storage unit located on multiple devices. The most telling type of I/O metric is the I/O response time. The I/O response time should be viewed in conjunction with activity rates to provide an understanding of the impact to the response time on the overall performance. There is a need to limit analysis to only the disks used by the application having a problem. LUN Serial Numbers can be used to correlate the Storage Server performance data with the server physical device information.
• #10: Different people have different ways of identifying the ripest fruit to harvest. The key is to make sure that the resource you identify and eventually take action upon is the resource the one that has the best balance between providing the biggest performance benefit and minimizing the impact of the change
• #11: PDCU Requires Root (AIX, Linux) or Administrator (Windows) privileges are not required for installation or execution of PDCU Note: File sizes are large (40-60MB) Windows note: Do not install PDCU into a path that contains the character sequence "pd“ (a common choice!). Open a Tech Support Request to engage a specialist http://dalnotes1.sl.dfw.ibm.com/atss/techxpress.nsf/request?OpenForm There are ways to gather performance data about I/Os issued to Volumes residing on external storage subsystems from a server perspective. MVS probably has the most comprehensive set of measurements available. Distributed systems have information also and it varies from platform to platform.
• #12: With utilizations > 50% the response time starts to increase noticeably Port Weirdness…You can attach 4 channels(ports) to a Host Adapter. With one channel, you can get 200 MB/second; 2 channels, 400 MB/sec; 3 Channels 540 MB /sec, and 4 channels 540 MB/sec The response times are not completely arbitrary. For Ports, the write response time should be < 1 ms as it should be write hits 100% of time. Another measurement that is available for ports is the population. Busy = Population / (1+Population) For volume response time, it is my experience that the customers will not typically complain if the avg response time is < 6 ms for volume I/Os…this includes cache hits/cache misses. If the avg response times are > 6 ms it typically indicates contention on the backend at DA or DDM. Typically DDM for high I/Os small size transfer & DA if transfer size is > 256 KB For array response times > 6 ms is where you see the ddm utilization start to go beyond 50 % although 10ms is a more reasonable number.
• #13: In this case the read response times for several of the ports were > 6 ms…this could be an issue where 1 or more servers zoned to the hba is accessing a volume or set of volumes that are on a constrained array During this period there was no corresponding spikes in read i/o rate or throughput. Drill down to volume to see if there is any correlation
• #14: There appears to be correlation in the average read response time for all volumes and the average read response time for ports The next step is to drill down to the volumes that had high avg read response times during the point 7 – 16 on the x-axis
• #15: Put the Volume Statistics in Excel Table Create a pivot with Volumes in the row fields and time in the column field (Limit time to desired range) Place Average Read Response Time in Data Field! Create an average in the grand total field Copy volume ID to a new analysis table along with avg response times
• #16: In the pivot table add the summary of read i/o rate and copy to analysis table Create a Total I/O Response Time field (=sum (Avg Read RT * sum Read I/O Rate) + sum(Avg Write RT* SUM(Write I/O Rate). In this example I did not include the writes as they were neglible. I like to create a % Total colum which is the Total I/O RT for an individual Volume / the sum of the total response time for all Volumes. This provides an I/O metric that gives a sense of the overall impact and is a good metric to sort on.
• #17: In order to issue the CLI commands you must have installed the DSCLI component on a system with connectivity to the DS8000 management console (HMC) and you must have a userid with read privileges The column headers provide a sample of the output. In this case we are looking to ID the extpool (extent pool) id of the hot volumes previously identified and use the extent pool ID to identify the arrays in the lsarray output Drop the first character of the Rank ID and convert the rank to hex to match the output of ‘lsarray’ with that gathered in SDD output.
• #18: Now you that you have identified the arrays in question, look at the array response time and see if there is any correlation between the array response time and the volume response time. In this case there is a slight correlation between the spike in vol read response time and the response time on the array In reality you should look at the other metrics available for the array including I/O rates, transfer sizes, etc.
• #19: TPC 3.1.1 – The current GA version has several views that may be useful in the bottom up approach. Graph the array utilization for the subsystem you are interested in… By hovering over the spikes in the array utilization you can determine the time of the spikes. By referencing the legend, you can identify the array with the spike. In this case, the spikes were on array 18 at: 2:35pm, 4:18pm, 7:05pm, 11:32pm, 1:14 am,5:22 am,12:24 pm. By going back to the list of arrays you can drill down on the array to see the volumes stored on the array The first view is the Array view. This allows you to historically chart an array metric for up to 10 arrays within a single view. You can export the data to CSV. You can view additional arrays by selecting Next You can also modify the time window by checking the Limit days check box and specifying a start and stop time The array provides information on the lowest level of the disk storage subsystem (the physical disks) and can be very useful in PD. A list of metrics can be found in the following publication: IBM TotalStorage Productivity Center User’s Guide GC32-1775
• #20: After you drill down from the array to the volumes you are provided a list of volumes. At this point you can either chart the volume performance data (10 per chart) or export the data.
• #21: Requires that the CLI component gets installed either locally on the TPC server or remotely If remote, then access will need to be set up and you will need a userid/password. The userid/pwd you have to access the GUI works In order to gather stats for more than 1 component you need to issue the command multiple times
• #22: Confirm that the issue is NOT with server resources Verify that host CPU utilization, Paging I/O, and local HBA saturation are not source of performance issues Identify any host disks with high I/O response time Map the host disk to Storage Server device name Gather Storage Server performance and configuration data Gather SAN fabric configuration and exception data If port saturation then contact SAN design team Analyze storage server configuration and performance data If DS8000 issue exists then recommend corrective actions Why do we use I/O response time? On most systems with virtualized storage with multiple paths the disk utilization numbers are misleading. You might have a device that shows 100% busy but it could have excellent response time. The device is not actually 100% busy because it is not really a device but a path to a logical storage unit located on multiple devices. The most telling type of I/O metric is the I/O response time. LUN Serial Numbers can be used to correlate the Storage Server performance data with the server physical device information.
• #23: The AIX filemon tool is a trace based facility and should only be ran for a couple of minutes at a time. The other UNIX flavors provide I/O response time data that can be gathered continuously at reasonable intervals as they are not trace based (See Appendix C for other flavors). The read size is always in 512 byte blocks chunks. So in this case there were 620 reads. The avg read size was 8 Blocks (512 Byte blocks) or 4096 Bytes (4 KB) chunks. These are random I/Os as the number of read sequences is the same as the number of I/Os. The minimal information that you need to pull from this is: Time When filemon started, Volume, Reads, Avg Read Time, Writes, Avg Write Time. I would filter out any records that have 1 I/O or less. For the LUN >hdisk mapping, run the SDD command ‘datapath query essmap’ command. Minimally you will want to pull the Hostname and the hdisk information on a daily basis if you have access to the servers or install an agent that ftp’s the information to somewhere where you can load it in a configuration database. After the data has been formatted, sort by the highest average response time. It is helpful to create a pivot table and average the I/O response times for each of the LUNs and create a sorted list of the LUNs with the highest response time.
• #24: The first view sorts and sums the total Response Time (RT) for all I/Os associated with each LSS. In this case I/Os to volumes on LSS 7,10,8,9 & 12 make up 81% of the total I/O RT. These are the ones you should drill down on. I/Os are evenly distributed across the volumes on Rank ‘ffff’ – By sorting on different DS8000 components you can drill down to different physical levels. This is a good method to find saturated components when the components are consistently saturated. Another potentially beneficial view is that of the hot Ranks/Volumes over time. This can show the spikes that would not otherwise get observed in the summary tables.
• #25: In addition to the host Connection information, the output of the ‘datapath query essmap’ command contains the Rank and LSS. While there is no fixed association between LSS’s and physical placement on the DS8000, it is helpful in knowing that LUNs reside on different Server (Even numbered LSS’s belong to server0 and odd to server1) and on different ranks. Knowing the rank-> LUN association can help in determining which LUNs should be chosen for striped lvs or spread lvs.
• #26: The LUN ID provided in SDD is 11 Chars. The first 7 are the SN of the DS8000 image and the last 4 chars are the volume ID The first 2 columns of the ‘lsfbvol’ command are Name ID – The ID will correlate to the volume ID. There is also an Extent Pool Column The name is used in TPC for Data to identify the Volume The volume ID is used in PDCU and the Extent Pool can be used to identify the rank – using the Extent Pool as a key
• #28: VG = Volume Group Connection = The shark hba location Device Name = OS device LCU = Logical Control Unit ChPID = Channel Path ID =~ HBA or Path
• #29: Generally speaking the I/O response time is the amount of time it takes from the point where the I/O request hits the device driver until the I/O is returned from the device driver The iostat package for Linux is only valid with a 2.4 & 2.6 kernel