DB2 Workload Manager Histograms

Editor's Notes

• #2: Two of the most common applications of monitoring are performance tuning and problem determination. In these slides, we will see how histograms and other statistics can help with tuning exercises and how monitoring individual activities (e.g. queries) helps with problem determination.
• #3: This is a big topic, so I am going to limit this talk to the new functionality in DB2 for LUW 9.5 rather than cover the various monitoring features DB2 has gained over the years since its introduction. I am not going to talk about low-level details like bufferpool hit ratios, but high-level statistics like response time and throughput information. We aren't going to look at individual queries and analyze them with explains, but we will look at the system as a whole and see what steps can be taken when overall system performance is being impacted.
• #4: There are 3 types of new monitoring features. This talk is focused on histograms.
• #5: I'm going to start by talking about the distributions and I will start with the most famous one. The Bell Curve, or Gaussian, or Normal distribution has a special property: the mean, the median, and the mode all occur at the same spot on the curve, right in the middle.
• #6: Let's say you measure height. You have a sample of five men and they report their heights as follows: 5'10'', 5'8'', 5'10'', 6'0'', 5'10''. The arithmetic mean of those heights is the sum of the 5 heights divided by 5, or 5'10''. The median of those heights is the number that occurs in the middle when we order the heights from smallest to largest or largest to smallest. The mode is the most frequently occurring height. In a bell curve, the mode, median, and mean are all the same number. If you take a random sample of, say, 1000 people, the histogram of their heights would take the shape of a bell curve.
• #7: Another type of distribution is called the Power Law distribution, or long-tailed distribution. It has a very different set of properties than a bell curve. The smallest measurements are enormously more frequent than the larger ones. Also, the frequency of seeing a very large measurement doesn't fall off quickly towards zero as it would on a bell curve – it keeps going, which is where it gets the name “the long tail”.
• #8: Clay Shirky, in his book “Here Comes Everybody”, imagined what it would be like if height followed a power law instead of a bell curve and he sampled the heights of two hundred men.
• #10: We saw that, with a bell curve, the mean of 5'10'' was a good representative height – most of us don't stray too far from that. However, when the distribution of heights in the population is a power law, the mean of 5'10'' isn't very representative any more. Why am I talking about power laws? I am talking about power laws because almost everything having to do with database workloads, such as the distribution of response times, or the times between arrivals of queries into a database system, takes the form of a power law, or something very close to it.
• #11: If averages aren't useful, what do we do? We need to see the whole distribution, which only a histogram can provide.
• #12: To draw a histogram, you draw a horizontal axis and you divide it up into class intervals. That's your measurement axis. For example, height would go along this axis. You could break in off into: 4 to just under 5 feet, 5 to just under 6 feet, 6 to just under 7 feet, etc. Then you draw your vertical axis and this measures the number of observations. We had 5 observations before.
• #13: Four were between 5 and just under 6 feet, so we would draw a box whose height is four over the 5 to just under 6 feet class interval. One was between 6 and just under 7 feet so we would draw a box with a height of 1 over the 6 to just under 7 feet class interval.
• #14: Let's do another one. Here is some data on query response times that we'll turn into a histogram. They're sorted from smallest to largest.
• #15: On the right, we now have a table of class intervals that will fit all of the response times on the left. We just need to count the observations. How many queries on the left fell into the first bin [0 to 1]?
• #16: That's right: One.
• #17: How many fell into the range (1 to 2]? Four.
• #18: And (2 to 3]? Three.
• #19: And we keep going until we've finished building the histogram. That's the histogram in table form, but we can better look as a graph.
• #20: This is very little data, so it doesn't take the form of any particular curve. We'll come back to the importance of having enough data later.
• #21: I think we understand what a histogram is, so what is it good for? I'll talk about each of these advantages in detail in the next few slides.
• #22: Let's look at a scenario to illustrate why seeing the distribution is useful. Say we have a database and during the first month in which we examine it, all the users are happy with its performance. We look at both the average response time (19 seconds) and the standard deviation (6 seconds), and we assume that if we continue to get these numbers, users will be happy, but we are wrong.
• #23: Now we look at month 2. Users are complaining? What changed? Average response times are the same as month 1 aren’t they?
• #24: Going back to month 1, we see the histogram is a bell curve centered at 19 seconds
• #25: Now we see the problem. The average response time for half the users went up over 30% - they’re the ones complaining. It improved just as much for the other half of the users. That's why it is useful to be able to see the distribution.
• #26: Another reason for having histograms is to detect outliers. In his book, The Black Swan, Nassim Nicholas Taleb wrote about rare events that have a big, relevant impact. He called these rare events black swans because, up until the 17 th Century, everyone knew that all swans were white. But then black swans were discovered in Australia and a perceived impossibility had come to pass.
• #27: Taleb distinguished between two kinds of outliers. The first kind has a small, almost negligible impact. For example, say you have a group of 1000 people and you measure their weights and you compute an average. And then you add an elephant, the heaviest land animal on earth, to the group. It changes the average by less than 10%.
• #28: The other type of outlier can have a significant effect on the mean, in effect, making it useless as a measurement. Say you have the same thousand people and instead of measuring their weights, you measure their net worths. Then, instead of adding the heaviest land animal on earth, you add one of the richest men on earth. If their mean net worth had been, say $200,000, adding Bill Gates to the group changes that mean to $80 million – that's an increase of over 40,000%.
• #29: We can look at this in the form of a histogram. On the left, we see the net worths of the thousand people. Then we see a large number of empty bins.
• #30: Then we see Bill Gate's net worth on the far right. Take a look at that horizontal axis – that's an exponential scale. To get a useful idea of what a representative net worth is, we want exclude Bill Gates from the calculation and the histogram let's us do that. Another alternative might be to compute the median rather than the arithmetic mean. The problem is, unlike a running average, where we only need to keep track of the average and the count, for the median, we need to keep every data point and this becomes a problem when you're, say, trying to track the median of query response time when you have hundreds of thousands of queries per day, sometimes thousands per second. A histogram gives you the ability to identify and potentially filter out outliers without the performance and storage penalty of keeping around the median.
• #31: A final advantage of histograms is that they are easy to aggregate together, much like averages are. So you can keep a histogram of the daily response times of your database activities and then, once a month, aggregate these histograms into one histogram for the month.
• #32: I mentioned that averages are easy to calculate, but you still have to be careful. In his book Innumeracy, John Allen Paulos gives an example of a common misconception in combining averages that gives rise to an apparent paradox involving batting averages.
• #35: Paulos then proceeds to show us how this can occur. He says that Ruth bats .300 in the first half compared to Gehrig's .290, then Ruth bats .400 in the second half compared to Gehrig's .390. The key is that Ruth has 200 at-bats in the first half compared to Gehrig's 100 and Gehrig has 200 at-bats in the second half compared to Ruth's 100.
• #36: So we multiply the average by the at-bats and we get 60 for the first half for Ruth.
• #37: And we get 40 for the second half for Ruth.
• #38: We get 29 and 78 for Gehrig.
• #39: Ruth sums to 100.
• #40: Gehrig sums to 107 for the same number of at-bats as Ruth, giving Gehrig the higher batting average.
• #41: So, combining averages is fairly easy, you just have to remember to multiply by the count. But if you want to somehow combine anything more sophisticated than the average, such as standard deviation, you're out of luck. However, you can combine histograms and it is even easier than combining averages – you just add the observation counts. There is no multiplication necessary. Say you have a histogram of database query response times for January. It is a bell curve with an average between 5 and 6 seconds.
• #42: Then in February, you have a response time histogram with an average between 15 and 16, also a bell curve, but it is a taller one since there were more queries executed in February than January. If you combine them, you get the Jan + Feb column and adding these is easy since there is hardly any overlap, so in most cases, you are just adding a zero to the number from January or a zero to the number from February.
• #43: Then you add March. March is also a bell curve, with its average somewhere between 10 and 11 seconds. The combined histogram for the first quarter is now definitely not a bell curve, but it was easy to put together.
• #44: In support of the Workload Manager work in DB2 9.5, we added six histograms. Five are related to activities. Activities are general concept we use in DB2 Workload Manager to talk about queries, loads, and DDL. I am going to speak in more detail on the next few slides about what each of these activity histograms measure. Also, one histogram exists for requests. Requests are the OPEN, FETCH, CLOSE actions that make up a cursor activity, or may not be part of an activity at all, such as PREPARE request. They may also not be externalized. There are requests from one partition to another in systems that have the data partitioning feature.
• #45: Let's talk about the lifecycle of an activity. Here is an activity that illustrates the first three measurements that are made into histograms: execution time, queue time, and lifetime. This activity is a cursor activity: it gets opened, it fetches twice, and is closed. In between each fetch, there is a delay while DB2 waits for the client to submit a new request, such as the next FETCH request or the CLOSE request. This activity is also queued by the Workload Manager. We measure the queue time as the time between when the activity arrived into the system and when it comes off the queue to execute. In this case, the queue time was 20 seconds. The execution time is the time DB2 spends executing the activity, but not any idle time spent waiting for the next request from the client. So execution is just those four gray colored 3 second blocks for a total of 12 seconds. The lifetime is the total time an activity spends in the system, including the queue time, including the execution time, and including the idle time. Since the total idle time was also 12 seconds (4 + 4 + 4), the lifetime is 44 seconds (20 seconds of queue time + 12 of idle time + 12 of execution time). So those are the first 3 measurements for which we have histograms.
• #46: The fourth measurement is something called inter-arrival time. This is the time between the arrival of one activity into the system and the arrival of the next. It is the inverse of the arrival rate. In this example, we have three activities.
• #47: If we say activity 1 arrives at time 0 and activity 2 arrives at time 3, the inter-arrival time between activity 1 and activity 2 is 3. And between activity 2 and activity 3, the inter-arrival time is 12. So we have three activities arriving in a 15 second period. That's an average arrival rate of 0.2 activities per second, which converts to an average inter-arrival time of 5 seconds.
• #48: The last of the measurements for activity histograms is a measurement of estimated cost. This is the cost that the SQL compiler computes whenever you compile a DML activity and by DML I mean SELECT, INSERT, UPDATE, DELETE and the variations on that. This cost is a weighted combination of CPU and I/O resources that DB2 can tell from the access plan and assumptions about the cardinality of the tables that the activity is going to consume. As such, it is highly dependent on table and index statistics being up-to-date. Let's assume you have a query with negligible I/O demands and on a single CPU might look like the diagram. Here we have a query with a cost of 7. Costs are generally in units of timerons, but let's assume that we have somehow converted timerons into estimated timeslices on the CPU. The blue vertical bars represent a timeslice. Query 1 demands 7 timeslices and it is the only thing running on the system so its execution time is 7, indicated by the E=7 to the right of the query. We're assuming this isn't a cursor activity, so there is no idle time. The lifetime is exactly the same as its execution time, which is 7. This is indicated by the L=7. The queue time is zero, so we Q=0 to the right of the query as well. What do you think will happened when our query has to share the one CPU with another query.
• #49: What happens is its lifetime gets longer because it has to share the CPU with query 2. Query 2 suffers in the same way. The CPU will round robin between the two queries. When one them finishes (in this case, query 1 finishes first) the other query will get the CPU to itself. Notice that the lifetime and execution time of query 1 increased from 7 on the previous slide up to 12.
• #50: If we add another query , query 1's lifetime and execution time jump up to 16...
• #51: and then to 18 if we add a fourth query...
• #52: ... and then to 20 if we add a fifth. Query 1 now almost takes 3 times as long to finish as it did original when it finished in 7 seconds.
• #53: Let's go back to two queries at the same time. Let's say that instead of trying to run both queries are the same time, we use Workload Manager to limit the concurrency to 1 and then we queue anyone else that comes in. Can you guess what happens to query 1? To query 2.
• #54: Query 1 finishes in 7 timeslices because it arrived first and doesn't get queued. Then query 2 arrives while query 1 is still running. It has to queue until query 1 finishes which is 6 timeslices later. Then it gets the CPU to itself where it takes 9 timeslices to finish, for a total of 15. Notice that the sum of the queue time and execution time is the lifetime. I hope this explains the idealized concept of estimated cost and why you can't just assume estimated cost will tell you exactly how long your query will run without taking into account sharing of resources such as CPU resources.
• #55: I want to talk now about how much data you need to see the underlying distribution of your data. Here we have estimated cost histograms for two workloads that are identical in terms of their distribution. The individual costs are pseudo-random numbers . A pseudo-random number is generated by a pseudo-random number generator. A pseudo-random number generator takes a “seed” and generates one pseudo-random number after another. If you ask a pseudo-random number generator to generate a second set of numbers but you give it the same seed, you get the same set of numbers. If you give a pseudo-random number generator a different seed, you get a different set of pseudo-random numbers but these numbers are still distributed the same way no matter what the seed is. In the histogram I used here, I could have used a bell curve for the distribution, but to show another example of a distribution, I used an exponential distribution. The first thing you should notice is that these histograms look nothing alike. That's because you have too little data – only 10 activities. Let's see what happens when you have 50 activities...
• #56: A little better. We're starting to converge on a shape, but they're still quite different. Let's see what 250 activities can do...
• #57: The histograms are almost identical now. Additionally, both are starting to look like an idealized exponential distribution. This gives us insight into how we can tell when we have enough data to make good comparisons of one histogram with another. Here is a trick for ensuring you have enough data to do such comparisons: Say you have two periods of time that you want to compare. Say the first time period is between 2:00 and 3:00 PM on a Thursday afternoon. Then you take the following hour, between 3:00 and 4:00 PM and you measure histograms during each of these time periods and you compare them. If you get something like the first of the three examples, you double the size of the interval and you compare that to the next double-size interval. In our example, you would compare a 2:00 to 4:00 PM interval with a 4:00 to 6:00 PM interval. If they still don't match closely, double again. Eventually, the curves will either start to look similar as in the third example or else their distributions have changed in some way.
• #58: That's enough theory. Let's look at how we can use histograms in the DB2 for LUW Workload Manager. There are three steps to collecting histograms (and other aggregates like high watermarks and counters). First, alter the service class or work action set that you want to collect histograms on and specify the COLLECT AGGREGATE ACTIVITY DATA option. Use the BASE option to get the three basic histograms.
• #59: Use the EXTENDED option to get the original 3 histograms plus the inter-arrival and estimated cost histograms.
• #60: Finally, to collect the request histogram, specify the COLLECT AGGREGATE REQUEST DATA BASE clause. Note that there is no EXTENDED option.
• #61: The second step is to create an event monitor to write out the histograms and you need to set its state to 1 to activate it.
• #62: Finally, you need to say when you want a collection to happen. You can set it up to automatically collect periodically using the WLM_COLLECT_INT database configuration parameter. Here we have it collecting once a day. Otherwise, you can have it collect immediately using the WLM_COLLECT_STATS stored procedure.
• #63: When you turn on collection, activated your event monitor and triggered a collection, a table or file having this structure is what you get. You can get it as a table or a file depending on the type of event monitor you create. Note that this is a histogram bin table rather than a histogram table. Each row in this table is a bin in a histogram.
• #64: The name after HISTOGRAMBIN is the name of the event monitor that you created in step 2.
• #65: Let's look at the columns in this table: You can uniquely identify the histogram that this bin belongs to by looking at the combination of service_class_id, statistics_timestamp and histogram_type, assuming you set your service class up as generating this histogram.
• #66: If, on the other hand, the histogram belongs to a work action set, you uniquely identify it by using histogram_type and statistics_timestamp with a work_action_set_id and a work_class_id. The histogram_type column has values like “CoordActLifetime” for the activity lifetime histogram or “CoordActQueueTime” for the activity queue time histogram.
• #67: So now that you know how to uniquely identify a histogram, you know how to write a WHERE clause for queries against this table, but what should you return in your output? It turns out you only need two columns: TOP and NUMBER_IN_BIN. You don't necessarily need the BOTTOM column because it is just the TOP of the previous bin. The NUMBER_IN_BIN is just the number of observations.
• #68: Let's try to visualize a histogram by querying the histogram bin table. You select the two columns from the previous slide, top and number_in_bin.
• #69: Let's say we want a service classes histogram so we specify the service_class_id, say, 13. You can look up these numbers in the syscat.serviceclasses table or do a join with that table.
• #70: Say we want to look at query lifetimes and...
• #71: ...say we did a select distinct statistics_timestamp and we got back a timestamp of 2:30 PM on March 12, 2009.
• #72: Let's order it by the TOP column in ascending order.
• #73: This gives us an idea of the histogram shape, but it could be more visual.
• #74: Let's refactor this code a little bit first. We'll move the query into a WITH to make the main select simpler because we're going to complicate it a bit.
• #75: Let's replace number_in_bin with the proportion of the peak number_in_bin.
• #76: Now we multiply by the number of columns that we want a bar in our bar graph at most to take up, in this case, 60 columns.
• #77: Here's the fun part: this idea comes from a blog post by by Bernd Kuennen (http://154pm.blogspot.com/2008/01/generating-daily-usage-statistics.html)
• #78: And that gives us a histogram graph.
• #79: Here’s a better picture. Notice that, after the bin with a top of 29, everything else is empty. Only the first 8 bins have any data in them and there are 41 bins altogether. That’s a whole lot of empty bins. That is because the default is to fit 6 hours into the first 40 bins and the longest we ran was 29 milliseconds or under. As a result, every bin between 29 milliseconds and 6 hours is empty.
• #80: We can provide a tighter bound than 6 hours by modifying what is called the “template” of the histogram. We do this by creating a histogram template in the database with a new largest bin value to use instead of 6 hours. In this case, I chose 44 milliseconds. Then we tell the service class to use the template by altering it and specifying our template as the lifetime histogram template for that service class. Then we need to collect stats once more to make the change take effect.
• #81: You may have noticed that the bins aren’t all the same size. This is deliberate and I will talk about that in a bit, but one of the consequences that concerns us now is that when you reduce the top of the highest bin with something tiny like 44 milliseconds, several of your smallest bins now have TOP values that are “one point something” and since they are all integers they get rounded down to 1. So you get multiple bins with the same TOP. Only the first of those bins has any data put in it. It’s not pretty to see all those duplicates, so you can add a GROUP BY on the TOP column and put an aggregating function on the NUMBER_IN_BIN to make it look better.
• #82: Here we’ve added an aggregating function and a GROUP BY and we now see the histogram with a highest bin top of 44. Now almost every bin contains data. It can be useful to fit the histogram to your data to get better precision as we’ve done here, but remember to leave some room to cover any surprises.
• #83: If you don’t leave room, the surprises end up in what is called the “catch-all” bin. This bin is the one marked with a TOP of -1. A non-zero value in this bin alerts you that you need to modify your histogram template to have a higher top bin value. For help in choosing a good top bin value, you can use the COORD_ACT_LIFETIME_TOP high watermark for lifetime histograms and it should work fairly well for execution time histograms too. For estimated cost histograms, you can use the COST_ESTIMATE_TOP high watermark.
• #84: DB2 histogram bins are not all the same size – they increase exponentially This is because of the nature of the data The smallest activities finish in milliseconds, the longest in many hours Imagine if all bins were the same size High bin value is 6 hours
• #85: 6 hours / 40 bins = 9 minutes per bin The smallest queries had response times in tens of milliseconds The histogram we visualized earlier would have all data in the first bin and nothing in the other bins
• #86: A service level agreement (SLA) formally defines the level of service in a service contract It often contains performance metrics such as throughput or response time requirements Sometimes, it is specified as just an average Other times, it is expressed as “X% of activities must finish with Y minutes” e.g. “95% of activities must finish within 25 seconds”. Remember this example SLA since we’ll come back to it later.
• #87: We can use the activity lifetime histogram to easily see if we are meeting our SLA. First we convert NUMBER_IN_BIN to a percentage of the total number in the histogram. Then we sum the counts in each bin with those of all the bins that came before it. That will give us a cumulative percentage.
• #88: Let’s go back to the query we used earlier and modify it to allow us to measure against an SLA.
• #89: Here we are scaling the NUMBER_IN_BIN into a PERCENTAGE_IN_BIN…
• #90: Then we add another column called CUMULATIVE_PERCENTAGE and we’ll filter out the “catch-all” bin.
• #91: Here’s what we get. Let’s see if activities are meeting our SLA. If you recall, our SLA was 95% of activities finish within 25 seconds.
• #92: We see in the text highlighted in blue that only 53.99% of activities are finishing within 25.157 seconds rather than 95%, so these activities are not meeting their SLA.
• #93: If you found this talk interesting, there are three articles that I wrote on developerWorks on which I based these slides. If you remember anything about this talk, I hope that you remember that the distribution of your data is important, or else you’d see 200 ft giants walking around and most of us would be 1 foot tall. I hope you remember that when Bill Gates joins a crowd, he drastically changes the crowd’s net worth, but a histogram let’s us get back to the real data. Finally, I hope you remember that histograms are as easy to combine together as the batting averages of two baseball greats.