![(a) Workflow of framework construction.
(b) Workflow of framework evaluation.
Figure 1: Framework construction and evaluation. The symbol o denotes an observable and the symbol t denotes
a tunable.
2.2 Machine Learning Algorithms applied to the RBF functions, and an output layer which
Many different machine learning algorithms can be used produces the linear combination of the outputs from the
to generate the prediction model. We selected a large set of RBF functions.
algorithms and evaluated their performance over our train-
ing set. In this section, we summarize the algorithms we 3. APPLICATION TO HARDWARE PRE-
considered. FETCHERS
We evaluated several classical machine learning algorithms.
We apply the framework introduced previously to find the
The nearest neighbor (NN) [1] algorithm measures the sim-
best configuration of hardware prefetchers for a given appli-
ilarity of each data instance using Euclidean distance and
cation. In this section, we define the tunables and observ-
classifies a new data instance according to its nearest neigh-
ables used and explain how to build the training dataset.
bor in the training dataset. The na¨ Bayes (NB) [12] method
ıve
computes the probability that a given data instance belongs 3.1 Platform
to each class and classifies a new data instance according to Table 1 lists the major components of our experimental
the class with the highest probability. A tree-based classi- platform. One computing server consists of 2 or 4 sockets.
fier takes a divide-and-conquer strategy. The C4.5 decision Each socket is an Intel Core 2 Quad CPU. Core 2 Quad
tree (DT) [18] classifier uses a divide-and-conquer strategy. comprises 4 CPU cores, equipped with a shared L2 cache
It partitions the training dataset based on selecting pivots for every two cores. Each CPU core has a 32KB L1 data
that result in the greatest information gain. The partition- cache and a 32KB L1 instruction cache. Each of the two
ing is then recursively applied to each sub-partition. Alter- shared L2 caches is 4MB. All the programs were compiled
natively, a rule-based classifier uses a separate-and-conquer by GCC 4.3.1 with -O2 flag. We monitor execution and
technique. The Ripper classifier (RP) [4] greedily attempts collect hardware events with perfmon2 [16].
to construct a rule to classify as many instances as possible
within a class. It then separates out the instances that were 3.2 Tunables: Hardware Prefetchers
classified and continues on with those still unclassified.
On an Intel Core 2 CPU, there are four hardware prefetch-
We evaluated many modern machine learning algorithms
ers [10]:
as well. Support vector machines (SVM) [17] treat the in-
put data as vectors in an N-dimensional space and find a 1. Data Prefetch Logic (DPL) prefetcher: When the pro-
maximum-margin hyperplane to separate the vectors into cessor detects stride access, this prefetcher fetches
two classes, one on each side of the hyperplane. Logistic re- streams of instructions and data from memory to the
gression (LR) [19] can predict a categorical type and capture unified L2 cache.
the non-linear relationship between the input features and
2. Adjacent Cache Line (ACL) prefetcher: This prefetcher
output predictions. We also evaluated two types of neural
fetches the matching cache line in a cache line pair (128
networks. One is the typical multi-layer perceptron (MLP) [8]
bytes) to the L2 unified cache.
network with backward propagation. The other is the radial
basis function (RBF) [2] network which consists of three lay- 3. Data Cache Unit (DCU) prefetcher: This prefetcher
ers: an input layer, a hidden layer where the inputs are fetches the next cache line into the L1 data cache](https://image.slidesharecdn.com/machine-learningbased-prefetch-optimization-for-data-center1281/85/Machine-Learning-Based-Prefetch-Optimization-for-Data-Center-3-320.jpg)
![CPU Model Intel Core 2 Quad ample, if an application runs on a machine with all prefetch-
L1 cache (Data 256KB (4 * (32KB + 32KB)) ers disabled, a high cache miss rate may be a good indicator
and Instruc- that some of the prefetchers should be turned on to bring
tion) data into cache earlier and reduce the cache miss penalty.
L2 cache 8MB (2 * 4MB) However, in an environment where all prefetchers are en-
Main Memory 16 GB abled, a high cache miss rate suggests that the some of the
Operating Sys- Linux (Kernel version: 2.6.18) prefetchers should be turned off, because the cache misses
tem are likely caused by a too aggressive prefetcher setting. A
Compiler GCC 4.3.1 (Flags: -O2) high cache miss rate may also just be indicative of poor spa-
Performance perfmon2 tial locality of the application. We only highlight some of
Monitoring the difficulties interpreting cache miss rate. Similar analysis
Software can be done for the rest of the selected hardware events.
The relationship between these observables and perfor-
mance also depends on the particular processor. Different
Table 1: Details of the platform.
architectures have different performance models, and rela-
tionships derived on one architecture do not necessarily fol-
low on another architecture. The general difficulty estab-
when it detects multiple loads from the same cache lishing a relationship between observed application behav-
line within a fixed period of time. ior and optimal prefetcher configuration drives us towards
machine learning.
4. Instruction Pointer-based (IP) prefetcher: This
prefetcher keeps sequential load history information 3.4 Data Collection
per instruction pointer and uses this data to decide Before applying our application tuning framework, we
whether to fetch data for an upcoming delinquent load must address two issues: what is the training dataset and
instruction into the L1 data cache. what is the data format. The ideal training dataset should
Each processor core has one DCU prefetcher and one IP be as representative and large as possible. Currently, we
prefetcher for its L1 cache and also one DPL prefetcher and use profiles (i.e., hardware events) collected from running
one ACL prefetcher for the shared L2 cache. Each hardware all programs in the SPEC CPU2006 [22] benchmark, with
prefetcher can be enabled or disabled independently, so there both train and ref input data for each program, to build
are 16 prefetcher configurations for each core. For the re- our prediction models. We believe including other represen-
mainder of this paper, we encode the configurations with tative benchmarks such as SPECweb2009 [23] or TPC-C [24]
four bits: bit 0 represents the setting of DPL prefetcher; could further refine the prediction models.
bit 1, the ACL prefetcher; bit 2, the DCU prefetcher; bit 3, In SPEC CPU2006, CINT2006 and CFP2006 represent
the IP prefetcher. The bit value 0 denotes that the corre- two different classes of applications in terms of their per-
sponding prefetcher is enabled, and the bit value 1 denotes formance and other characteristics. We build models with
that the corresponding prefetcher is disabled. For example, profiles from CINT2006 (INT), CFP2006 (FP) and the whole
the prefetcher configuration Config-1010 means that the CPU2006 suite (ALL) to evaluate the sensitivity of the pre-
DPL and DCU prefetchers are enabled and the ACL and IP diction model to the training dataset.
prefetchers are disabled. We must also resolve how to formulate the prefetcher
In practice, there is no one default configuration for the configuration problem to the machine learning algorithms.
hardware prefetchers across different models of machines. Na¨ ıvely, we can consider the problem as selecting the best
For example, the default configuration for many of the In- configuration from the space of 16 possible choices. However,
tel Q-series Core 2 CPU is Config-0000 (all prefetchers en- for some programs, the performance difference between two
abled), but the default configuration for many of the In- configurations may be negligible. For example, 401.bzip2
tel Core 2 E-series is Config-0011 (the two L1 prefetchers with Config-0010 is only 0.25% slower than with Config-
enabled). The configuration can also be changed by BIOS 0011. Hence, for our experiments, we also try to specify a set
setup. We have observed other preferred configurations such of good prefetcher configurations that achieve performance
as Config-0110 (DPL and IP enabled) used as the default in within 0.5% to the best configuration, instead of using one
many machines as well. The variety of default configurations single best configuration. The task of the machine learning
combined with strong impact of prefetchers on performance algorithm is to predict good configurations.
(shown in Section 4) suggest that, at the very least, one Most of the machine learning algorithms mentioned in
should take the prefetcher configuration into consideration Section 2.2 can only predict one variable at a time, which
when evaluating application performance. dictates that each algorithm chooses one configuration out
of 16 possible candidates. But, we also consider decompos-
3.3 Observables: Hardware Events ing the prediction into choosing the best configuration for
On an Intel Core 2 CPU, there are hundreds of hardware each of the four prefetchers individually. We can combine
events that can be measured [10]. It is impractical and un- the four prediction results to select the overall prefetcher
necessary to monitor all the hardware events provided by the configuration.
Core 2 architecture. Instead, we carefully select potential in- The three problem formulations are summarized below:
dicators that help guide the optimization of the prefetcher 1. Single (S): specify the single optimal prefetcher config-
configuration (see Table 2). uration.
It is difficult to reason about the exact relationship be-
tween these indicators and prefetcher performance. For ex- 2. Multiple (M): specify a set of near-optimal prefetcher](https://image.slidesharecdn.com/machine-learningbased-prefetch-optimization-for-data-center1281/85/Machine-Learning-Based-Prefetch-Optimization-for-Data-Center-4-320.jpg)
![mis-predictions are higher. This may be a problem due to 5.2 Application of Machine Learning
the limited size of the training dataset or an inherent prob- In the computer architecture community, researchers to-
lem with the model itself. If the size of the training dataset day need to tackle a rapidly growing microarchitectural de-
is to blame, expanding the training dataset is a reasonable sign space. Detailed simulation can produce the most accu-
solution to the problem. If the problem is with the model rate performance estimation, but the long simulation time
itself, it is still easier for domain experts to plug-in their usually makes this strategy impractical and prohibitive. One
knowledge into the model by locally changing the branch- solution to this problem, temporal sampling [5, 26], identifies
ing structure of the decision tree. Also, one may find the a representative portion of the program execution for de-
condition C3 to be too strong because the two resulting de- tailed simulation and infers the overall performance based
cisions produce two opposite prefetcher configurations (i.e., on the simulated result of this portion. SimPoint [5], for
all prefetchers on or all prefetchers off). In the same man- example, records the frequency of each type of instruction
ner, one can put in new conditions manually to smooth the that appears in a basic block during detailed simulation and
decisions. then uses a k-means clustering algorithm [7] to find the dif-
Overall, this example decision tree demonstrates the ben- ferent phases of the program execution. It can recognize
efits of using machine learning techniques to solve the pa- and weight each phase to estimate the overall execution.
rameter optimization problem. First, it shows that equipped Another approach, spatial sampling [11, 15], performs de-
with a good algorithm, representative datasets, and appro- tailed simulation only for some selected points in the design
priate attribute sets, it is possible to build an accurate pre- space. For instance, ¨Ipek et al. [11] choose only a small num-
diction model. Additionally, the model built from the algo- ber of points for simulation and then use the result (that
rithm not only confirms the knowledge from domain experts is, the design parameters and the simulated performance)
but also exposes more sophisticated heuristics and thresh- to train neural networks for predicting the performances of
old values. Finally, it is straightforward for domain experts other points in the design space.
(from the computer architecture community, not the ma- Combining both temporal and spatial samplings, Lee et
chine learning one) to fine-tune the model if they discover al. [14] target an even harder problem, the design param-
inconsistencies. eters for adaptive microarchitectures. They use regression
modeling and genetic algorithms to avoid comprehensive de-
tailed simulations and speed up the performance evaluation
process.
5. RELATED WORK Our work specifically addresses the problem of perfor-
Data prefetching has long been researched and remains mance tuning from a data center perspective. The prob-
an active research topic since it can highly affect perfor- lem space differs from traditional compiler optimization in
mance of the memory system. Using machine learning to that we consider system-level optimization across the data
direct program and system optimization has become a pop- center. With respect to the related work in computer archi-
ular trend for architecture research communities as it has tecture, we examine a particular class of applications, data
been rendered effective for solving non-linear problems such center applications. Additionally, we consider the problem
as performance prediction and tuning. To the best of our of selecting appropriate hardware prefetcher configurations.
knowledge, our work is the first to use machine leanrning The right configuration can have a significant impact on per-
techniques to build a data-center performance optimization formance. In this work, we describe successful strategies for
framework. We target efficient execution of large-scale sever- choosing prefetcher configurations.
side applications in a data center. This particular paper
focuses on finding the best configurations for the processor 6. CONCLUSION
prefetchers in data center machines.
In this paper, we first highlight some of the difficulties
in optimizing performance within the data center environ-
5.1 Data Prefetching ment. In response we design and implement a framework
Data prefetching is essential to the memory system per- for optimizing parameter values based on machine learning.
formance. Many researchers have identified and exploited Equipped with the framework, we conduct multiple exper-
different types of correlation. For example, Wenisch et al. iments in order to understand the sensitivity of the frame-
identified the temporal correlation property [25] and Som- work with regard to several influential components: ma-
ogyi et al. observed the spatial correlation property [21]. chine learning algorithms, training datasets selection, and
In [20], the authors extended their work and exploited both modeling or problem formulation methods. For the specific
temporal and spatial correlations to cover the drawbacks of problem of choosing hardware prefetcher configurations, we
taking advantage of only one type of correlation. Moreover, show that our framework can achieve performance within
many mechanisms for detecting and exploiting the correla- 1% of the best configuration. We have shown that judicious
tion have been proposed [9, 3, 13, 6]. Instead of devising a exploitation of the machine learning-based framework is a
more complicated prefetcher, we rely solely on the relatively promising approach to tackle the parameter value optimiza-
simple prefetchers found on modern processors and attempt tion problem.
to obtain the maximal benefit from the existing hardware.
Thus, our apporach is more likely to be deployed in a data 7. REFERENCES
center today that is filled with commodity CPUs. In addi-
tion, we believe that the proposed performance optimization [1] David W. Aha and Dennis Kibler. Instance-based
framework can also be tailored to solve other system perfor- learning algorithms. Machine Learning, 6(1):37–66,
mance problems. 1991.](https://image.slidesharecdn.com/machine-learningbased-prefetch-optimization-for-data-center1281/85/Machine-Learning-Based-Prefetch-Optimization-for-Data-Center-9-320.jpg)
![[2] M. D. Buhmann. Radial Basis Functions : Theory and Proceedings of the 12th international conference on
Implementations. Cambridge University Press, Architectural support for programming languages and
Cambridge, UK, 2003. operating systems, pages 185–194, New York, NY,
[3] Y. Chou. Low-cost epoch-based correlation prefetching USA, 2006. ACM.
for commercial applications. In MICRO ’07: [16] perfmon2: the hardware-based performance
Proceedings of the 40th Annual IEEE/ACM monitoring interface for Linux.
International Symposium on Microarchitecture, pages perfmon2.sourceforge.net.
301–313, Washington, DC, USA, 2007. IEEE [17] J. C. Platt. Fast training of support vector machines
Computer Society. using sequential minimal optimization. In Advances in
[4] W. W. Cohen. Fast effective rule induction. In kernel methods: support vector learning, pages
Proceedings of the Twelfth International Conference 185–208, Cambridge, MA, USA, 1999. MIT Press.
on Machine Learning, pages 115–123. Morgan [18] J. R. Quinlan. C4.5: programs for machine learning.
Kaufmann, 1995. Morgan Kaufmann Publishers Inc., San Francisco,
[5] G. Hamerly, E. Perelman, J. Lau, B. Calder, and CA, 1993.
T. Sherwood. Using machine learning to guide [19] S. S. le Cessie and J. van Houwelingen. Ridge
architecture simulation. J. Mach. Learn. Res., estimators in logistic regression. Applied Statistics,
7:343–378, 2006. 41(1):191–201, 1992.
[6] R. A. Hankins, T. Diep, M. Annavaram, B. Hirano, [20] S. Somogyi, T. F. Wenisch, A. Ailamaki, and
H. Eri, H. Nueckel, and J. P. Shen. Scaling and charact B. Falsafi. Spatio-temporal memory streaming. In
rizing database workloads: Bridging the gap between ISCA ’09: Proceedings of the 36th annual
research and practice. In MICRO 36: Proceedings of international symposium on Computer architecture,
the 36th annual IEEE/ACM International Symposium pages 69–80, New York, NY, USA, 2009. ACM.
on Microarchitecture, page 151, Washington, DC, [21] S. Somogyi, T. F. Wenisch, A. Ailamaki, B. Falsafi,
USA, 2003. IEEE Computer Society. and A. Moshovos. Spatial memory streaming. In ISCA
[7] J. A. Hartigan and M. A. Wong. A k-means clustering ’06: Proceedings of the 33rd Annual International
algorithm. Applied Statistics, 28(1):100–108, 1979. Symposium on Computer Architecture, pages 252–263,
[8] S. Haykin. Neural Networks: A Comprehensive Washington, DC, USA, June 2006. IEEE Computer
Foundation, 2nd edition. Prentice Hall, Upper Saddle Society.
River, NJ, 1999. [22] SPEC CPU2006. Standard Performance Evaluation
[9] I. Hur and C. Lin. Memory prefetching using adaptive Corporation. www.spec.org/cpu2006.
stream detection. In MICRO 39: Proceedings of the [23] SPECweb2009. Standard Performance Evaluation
39th Annual IEEE/ACM International Symposium on Corporation. www.spec.org/web2009.
Microarchitecture, pages 397–408, Washington, DC, [24] TPC-C. www.tpc.org/tpcc.
USA, 2006. IEEE Computer Society. [25] T. F. Wenisch, S. Somogyi, N. Hardavellas, J. Kim,
[10] Intel 64 and IA-32 Architectures Software Developer’s A. Ailamaki, and B. Falsafi. Temporal streaming of
Manuals. www.intel.com/products/processor/manuals. shared memory. In ISCA ’05: Proceedings of the 32nd
[11] E. ¨Ipek, S. A. McKee, R. Caruana, B. R. de Supinski, annual international symposium on Computer
and M. Schulz. Efficiently exploring architectural Architecture, pages 222–233, Washington, DC, USA,
design spaces via predictive modeling. In 2005. IEEE Computer Society.
ASPLOS-XII: Proceedings of the 12th international [26] R. E. Wunderlich, T. F. Wenisch, B. Falsafi, and J. C.
conference on Architectural support for programming Hoe. Smarts: Accelerating microarchitecture
languages and operating systems, pages 195–206, New simulation via rigorous statistical sampling. In
York, NY, USA, 2006. ACM. Proceedings of the 30th annual international
[12] G. H. John and P. Langley. Estimating continuous symposium on Computer architecture, pages 84–97.
distributions in bayesian classifiers. In Proceedings of ACM Press, 2003.
the Eleventh Conference on Uncertainty in Artificial
Intelligence, pages 338–345. Morgan Kaufmann,
August 1995.
[13] D. Joseph and D. Grunwald. Prefetching using markov
predictors. In ISCA ’97: Proceedings of the 24th
annual international symposium on Computer
architecture, pages 252–263, New York, NY, USA,
1997. ACM.
[14] B. C. Lee and D. Brooks. Efficiency trends and limits
from comprehensive microarchitectural adaptivity. In
ASPLOS XIII: Proceedings of the 13th international
conference on Architectural support for programming
languages and operating systems, pages 36–47, New
York, NY, USA, 2008. ACM.
[15] B. C. Lee and D. M. Brooks. Accurate and efficient
regression modeling for microarchitectural
performance and power prediction. In ASPLOS-XII:](https://image.slidesharecdn.com/machine-learningbased-prefetch-optimization-for-data-center1281/85/Machine-Learning-Based-Prefetch-Optimization-for-Data-Center-10-320.jpg)
Machine Learning-Based Prefetch Optimization for Data Center ...
- 1. Machine Learning-Based Prefetch Optimization for Data Center Applications Shih-wei Liao1,2 , Tzu-Han Hung3 , Donald Nguyen4 , Chinyen Chou2 , Chiaheng Tu2 , and Hucheng Zhou5 1 Google Inc., Mountain View, California, USA 2 National Taiwan University, Taipei, Taiwan 3 Princeton University, Princeton, New Jersey, USA 4 University of Texas at Austin, Austin, Texas, USA 5 Tsinghua University, Beijing, China Email addresses: sliao@google.com, thhung@cs.princeton.edu, ddn@cs.utexas.edu, r96079@csie.ntu.edu.tw, d94944008@csie.ntu.edu.tw, zhou-hc07@mails.tsinghua.edu.cn Performance tuning for data centers is essential and com- the data center. This adds new dimensions to the config- plicated. It is important since a data center comprises thou- uration space and adds external dependencies into the op- sands of machines and thus a single-digit performance im- timization problem. Consolidation is driven by application provement can significantly reduce cost and power consump- scale, the need for high utilization and emerging applica- tion. Unfortunately, it is extremely difficult as data centers tions. Consolidation dually multiplies the impact of appli- are dynamic environments where applications are frequently cation optimizations and increases its difficulty. released and servers are continually upgraded. We propose a system to optimize applications according In this paper, we study the effectiveness of different pro- to data center constraints. We model optimization as a pa- cessor prefetch configurations, which can greatly influence rameter search problem and use machine learning techniques the performance of memory system and the overall data cen- to predict the best parameter values. This framework is ter. We observe a wide performance gap when comparing designed for seamless integration into existing data center the worst and best configurations, from 1.4% to 75.1%, for practices and requires minimal operators’ intervention. 11 important data center applications. We then develop In this paper, we focus on optimizing memory system per- a tuning framework which attempts to predict the optimal formance by configuring memory prefetchers. As hardware configuration based on hardware performance counters. The moves towards integrated memory controllers and higher framework achieves performance within 1% of the best per- bandwidth, configuring prefetchers properly will play an im- formance of any single configuration for the same set of ap- portant rule in maximizing system performance. Proces- plications. sor manufacturers or data center operators may design or tune prefetchers for aggressive prefetching. This produces superior performance for applications that are not limited 1. INTRODUCTION by memory bandwidth, but aggressive prefetching produces Data center operators leverage economies of scale to in- pathological behavior for applications that have finely tuned crease performance per watt. The large scale of data center memory behavior. Due to the impact of scale, programmers applications encourages operators to be meticulous about of data center applications commonly apply memory opti- application performance. A single-digit improvement can mizations that increase the effective utilization of bandwidth obviate thousands of machines. However, optimizing data like data structure compaction and alignment, and software center applications is a delicate task. To increase machine prefetching. Aggressive prefetching can destroy program- utilization, operators host several applications on the same mers’ careful orchestration of memory. physical machine, but this may reduce aggregate perfor- We have found few papers that address the impact of mance by increasing contention on shared hardware resources. prefetcher configuration from the perspective of the applica- Cloud Computing introduces third-party applications into tion programmer or the data center operator, even though proper configuration has a significant performance impact. Instead, most papers address the concerns of hardware de- signers. Choosing the right configuration, however, has a Permission to make digital or hard copies of all or part of this work for large impact on performance. On a collection of 11 data personal or classroom use is granted without fee provided that copies are center applications, we find a range of improvement between not made or distributed for profit or commercial advantage and that copies 1.4% to 75.1% comparing the worst to best performing con- bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific figurations. permission and/or a fee. We apply machine learning to predict the best configura- SC 09 November 14-20, 2009, Portland, Oregon, USA tion using data from hardware performance counters, and Copyright 2009 ACM 978-1-60558-744-8/09/11 ...$10.00.
- 2. we show a methodology that predicts configurations that modified. Also, the inputs to the machine learning algo- achieve performance within 1% of the per-application best rithms, hardware and kernel statistics, measure system-level configuration. effects that are most relevant to a data center operator. The To address the context of dynamic data centers, we con- framework easily extends to more inputs, and since it does sider co-hosted applications, dynamic application mix and not require modifying application binaries, it applies widely large-scale deployments. For instance, when a job is sched- to many applications and deployment environments. uled to be run next, the general tuning framework either The framework is as follows: predicts or looks up this job’s best configuration. In this paper we present the general framework but only report the 1. Specify the tunables. The tunables are the parame- results on isolated application executions. That is, evalua- ters that can be adjusted to improve the performance tion on a more dynamic environment is the subject of future (throughput, latency, etc.) of a program. The opti- papers. To summarize, this paper consists of the following mization choices can be at the level of source code, contributions: library, system, or processor. The setting of the tun- ables will be the output of the machine learning engine. 1. A system that significantly improves application per- formance transparently to the application programmer 2. Identify the observables. The observables are events and with minimal overhead to data center operators. that can be measured by the framework and are de- pendent on the tunables. For example, cycles per in- 2. A study of the performance of real data center appli- struction is an appropriate observable if a data center cations under varying hardware prefetcher configura- operator wants to improve throughput. If the memory tions. system performance is of interest, the cache miss rate is an appropriate observable. The observables, also 3. An instantiation of machine learning, including select- called as features in the machine learning terminology, ing the training dataset and problem formulation, that will be the input of the machine learning engine. predicts near-optimal prefetch configurations. 3. Create a training dataset for machine learning algo- The rest of the paper is organized as follows. Section 2 rithms. Given a set of applications, the framework runs introduces our general framework for parameter space opti- the applications under varying configurations of tun- mization and machine learning-based prediction. Section 3 ables and collects the observables specified by the user. demonstrates the application of the proposed framework to The benchmark set should be representative of the ap- hardware prefetchers. We present and analyze experimental plications likely to be seen in the data center. We results in Section 4. Section 5 summarizes the related work discuss some issues with selecting application bench- in parameter search and optimization. Finally, we conclude marks in Section 3.4. To produce a training dataset, this paper in Section 6. the framework applies exhaustive search or random- ized sampling to find the best or good tunable con- 2. PARAMETER SPACE EXPLORATION VIA figurations. Because the space of hardware prefetcher configurations is small, we use exhaustive search in this MACHINE LEARNING paper. The observable data and tunable configura- This section describes an application tuning framework for tions form the training dataset for a machine learning data center operators that uses machine learning for rapid engine. Note that exhaustive search without any sta- parameter space exploration. tistical techniques is not a general solution for dynamic data centers, but is an acceptable one for creating the 2.1 Framework and Methodology training dataset. In the data center, the environment for program optimiza- tion is different from traditional compiler optimization. Ap- 4. Build the prediction models. After collecting the train- plication programmers give program binaries and execution ing dataset, the framework runs machine learning al- constraints to a data center operator who, in turn, must allo- gorithms to construct prediction models. Section 2.2 cate sufficient machine resources to execute the application summarizes the algorithms we use in this experiment. and satisfy the execution constraints while maximizing the Figure 1(a) illustrates the procedure of model con- effective utilization of the data center as a whole. struction. The framework gives each algorithm a set To facilitate scheduling, the details of the underlying ma- of observables and tunables and asks the algorithm to chine hardware are obscured from the programmer. Even generate a prediction model that, when given a set of if these details were available, taking advantage of the in- observables, predicts the best set of tunables. formation would be difficult. In the data center, tasks mi- grate from machine to machine, fail on one machine and are 5. Use the models for prediction. Now, in the data center, restarted on another, and are co-hosted with other applica- when an application is executed, its execution will be tions. Additionally, the data center operators are concerned monitored by the framework. Initially, the application with optimizing the performance of the data center as a will run with a default configuration of tunables, but whole. the framework will collect observable data during ex- Thus, we propose a simple application tuning framework ecution. After a predetermined period of monitoring, based on applying machine learning over system-level events the framework will predict the best tunable configura- like hardware event counters and kernel statistics. This ap- tion by applying the collected observable data to the proach is particularly suitable for data center operators be- prediction model. Figure 1(b) illustrates the procedure cause application binaries do not need to be recompiled or of predicting the tunables of an incoming application.
- 3. (a) Workflow of framework construction. (b) Workflow of framework evaluation. Figure 1: Framework construction and evaluation. The symbol o denotes an observable and the symbol t denotes a tunable. 2.2 Machine Learning Algorithms applied to the RBF functions, and an output layer which Many different machine learning algorithms can be used produces the linear combination of the outputs from the to generate the prediction model. We selected a large set of RBF functions. algorithms and evaluated their performance over our train- ing set. In this section, we summarize the algorithms we 3. APPLICATION TO HARDWARE PRE- considered. FETCHERS We evaluated several classical machine learning algorithms. We apply the framework introduced previously to find the The nearest neighbor (NN) [1] algorithm measures the sim- best configuration of hardware prefetchers for a given appli- ilarity of each data instance using Euclidean distance and cation. In this section, we define the tunables and observ- classifies a new data instance according to its nearest neigh- ables used and explain how to build the training dataset. bor in the training dataset. The na¨ Bayes (NB) [12] method ıve computes the probability that a given data instance belongs 3.1 Platform to each class and classifies a new data instance according to Table 1 lists the major components of our experimental the class with the highest probability. A tree-based classi- platform. One computing server consists of 2 or 4 sockets. fier takes a divide-and-conquer strategy. The C4.5 decision Each socket is an Intel Core 2 Quad CPU. Core 2 Quad tree (DT) [18] classifier uses a divide-and-conquer strategy. comprises 4 CPU cores, equipped with a shared L2 cache It partitions the training dataset based on selecting pivots for every two cores. Each CPU core has a 32KB L1 data that result in the greatest information gain. The partition- cache and a 32KB L1 instruction cache. Each of the two ing is then recursively applied to each sub-partition. Alter- shared L2 caches is 4MB. All the programs were compiled natively, a rule-based classifier uses a separate-and-conquer by GCC 4.3.1 with -O2 flag. We monitor execution and technique. The Ripper classifier (RP) [4] greedily attempts collect hardware events with perfmon2 [16]. to construct a rule to classify as many instances as possible within a class. It then separates out the instances that were 3.2 Tunables: Hardware Prefetchers classified and continues on with those still unclassified. On an Intel Core 2 CPU, there are four hardware prefetch- We evaluated many modern machine learning algorithms ers [10]: as well. Support vector machines (SVM) [17] treat the in- put data as vectors in an N-dimensional space and find a 1. Data Prefetch Logic (DPL) prefetcher: When the pro- maximum-margin hyperplane to separate the vectors into cessor detects stride access, this prefetcher fetches two classes, one on each side of the hyperplane. Logistic re- streams of instructions and data from memory to the gression (LR) [19] can predict a categorical type and capture unified L2 cache. the non-linear relationship between the input features and 2. Adjacent Cache Line (ACL) prefetcher: This prefetcher output predictions. We also evaluated two types of neural fetches the matching cache line in a cache line pair (128 networks. One is the typical multi-layer perceptron (MLP) [8] bytes) to the L2 unified cache. network with backward propagation. The other is the radial basis function (RBF) [2] network which consists of three lay- 3. Data Cache Unit (DCU) prefetcher: This prefetcher ers: an input layer, a hidden layer where the inputs are fetches the next cache line into the L1 data cache
- 4. CPU Model Intel Core 2 Quad ample, if an application runs on a machine with all prefetch- L1 cache (Data 256KB (4 * (32KB + 32KB)) ers disabled, a high cache miss rate may be a good indicator and Instruc- that some of the prefetchers should be turned on to bring tion) data into cache earlier and reduce the cache miss penalty. L2 cache 8MB (2 * 4MB) However, in an environment where all prefetchers are en- Main Memory 16 GB abled, a high cache miss rate suggests that the some of the Operating Sys- Linux (Kernel version: 2.6.18) prefetchers should be turned off, because the cache misses tem are likely caused by a too aggressive prefetcher setting. A Compiler GCC 4.3.1 (Flags: -O2) high cache miss rate may also just be indicative of poor spa- Performance perfmon2 tial locality of the application. We only highlight some of Monitoring the difficulties interpreting cache miss rate. Similar analysis Software can be done for the rest of the selected hardware events. The relationship between these observables and perfor- mance also depends on the particular processor. Different Table 1: Details of the platform. architectures have different performance models, and rela- tionships derived on one architecture do not necessarily fol- low on another architecture. The general difficulty estab- when it detects multiple loads from the same cache lishing a relationship between observed application behav- line within a fixed period of time. ior and optimal prefetcher configuration drives us towards machine learning. 4. Instruction Pointer-based (IP) prefetcher: This prefetcher keeps sequential load history information 3.4 Data Collection per instruction pointer and uses this data to decide Before applying our application tuning framework, we whether to fetch data for an upcoming delinquent load must address two issues: what is the training dataset and instruction into the L1 data cache. what is the data format. The ideal training dataset should Each processor core has one DCU prefetcher and one IP be as representative and large as possible. Currently, we prefetcher for its L1 cache and also one DPL prefetcher and use profiles (i.e., hardware events) collected from running one ACL prefetcher for the shared L2 cache. Each hardware all programs in the SPEC CPU2006 [22] benchmark, with prefetcher can be enabled or disabled independently, so there both train and ref input data for each program, to build are 16 prefetcher configurations for each core. For the re- our prediction models. We believe including other represen- mainder of this paper, we encode the configurations with tative benchmarks such as SPECweb2009 [23] or TPC-C [24] four bits: bit 0 represents the setting of DPL prefetcher; could further refine the prediction models. bit 1, the ACL prefetcher; bit 2, the DCU prefetcher; bit 3, In SPEC CPU2006, CINT2006 and CFP2006 represent the IP prefetcher. The bit value 0 denotes that the corre- two different classes of applications in terms of their per- sponding prefetcher is enabled, and the bit value 1 denotes formance and other characteristics. We build models with that the corresponding prefetcher is disabled. For example, profiles from CINT2006 (INT), CFP2006 (FP) and the whole the prefetcher configuration Config-1010 means that the CPU2006 suite (ALL) to evaluate the sensitivity of the pre- DPL and DCU prefetchers are enabled and the ACL and IP diction model to the training dataset. prefetchers are disabled. We must also resolve how to formulate the prefetcher In practice, there is no one default configuration for the configuration problem to the machine learning algorithms. hardware prefetchers across different models of machines. Na¨ ıvely, we can consider the problem as selecting the best For example, the default configuration for many of the In- configuration from the space of 16 possible choices. However, tel Q-series Core 2 CPU is Config-0000 (all prefetchers en- for some programs, the performance difference between two abled), but the default configuration for many of the In- configurations may be negligible. For example, 401.bzip2 tel Core 2 E-series is Config-0011 (the two L1 prefetchers with Config-0010 is only 0.25% slower than with Config- enabled). The configuration can also be changed by BIOS 0011. Hence, for our experiments, we also try to specify a set setup. We have observed other preferred configurations such of good prefetcher configurations that achieve performance as Config-0110 (DPL and IP enabled) used as the default in within 0.5% to the best configuration, instead of using one many machines as well. The variety of default configurations single best configuration. The task of the machine learning combined with strong impact of prefetchers on performance algorithm is to predict good configurations. (shown in Section 4) suggest that, at the very least, one Most of the machine learning algorithms mentioned in should take the prefetcher configuration into consideration Section 2.2 can only predict one variable at a time, which when evaluating application performance. dictates that each algorithm chooses one configuration out of 16 possible candidates. But, we also consider decompos- 3.3 Observables: Hardware Events ing the prediction into choosing the best configuration for On an Intel Core 2 CPU, there are hundreds of hardware each of the four prefetchers individually. We can combine events that can be measured [10]. It is impractical and un- the four prediction results to select the overall prefetcher necessary to monitor all the hardware events provided by the configuration. Core 2 architecture. Instead, we carefully select potential in- The three problem formulations are summarized below: dicators that help guide the optimization of the prefetcher 1. Single (S): specify the single optimal prefetcher config- configuration (see Table 2). uration. It is difficult to reason about the exact relationship be- tween these indicators and prefetcher performance. For ex- 2. Multiple (M): specify a set of near-optimal prefetcher
- 5. Event Description Relationship with Hardware Prefetching CPI Clock per (retired) instruction Prefetching may improve or worsen CPI L1D LD MISS L1 data cache miss events due to loads Prefetching may reduce cache misses (when the prefetching delivers required data to the cache on time) or increase cache misses (when prefetching delivers the wrong data to the cache) L1D ALLOC Cache lines allocated in the L1 data Similar to L1 cache miss cache L1D M EVICT Modified cache lines evicted from the Similar to L1 cache miss L1 data cache L2 LD MISS L2 cache miss events due to loads Similar to L1 cache miss L2 LINES IN Number of cache lines allocated in the Suppressing prefetching might be helpful when this L2 cache (due to prefetching) number is too high L2 LINES OUT Number of cache lines evicted from Suppressing prefetching might be helpful when this the L2 cache (due to prefetching) number is too high DTLB MISS DTLB miss events due to all memory A high DTLB miss suggests to not use prefetch- operations ing because the program working set is large and unpredictable DTLB LD MISS DTLB miss events due to loads Similar to DTLB miss BLOCKED LOAD Loads blocked by a preceding store Prefetching may not be useful if there are lots of (with unknown address or data) or by blocked load instructions the L1 data cache BUS TRAN BURST Number of completed burst transac- Prefetching may deteriorate the use of bus band- tions width BR PRED MISS Branch mis-prediction events Prefetching on a wrong path (due to branch mis- prediction) may unnecessarily pollute the cache Table 2: Hardware events that are measured in the experiments. configurations. 4.1 Evaluation of Training Dataset Figure 2 compares the speedup (i.e., the improvement of wall-clock running time) of our DCA testing dataset using 3. Independent (I): use four instances of the training prediction models trained on the INT, FP, and ALL training dataset and independently specify the configuration datasets for various machine learning algorithms and using (enabled or disabled) of each prefetcher. the M problem formulation. From our experiments, INT is the best training dataset. The reason is that the programs in INT are the most similar 4. EXPERIMENTAL RESULTS to the applications in DCA. They share similar characteris- We use another set of benchmarks, our data center ap- tics, not only in terms of the high-level tasks (e.g., searching, plications (DCA), to evaluate the benefit of our prediction counting, etc.) but also in terms of the low-level hardware models. Table 3 summarizes the applications, whose total events (e.g., CPI, cache miss rate, branch mis-prediction, codebase is more than one million lines of code. Among etc.) that are observed by the performance monitoring soft- many data center applications, we selected 11 of them that ware. The similarity of these programs is a strong indication were representative of the overall application mix. Since that models built from the INT dataset can best guide the we are concerned with optimizing the total system perfor- prediction of prefetcher configuration for our data center ap- mance, we weigh each application according to its relative plications. and approximate execution time within a data center. On the other hand, models built with the FP dataset are In this section, we examine the effects of using different consistently worse than those built with the INT dataset. training datasets and problem formulations when construct- From the high-level point of view, the floating-point pro- ing the prediction models. Then, we analyze in depth the grams tend to have more regular structure (e.g., control flow performance improvement of each application in DCA. Fi- and data access patterns), which is not the case for appli- nally, we carefully examine a decision tree generated by a cations in the DCA set. These disparities are also visible in machine learning algorithm and see if it supports the do- the low-level hardware events. For example, the event count main knowledge of an expert. of L2 LINE IN and L2 LINE OUT are both much higher To report meaningful performance improvement, all the for FP. The regular data access patterns give the hardware performance numbers are compared with the per-applica- prefetchers more opportunities to fetch data into and evict tion best results. When calculating the overall performance data from the L2 cache. The poor representation of DCA by for DCA suite, we use the weighted geometric mean for all FP leads to poor predictive models as well. In many cases, the applications in DCA to reflect the true enhancement ob- models trained with FP predict configurations that do worse served in our data center environment. The weights of DCA than the default prefetcher configuration (see NN, NB, MLP, applications are listed in Table 3. RBF).
- 6. App Category Weight Gap App Category Weight Gap PageScorer Searching ∼20% 24.5% PhraseQueryServer Searching ∼20% 23.6% ContentIndexer Indexing ∼1% 24.5% Clustering Indexing ∼1% 11.1% DocCluster Learning ∼5% 24.2% KeywordCluster Learning ∼5% 51.8% HugeSort Sorting ∼1% 50.6% DataProcessor MapReduce ∼20% 8.9% ContentAds Ads ∼5% 1.4% StreetImaging Imaging ∼1% 75.1% OfflineContent OCR ∼5% 44.0% Table 3: Data center applications with categories. Weights represent approximate contributions of appli- cations or similar application types to the total execution time of a data center. We partition applications into three groups: heavily used (weighted ∼20%), typically used (weighted ∼5%), and rarely used (weighted ∼1%). Naturally the first group represents a wider range of weights than the latter two groups do. Gaps represent the performance difference between the best and worst prefetch configurations. Figure 2: Effect of using different training datasets on machine learning algorithms. Results show the DCA performance with predicted configurations, compared against the per-application best configuration. The M problem formulation is used. The quality of the models built with the ALL dataset falls good configuration too low and thus include too many “good” between those built with the INT and the FP datasets. How- configurations in the training dataset, the models built are ever, for the NN model, ALL is slightly better than INT. This worse. The predictors are confused by too much misleading is because there are fewer data instances in FP and they are and redundant information. For example, the performance not close enough to the INT data instances to confuse the NN results of 445.gobmk for all the 16 prefetcher configurations algorithm. When evaluating the data instances from the DCA are within 0.3%, and it is unreasonable to pass all these con- dataset, the algorithm can still pick up the nearest neighbor figurations to the model, so we remove 445.gobmk from the from an INT data point. INT dataset for the M formulation. Surprisingly, the I formulation is not as good as expected. This approach transforms, not algorithmically, but with do- 4.2 Evaluation of Problem Formulation main knowledge, a multi-class classification problem into Figure 3 compares the speedup of our DCA testing suite four independent binary classification problems. This makes using different problem formulations with the INT training the problem conceptually easier to evaluate and thus makes dataset. Among the three approaches, the M formulation is the predictors run faster. The drawback is that this ap- generally the best. This is not surprising because this formu- proach ignores the relationships between the four hardware lation essentially provides more information to the machine prefetchers, even though some of them are related and their learning algorithms by including all the good prefetcher con- settings should be considered together. For example, the figurations in the training dataset. On the other hand, the L1 DCU prefetcher and the L2 ACL prefetcher both fetch S formulation only considers the best prefetcher configura- the next line to the cache. The difference is that the DCU tion, which is a significant information loss, especially be- prefetcher keeps a timer and checks if there are multiple cause most good configurations have very close performance loads from the same line within a time period. Thus, if with the best performance for some programs. However, enabling the L1 DCU prefetcher fails to boost the perfor- we do find that if we set the threshold value for defining a
- 7. Figure 3: Effect of using different problem formulations. Results show the DCA performance with predicted configurations, compared against the per-application best configuration. The INT training dataset is used. mance, it is likely that enabling the L2 ACL prefetcher will high performance variations. not help either. With the I formulation, the correlation be- The second category has a performance gap between 20%– tween these decisions are neglected because the settings of 40%. Although most predictors can obtain near-optimal the four prefetchers are determined separately. speedup, some predictors do suffer from a substantial perfor- mance degradation. For instance, for DocCluster, RBF loses more than 20% to the best configuration; for PageScorer, NB 4.3 Individual Prediction Results for DCA Ap- and SVM both lose more than 10%; for Clustering, RP and plications RBF both lose around 20%. However, among these programs, In the following experiment, we examine the sensitivity of PageScorer is one of the most important applications and DCA applications to prefetcher configurations and compare many predictors get the near-optimal performance improve- the performance of three selected machine learning mod- ment. These predictors are two classical algorithms, RP and els (DT, LR, and MLP) against three commonly used default DT, and two advanced neural networks, MLP and RBF. configurations. Table 4 shows the individual speedup for The last category includes applications that show a per- each application and the weighted (geomean) speedup for formance gap greater than 40%. The impact of the different the whole DCA suite. prediction models varies dramatically. For OfflineContent, For many DCA applications, a good configuration is to three predictors can only achieve less than 90% of the opti- have all prefetchers enabled. Four notable exceptions are mal performance, although the other five predictors can do PageScorer, DocCluster, KeywordCluster, and Cluster- much better. For StreetImaging, five predictors lose to the ing. In the case of PageScorer which typically consumes optimal performance by 20%, and only three predictors are a plurality of cycles, one of the good configurations was actually helpful. Overall, for the programs with high perfor- Config-1111 which disables all hardware prefetchers. This mance variations, DT and LR are reasonably stable and good application already performs many low-level memory opti- enough to achieve the near-optimal speedup. mizations, so we believe that the addition of prefetchers ac- It is worth noting that none of the three default configura- tually hinders performance. tions can deliver the best overall performance. Config-0011 Based on the performance gap of the best and the worst achieves the best performance, although it is still outper- prefetcher configurations, shown in Table 3, the applications formed by the DT, LR, and MLP predictors. The widely used in the DCA benchmark can be roughly classified into three configuration, Config-0000, is the worst among these three, categories as shown in Table 4. In the first category, there is which confirms that the DCA applications and the data cen- about a 10%, or even less, difference between the worst and ter environment are different from traditional optimization best configuration. These applications are relatively insen- contexts. sitive to hardware prefetching. Take ContentAds for exam- Overall, most of our data center applications can benefit ple. Even the optimal prefetcher configuration only achieves from a better setting of hardware prefetchers: In addition 1.4% performance speedup over the worst prefetcher config- to OfflineContent, HugeSort, and StreetImaging, all the uration (see Table 3). Although the room for performance other eight programs can usually obtain substantial perfor- improvement is limited for applications in this group, most mance improvement with the use of our predicted config- predictors still obtain near-optimal performance. The only uration. Among those prediction models, the DT (decision exception is NB for Clustering. In fact, the performance tree) predictor is usually good enough to have near-optimal improvement of using the NB predictor is less stable across predictions, and it is attractive because it produces simple programs than other predictors, especially for programs with
- 8. Benchmark DT LR MLP Config-0000 Config-0011 Config-0110 ContentAds 0.991 0.993 0.996 0.999 0.996 0.993 LogProcessing 0.985 0.995 0.998 0.999 0.989 0.996 Clustering 1.000 0.980 1.00 0.989 0.978 0.993 PhraseQueryServer 0.987 0.981 0.970 1.000 0.980 0.998 DocCluster 0.937 0.988 0.962 0.972 0.974 0.962 ContentIndexer 0.994 0.956 0.994 1.000 0.867 0.964 PageScorer 1.000 0.961 1.000 0.803 0.989 0.919 OfflineContent 1.000 1.00 0.999 0.999 0.858 0.986 Sorting 0.900 0.812 0.864 1.000 0.923 0.792 KeywordCluster 1.000 0.994 0.995 0.967 0.993 0.659 StreetImaging 1.000 0.991 0.827 1.000 0.784 0.995 Weighted Geomean 0.989 0.978 0.988 0.921 0.976 0.944 Table 4: Performance of applications in DCA with selected prediction models or prefetcher configurations compared to the best prefetcher configuration for each application. The INT training dataset and the M problem formulation are used. models and tree structures which are easy to interpret. tures (i.e., hardware events) to decide the use of hardware prefetchers, it would be difficult to come up with a rule hi- 4.4 A Decision Tree Example erarchy that is five levels deep. Furthermore, the decision tree model greatly confirms the DTLB_LD_MISS <= 1.13841 (C1) domain knowledge that is essential to the prefetching opti- | DTLB_MISS <= 3.161832 (C2) mization problem. The features that frequently appear in | | BR_PRED_MISS <= 18.97335 (C3) the decision tree are: DTLB MISS, L1D LD MISS, | | | DTLB_MISS <= 1.062258: 0 (D1) BLOCKED LOAD, BR PRED MISS. | | | DTLB_MISS > 1.062258: 15 (D2) The event DTLB MISS measures the DTLB misses due to | | BR_PRED_MISS > 18.97335 (C4) all memory operations. A high miss rate for DTLB suggests | | | L1D_LD_MISS <= 0.612822: 2 (D3) that the program has a large and relatively unpredictable | | | L1D_LD_MISS > 0.612822 (C5) working set and thus the prefetching mechanism is more | | | | BR_PRED_MISS <= 22.411147: 6 (D4) likely to pollute the cache system by fetching unnecessary | | | | BR_PRED_MISS > 22.411147: 4 (D5) data into it. A high DTLB miss rate is a signal for disabling | DTLB_MISS > 3.161832 (C6) hardware prefetchers. Comparing the sub-trees following | | BLOCKED_LOAD <= 51.386555: 8 (D6) conditions C2 and C6, we see that the decisions in the C2 | | BLOCKED_LOAD < 51.386555: 5 (D7) sub-tree are more inclined to turn on DPL, the most effective DTLB_LD_MISS > 1.13841 (C7) L2 prefetcher, because the DTLB miss rates are lower. | BLOCKED_LOAD <= 50.275612 (C8) The event L1D LD MISS measures load instructions that | | L1D_LD_MISS <= 26.877868 (C9) cause L1 data cache misses. In our training experiments, the | | | BLOCKED_LOAD <= 46.849349: 2 (D8) programs are run with all the prefetchers enabled. There- | | | BLOCKED_LOAD > 46.849349: 4 (D9) fore, a high L1 miss rate in our training dataset implies that | | L1D_LD_MISS > 26.877868: 14 (D10) some of the prefetchers should be disabled because the high | BLOCKED_LOAD > 50.275612 (C10) L1 miss rate is likely due to overly aggressive prefetching. | | DTLB_MISS <= 15.18429: 1 (D11) In Figure 4, D3 suggests turning on both L1 prefetchers be- | | DTLB_MISS > 15.18429: 10 (D12) cause L1 miss rate is lower, while D4 and D5 suggest turning on only one L1 prefetcher, the IP prefetcher. The event BLOCKED LOAD measures the blocked loads. Figure 4: A decision tree model for the prefetching A load instruction may be blocked if a preceding store in- optimization problem. struction has an address or data that is not yet known to the hardware or the number of L1 data misses exceeds the Figure 4 shows a decision tree used in our model. The maximum number of outstanding misses that the processor internal nodes represent conditions and the leaf nodes rep- can handle. With more blocked load instructions, the ben- resent decisions. A condition is composed of a threshold efits of prefetching tend to be less since prefetching will not value and a two-way branch. In this example, the threshold be able to hide the miss latency if the blocking cannot be values are the numbers of occurrences of the corresponding resolved. In Figure 4, with fewer blocked loads, D6 decides hardware events in 1000 retired instructions, and the deci- to adopt more aggressive prefetching than D7 does. sions are the 4-bit encodings of the prefetcher configurations. One interesting contradiction in this decision tree is the For example, a leaf node labeled with 0 is a decision that all use of BR PRED MISS (C5). Generally speaking, if the the four hardware prefetchers should be enabled. program incurs more branch mis-predictions and thus more This decision tree represents a complex decision-making wrongly speculated executions, aggressive optimizations system that even a domain expert can hardly compete against. should be disabled to preserve performance. However, D4 It contains 10 conditions and 12 decisions, with a decision suggests turning on two prefetchers when mis-predictions are depth of five. Although it is possible for an expert in the lower while D5 suggests turning on three prefetchers when prefetching optimization problem to select the useful fea-
- 9. mis-predictions are higher. This may be a problem due to 5.2 Application of Machine Learning the limited size of the training dataset or an inherent prob- In the computer architecture community, researchers to- lem with the model itself. If the size of the training dataset day need to tackle a rapidly growing microarchitectural de- is to blame, expanding the training dataset is a reasonable sign space. Detailed simulation can produce the most accu- solution to the problem. If the problem is with the model rate performance estimation, but the long simulation time itself, it is still easier for domain experts to plug-in their usually makes this strategy impractical and prohibitive. One knowledge into the model by locally changing the branch- solution to this problem, temporal sampling [5, 26], identifies ing structure of the decision tree. Also, one may find the a representative portion of the program execution for de- condition C3 to be too strong because the two resulting de- tailed simulation and infers the overall performance based cisions produce two opposite prefetcher configurations (i.e., on the simulated result of this portion. SimPoint [5], for all prefetchers on or all prefetchers off). In the same man- example, records the frequency of each type of instruction ner, one can put in new conditions manually to smooth the that appears in a basic block during detailed simulation and decisions. then uses a k-means clustering algorithm [7] to find the dif- Overall, this example decision tree demonstrates the ben- ferent phases of the program execution. It can recognize efits of using machine learning techniques to solve the pa- and weight each phase to estimate the overall execution. rameter optimization problem. First, it shows that equipped Another approach, spatial sampling [11, 15], performs de- with a good algorithm, representative datasets, and appro- tailed simulation only for some selected points in the design priate attribute sets, it is possible to build an accurate pre- space. For instance, ¨Ipek et al. [11] choose only a small num- diction model. Additionally, the model built from the algo- ber of points for simulation and then use the result (that rithm not only confirms the knowledge from domain experts is, the design parameters and the simulated performance) but also exposes more sophisticated heuristics and thresh- to train neural networks for predicting the performances of old values. Finally, it is straightforward for domain experts other points in the design space. (from the computer architecture community, not the ma- Combining both temporal and spatial samplings, Lee et chine learning one) to fine-tune the model if they discover al. [14] target an even harder problem, the design param- inconsistencies. eters for adaptive microarchitectures. They use regression modeling and genetic algorithms to avoid comprehensive de- tailed simulations and speed up the performance evaluation process. 5. RELATED WORK Our work specifically addresses the problem of perfor- Data prefetching has long been researched and remains mance tuning from a data center perspective. The prob- an active research topic since it can highly affect perfor- lem space differs from traditional compiler optimization in mance of the memory system. Using machine learning to that we consider system-level optimization across the data direct program and system optimization has become a pop- center. With respect to the related work in computer archi- ular trend for architecture research communities as it has tecture, we examine a particular class of applications, data been rendered effective for solving non-linear problems such center applications. Additionally, we consider the problem as performance prediction and tuning. To the best of our of selecting appropriate hardware prefetcher configurations. knowledge, our work is the first to use machine leanrning The right configuration can have a significant impact on per- techniques to build a data-center performance optimization formance. In this work, we describe successful strategies for framework. We target efficient execution of large-scale sever- choosing prefetcher configurations. side applications in a data center. This particular paper focuses on finding the best configurations for the processor 6. CONCLUSION prefetchers in data center machines. In this paper, we first highlight some of the difficulties in optimizing performance within the data center environ- 5.1 Data Prefetching ment. In response we design and implement a framework Data prefetching is essential to the memory system per- for optimizing parameter values based on machine learning. formance. Many researchers have identified and exploited Equipped with the framework, we conduct multiple exper- different types of correlation. For example, Wenisch et al. iments in order to understand the sensitivity of the frame- identified the temporal correlation property [25] and Som- work with regard to several influential components: ma- ogyi et al. observed the spatial correlation property [21]. chine learning algorithms, training datasets selection, and In [20], the authors extended their work and exploited both modeling or problem formulation methods. For the specific temporal and spatial correlations to cover the drawbacks of problem of choosing hardware prefetcher configurations, we taking advantage of only one type of correlation. Moreover, show that our framework can achieve performance within many mechanisms for detecting and exploiting the correla- 1% of the best configuration. We have shown that judicious tion have been proposed [9, 3, 13, 6]. Instead of devising a exploitation of the machine learning-based framework is a more complicated prefetcher, we rely solely on the relatively promising approach to tackle the parameter value optimiza- simple prefetchers found on modern processors and attempt tion problem. to obtain the maximal benefit from the existing hardware. Thus, our apporach is more likely to be deployed in a data 7. REFERENCES center today that is filled with commodity CPUs. In addi- tion, we believe that the proposed performance optimization [1] David W. Aha and Dennis Kibler. Instance-based framework can also be tailored to solve other system perfor- learning algorithms. Machine Learning, 6(1):37–66, mance problems. 1991.
- 10. [2] M. D. Buhmann. Radial Basis Functions : Theory and Proceedings of the 12th international conference on Implementations. Cambridge University Press, Architectural support for programming languages and Cambridge, UK, 2003. operating systems, pages 185–194, New York, NY, [3] Y. Chou. Low-cost epoch-based correlation prefetching USA, 2006. ACM. for commercial applications. In MICRO ’07: [16] perfmon2: the hardware-based performance Proceedings of the 40th Annual IEEE/ACM monitoring interface for Linux. International Symposium on Microarchitecture, pages perfmon2.sourceforge.net. 301–313, Washington, DC, USA, 2007. IEEE [17] J. C. Platt. Fast training of support vector machines Computer Society. using sequential minimal optimization. In Advances in [4] W. W. Cohen. Fast effective rule induction. In kernel methods: support vector learning, pages Proceedings of the Twelfth International Conference 185–208, Cambridge, MA, USA, 1999. MIT Press. on Machine Learning, pages 115–123. Morgan [18] J. R. Quinlan. C4.5: programs for machine learning. Kaufmann, 1995. Morgan Kaufmann Publishers Inc., San Francisco, [5] G. Hamerly, E. Perelman, J. Lau, B. Calder, and CA, 1993. T. Sherwood. Using machine learning to guide [19] S. S. le Cessie and J. van Houwelingen. Ridge architecture simulation. J. Mach. Learn. Res., estimators in logistic regression. Applied Statistics, 7:343–378, 2006. 41(1):191–201, 1992. [6] R. A. Hankins, T. Diep, M. Annavaram, B. Hirano, [20] S. Somogyi, T. F. Wenisch, A. Ailamaki, and H. Eri, H. Nueckel, and J. P. Shen. Scaling and charact B. Falsafi. Spatio-temporal memory streaming. In rizing database workloads: Bridging the gap between ISCA ’09: Proceedings of the 36th annual research and practice. In MICRO 36: Proceedings of international symposium on Computer architecture, the 36th annual IEEE/ACM International Symposium pages 69–80, New York, NY, USA, 2009. ACM. on Microarchitecture, page 151, Washington, DC, [21] S. Somogyi, T. F. Wenisch, A. Ailamaki, B. Falsafi, USA, 2003. IEEE Computer Society. and A. Moshovos. Spatial memory streaming. In ISCA [7] J. A. Hartigan and M. A. Wong. A k-means clustering ’06: Proceedings of the 33rd Annual International algorithm. Applied Statistics, 28(1):100–108, 1979. Symposium on Computer Architecture, pages 252–263, [8] S. Haykin. Neural Networks: A Comprehensive Washington, DC, USA, June 2006. IEEE Computer Foundation, 2nd edition. Prentice Hall, Upper Saddle Society. River, NJ, 1999. [22] SPEC CPU2006. Standard Performance Evaluation [9] I. Hur and C. Lin. Memory prefetching using adaptive Corporation. www.spec.org/cpu2006. stream detection. In MICRO 39: Proceedings of the [23] SPECweb2009. Standard Performance Evaluation 39th Annual IEEE/ACM International Symposium on Corporation. www.spec.org/web2009. Microarchitecture, pages 397–408, Washington, DC, [24] TPC-C. www.tpc.org/tpcc. USA, 2006. IEEE Computer Society. [25] T. F. Wenisch, S. Somogyi, N. Hardavellas, J. Kim, [10] Intel 64 and IA-32 Architectures Software Developer’s A. Ailamaki, and B. Falsafi. Temporal streaming of Manuals. www.intel.com/products/processor/manuals. shared memory. In ISCA ’05: Proceedings of the 32nd [11] E. ¨Ipek, S. A. McKee, R. Caruana, B. R. de Supinski, annual international symposium on Computer and M. Schulz. Efficiently exploring architectural Architecture, pages 222–233, Washington, DC, USA, design spaces via predictive modeling. In 2005. IEEE Computer Society. ASPLOS-XII: Proceedings of the 12th international [26] R. E. Wunderlich, T. F. Wenisch, B. Falsafi, and J. C. conference on Architectural support for programming Hoe. Smarts: Accelerating microarchitecture languages and operating systems, pages 195–206, New simulation via rigorous statistical sampling. In York, NY, USA, 2006. ACM. Proceedings of the 30th annual international [12] G. H. John and P. Langley. Estimating continuous symposium on Computer architecture, pages 84–97. distributions in bayesian classifiers. In Proceedings of ACM Press, 2003. the Eleventh Conference on Uncertainty in Artificial Intelligence, pages 338–345. Morgan Kaufmann, August 1995. [13] D. Joseph and D. Grunwald. Prefetching using markov predictors. In ISCA ’97: Proceedings of the 24th annual international symposium on Computer architecture, pages 252–263, New York, NY, USA, 1997. ACM. [14] B. C. Lee and D. Brooks. Efficiency trends and limits from comprehensive microarchitectural adaptivity. In ASPLOS XIII: Proceedings of the 13th international conference on Architectural support for programming languages and operating systems, pages 36–47, New York, NY, USA, 2008. ACM. [15] B. C. Lee and D. M. Brooks. Accurate and efficient regression modeling for microarchitectural performance and power prediction. In ASPLOS-XII: