![Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation
ACQUISITION FUNCTION u
measures the expected utility of evaluating the objective
function at a point λnext
exploitation vs. exploration trade-off
(picture adopted from [1])](https://image.slidesharecdn.com/slides-150310074926-conversion-gate01/85/apsis-Automatic-Hyperparameter-Optimization-Framework-for-Machine-Learning-10-320.jpg)
![Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation
OPTIMIZATION - SUCCESSIVELY UPDATING THE GP
1. find
max
λ
(u(λ)) = λnext
max of acquisition
2. Evaluate M at λnext
3. Update the GP
(picture adopted from [1])](https://image.slidesharecdn.com/slides-150310074926-conversion-gate01/85/apsis-Automatic-Hyperparameter-Optimization-Framework-for-Machine-Learning-11-320.jpg)
![Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation
NUMERIC PARAMETERS IN apsis
Warping Mechanism
parameters are warped into [0, 1] interval
optimization core can assume a uniform and equal
distribution in [0, 1] space
warping can be user defined
Provided Warpings for
normalization of arbitrary intervals [a, b] into [0, 1] space.
asymptotic parameters, e.g. learning rate asymptotic at 0](https://image.slidesharecdn.com/slides-150310074926-conversion-gate01/85/apsis-Automatic-Hyperparameter-Optimization-Framework-for-Machine-Learning-23-320.jpg)
![Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation
TAKING THE PROJECT TO THE NEXT LEVEL -
BAYESIAN OPTIMIZATION
deal with nominal parameters
try replacing GPs with Student-t processes
try to take tree structured configuration space into account
account for evaluation cost depending on hyperparameter
setting
implement freeze-thaw optimization idea [3]
automated learning of input warping [2]](https://image.slidesharecdn.com/slides-150310074926-conversion-gate01/85/apsis-Automatic-Hyperparameter-Optimization-Framework-for-Machine-Learning-39-320.jpg)
apsis - Automatic Hyperparameter Optimization Framework for Machine Learning
- 1. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation apsis Automated Hyperparameter Optimization Using Bayesian Optimization Frederik Diehl Andreas Jauch March 10, 2015 State March 10, 2015
- 2. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation AGENDA Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation
- 3. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation MOTIVATION Why Hyperparameter Optimization and why automating it? hyperparameter tuning often leads to huge performance gain ”more of an art than a science” reproducibility of published results automatic methods might be better than humans provide ml algorithms to non-expert users
- 4. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation ML PROCESS OVERVIEW
- 5. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation FORMAL PROBLEM DESCRIPTION λ : hyperparameter vector λ = (λ(1) , ..., λ(n) ) L(X, f) : loss function evaluated for model f and dataset x Aλ(X) : learning algorithm with hyperparameter vector λ learning on dataset X Xtrain : training data, Xvalid : validation data, Xtest : test data Ψ(λ) : hyperparameter response function/surface Hyperparameter Optimization Problem ˆλ ≈ argmin λ Λ mean Xi∈Xvalid (L(xi, Aλ(Xtrain)) Ψ(λ) (1) = argmin λ Λ (Ψ(λ)) (2)
- 6. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation KEY PROBLEM PROPERTIES unknown, probably non-convex response surface Ψ no derivative-based optimization possible every evaluation of Ψ is expensive evaluation time of Ψ depends on individual value of λ low effective dimensionality of Ψ which dimensions are important is dataset dependent tree-structured configuration space1 1 not addressed in apsis yet
- 7. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation STATE OF THE ART optimization still manual in many projects grid search most common method often the only provided method by many ml frameworks random search bayesian optimization code of Jasper Snoek et. al. Harvard/Toronto whetlab - bay opt in the cloud
- 8. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation BAYESIAN OPTIMIZATION approximate Ψ(λ) by a surrogate function M(λ) = y surrogate function cheaper to evaluate than Ψ interpret model to find minimization candidates for Ψ evaluate Ψ for promising candidates
- 9. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation BAYESIAN OPTIMIZATION FUNDAMENTALS We need two design choices Surrogate Modelling Function - Gaussian Processes universal approximation very flexible and have many useful properties closed under sampling Acquisition Function Probability of Improvement Expected Improvement
- 10. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation ACQUISITION FUNCTION u measures the expected utility of evaluating the objective function at a point λnext exploitation vs. exploration trade-off (picture adopted from [1])
- 11. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation OPTIMIZATION - SUCCESSIVELY UPDATING THE GP 1. find max λ (u(λ)) = λnext max of acquisition 2. Evaluate M at λnext 3. Update the GP (picture adopted from [1])
- 12. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation FITTING THE GP TO THE PROBLEM by tuning the covariance! Squared Exponential Kernel KSE(λ, λ ) = exp − 1 2l2 · 1..dim(D) (λd − λd)2 use Automatic Relevance Determination (ARD) KSE(λ, λ ) = θ0 · exp − 1 2 · 1..dim(D) 1 θd 2 (λd − λd)2 with ARD vector θ θ = ( θ0 bias , θ1, . . . , θd dimension weights )
- 13. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation HOW DO ACQUISITION FUNCTIONS LOOK LIKE? Expected Improvement (EI) uEI(λ) = −∞ ∞ max(Ψ(λ∗ ) − y, 0) · pM(y|λ) dy closed form solution for GPs available uEI(λ|Mt) = σ(λ) · f(λ∗) − µ(λ) σ(λ) · Φ(λ) + φ(λ) gradient analytically derived in apsis for more effective optimization
- 14. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation THE apsis TOOLKIT Automated Hyperparameter Optimization Framework for random search bayesian optimization as an open source framework featuring flexible architecture, ready to be extended for more optimizers ready for use with scikit-learn and theano implemented in Python
- 15. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation PROJECT OBJECTIVES open source implementation of state of the art research in bayesian optimization extendible project to encourage collaboration with other researchers easy integration with existing machine learning frameworks multi core support
- 16. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation apsis ARCHITECTURE OVERVIEW
- 17. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation apsis CORE MODEL COMPONENTS Parameter Definitions define the meta information for each hyperparameter Candidates represent a specific hyperparameter vector and its value holds function value if available Experiments represent an optimization object keeps track of finished and unfinished Candidates
- 18. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation USING apsis - EXPERIMENT ASSISTANTS single experiment interaction interface provides plots and result bookkeeping
- 19. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation USING apsis - LAB ASSISTANTS multiple experiments to compare different optimization techniques cross validation
- 20. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation EXTENSIVE EXPERIMENT TRACKING automated plot writing automated results writing write out information at every step
- 21. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation AUTOMATED PLOTTING plot function evaluations and best results plot confidence bars when using cross validation write out plots at every step
- 22. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation PARAMETER REPRESENTATION IN apsis different representation by parameter type various nominal and numeric types
- 23. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation NUMERIC PARAMETERS IN apsis Warping Mechanism parameters are warped into [0, 1] interval optimization core can assume a uniform and equal distribution in [0, 1] space warping can be user defined Provided Warpings for normalization of arbitrary intervals [a, b] into [0, 1] space. asymptotic parameters, e.g. learning rate asymptotic at 0
- 24. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation NOMINAL PARAMETERS IN apsis generally supported in apsis no support in Bayesian Optimization GP kernels based on distance metrics between parameters interesting topic for further research no publications on this topic so far whetlab pretends to deal well with them - but doesn’t say how
- 25. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation EXPECTED IMPROVEMENT OPTIMIZATION gradient analytically derived2 1000 Steps Random Search for Initialization Several iterative optimization methods integrated L-BFGS-B Bounded Low Memory Quasi Newton Method BFGS Quasi Newton Method Nelder-Mead Inexact Newton with Conjugate Gradient Solver ... 2 See our paper for derivation.
- 26. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation DEALING WITH GP HYPERPARAMETERS the GP surrogate model introduces new hyperparameter not subject of optimization ⇒ hyper-hyperparameters optimization by maximum likelihood method integrating over these parameters in the acquisition function using Hybrid Monte Carlo sampling
- 27. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation apsis PROJECT SET UP Open-Source project from the beginning MIT-License active issue tracking PEP-8 code styling convention Fully automated sphinx documentation build on every commit 90% test coverage clear commit messages
- 28. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation apsis GITHUB REPOSITORY Check out http://github.com/FrederikDiehl/apsis !
- 29. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation ISSUE TRACKING AND DISCUSSION
- 30. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation UNIT TESTS 90% overall test coverage 100% in most core components
- 31. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation GOOD CODE DOCUMENTATION almost 50:50 ratio of code vs. doc documented according to sphinx standard
- 32. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation FULLY AUTOMATED DOCUMENTATION BUILD builds on every commit Visit http://apsis.readthedocs.org !
- 33. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation BRANIN HOO OPTIMIZATION Best Value by Optimizer Values by Optimizer random search finds better end result but bay opt is more stable similar performance as in other bay opt literature no other group publishes comparison to random search
- 34. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation OPTIMIZING ARTIFICIAL NOISE FUNCTION One dimensional noise function with several smoothing variances
- 35. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation OPTIMIZING ARTIFICIAL NOISE FUNCTION Minimization result on 3d noise by smoothing factor.
- 36. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation BREZE MNIST NEURAL NETWORK Neural Network on MNIST using uniform parameters
- 37. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation BREZE MNIST NEURAL NETWORK (2) Neural Network on MNIST using asymptotic parameters for learning rate and learning rate decay
- 38. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation TAKING THE PROJECT TO THE NEXT LEVEL - PROGRAM implement full multicore support implement a REST web-service to offer interoperability with any language improve integration of matplotlib
- 39. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation TAKING THE PROJECT TO THE NEXT LEVEL - BAYESIAN OPTIMIZATION deal with nominal parameters try replacing GPs with Student-t processes try to take tree structured configuration space into account account for evaluation cost depending on hyperparameter setting implement freeze-thaw optimization idea [3] automated learning of input warping [2]
- 40. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation Thank You!
- 41. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation REFERENCES I Eric Brochu, Vlad M. Cora, and Nando de Freitas. A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning. IEEE Transactions on Reliability, abs/1012.2, 2010. Jasper Snoek, Kevin Swersky, Richard S Zemel, and Ryan P Adams. Input warping for bayesian optimization of non-stationary functions. arXiv preprint arXiv:1402.0929, 2014. Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. Freeze-thaw bayesian optimization. arXiv preprint arXiv:1406.3896, 2014.
- 42. Problem Description Bayesian Optimization apsis and its Architecture Project Organisation Performance Evaluation REFERENCES II