Tuning for Systematic Trading: Talk 2: Deep Learning
0 likes125 views
The document presents a detailed overview of SigOpt's advanced features that optimize training and tuning for deep learning models, focusing on efficient experiment management and resource utilization. Key topics include Bayesian optimization, async parallel optimization, and strategies for reducing training costs. The talk aims to enhance model performance while minimizing compute budgets and time-to-market for complex models.
![SigOpt. Confidential.
Parallel function evaluations: find the best set of suggestions
20
Parallel function evaluations are a way of efficiently maximizing a function
while using all available compute resources [Ginsbourger et al, 2008, Garcia-Barcos et al. 2019].
• Choosing points by jointly maximizing criteria over the entire set of open resources
• Asynchronously evaluating over a collection of points
• Fixing points which are currently being evaluated while sampling new ones
Jointly Optimize Multiple Next Points to Sample
1D - Acquisition Function 2D - Acquisition Function](https://image.slidesharecdn.com/quantseries-4-200422175240/85/Tuning-for-Systematic-Trading-Talk-2-Deep-Learning-20-320.jpg)
![SigOpt. Confidential.
Using cheap or free information to speed learning
We can build better models using inaccurate data to help point the
actual optimization in the right direction with less cost.
• Using a warm start through multi-task learning logic [Swersky et al, 2014]
• Combining good anytime performance with active learning [Klein et al, 2018]
• Accepting data from multiple sources without priors [Poloczek et al, 2017]
29
Expensive Training Cost](https://image.slidesharecdn.com/quantseries-4-200422175240/85/Tuning-for-Systematic-Trading-Talk-2-Deep-Learning-29-320.jpg)
Tuning for Systematic Trading: Talk 2: Deep Learning
- 1. SigOpt. Confidential. Talk #2 Optimize Training and Tuning for Deep Learning SigOpt Talk Series Tuning for Systematic Trading Tobias Andreasen — Machine Learning Engineer Tuesday, April 21, 2020
- 2. SigOpt. Confidential. Abstract SigOpt provides an extensive set of advanced features, which help you, the expert, save time while increasing model performance via experimentation. Today, we will continue this talk series by discussing how to best utilize your infrastructure, reduce experiment time and accelerate training for deep learning models.
- 3. SigOpt. Confidential. Motivation 1. Overview of SigOpt 2. Recap on bayesian optimization 3. How to continuously and efficiently utilize your project’s allotted compute infrastructure 4. How to tune models with expensive training costs
- 4. SigOpt. Confidential. Overview of SigOpt1
- 5. SigOpt. Confidential. Accelerate and amplify the impact of modelers everywhere
- 6. SigOpt. Confidential. Experiment Insights Optimization Engine Track, analyze and reproduce any model to improve the productivity of your modeling Enterprise Platform Automate hyperparameter tuning to maximize the performance and impact of your models Standardize experimentation across any combination of library, infrastructure, model or task On-Premise Hybrid/Multi Solution: Experiment, optimize and analyze at scale 6
- 7. SigOpt. Confidential. SigOpt Features Enterprise Platform Optimization Engine Experiment Insights Reproducibility Intuitive web dashboards Cross-team permissions and collaboration Advanced experiment visualizations Usage insights Parameter importance analysis Multimetric optimization Continuous, categorical, or integer parameters Constraints and failure regions Up to 10k observations, 100 parameters Multitask optimization and high parallelism Training Monitor and Automated Early Stopping Infrastructure agnostic REST API Parallel Resource Scheduler Black-Box Interface Tunes without accessing any data Libraries for Python, Java, R, and MATLAB
- 8. SigOpt. Confidential. Recap on bayesian optimization2
- 9. SigOpt. Confidential. Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters Black Box Optimization
- 10. SigOpt. Confidential. A graphical depiction of the iterative process 10 Build a statistical model Sequential Model Based Optimization (SMBO)
- 11. SigOpt. Confidential. A graphical depiction of the iterative process 11 Build a statistical model Choose the next point to maximize the acquisition function Sequential Model Based Optimization (SMBO)
- 12. SigOpt. Confidential. A graphical depiction of the iterative process 12 Build a statistical model Build a statistical model Choose the next point to maximize the acquisition function Sequential Model Based Optimization (SMBO)
- 13. SigOpt. Confidential. A graphical depiction of the iterative process 13 Build a statistical model Build a statistical model Choose the next point to maximize the acquisition function Sequential Model Based Optimization (SMBO) Choose the next point to maximize the acquisition function
- 14. SigOpt Blog Posts: Intuition Behind Bayesian Optimization Some Relevant Blog Posts ● Intuition Behind Covariance Kernels ● Approximation of Data ● Likelihood for Gaussian Processes ● Profile Likelihood vs. Kriging Variance ● Intuition behind Gaussian Processes ● Dealing with Troublesome Metrics Find more blog posts visit: https://sigopt.com/blog/
- 15. SigOpt. Confidential. How to continuously and efficiently utilize your project’s allotted compute infrastructure 3
- 16. SigOpt. Confidential. Utilize compute by asynchronous parallel optimization SigOpt natively handles Parallel Function Evaluation with the primary goal of minimizing the Overall Wall-Clock Time. Parallelism also provides: • Faster time-to-results — minimized overall wall-clock time • Full resource utilization — asynchronous parallel optimization • Scaling with infrastructure — optimize across the number of available compute resources This is essential to increase Research Productivity by lowering the time-to-results and scaling with available infrastructure. 16 Continuously and efficiently utilize infrastructure
- 17. SigOpt. Confidential. Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters Continuously and efficiently utilize infrastructure
- 18. SigOpt. Confidential. Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Worker Continuously and efficiently utilize infrastructure
- 19. SigOpt. Confidential. EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Worker Continuously and efficiently utilize infrastructure Worker Worker Worker #1 Worker #2 Worker #100
- 20. SigOpt. Confidential. Parallel function evaluations: find the best set of suggestions 20 Parallel function evaluations are a way of efficiently maximizing a function while using all available compute resources [Ginsbourger et al, 2008, Garcia-Barcos et al. 2019]. • Choosing points by jointly maximizing criteria over the entire set of open resources • Asynchronously evaluating over a collection of points • Fixing points which are currently being evaluated while sampling new ones Jointly Optimize Multiple Next Points to Sample 1D - Acquisition Function 2D - Acquisition Function
- 21. SigOpt. Confidential. Parallel optimization: different parallel bandwidth leads to different search 21 Parallel bandwidth = 1 Parallel bandwidth = 2 Parallel bandwidth = 3 Parallel bandwidth = 4 Parallel bandwidth = 5 Next point(s) to evaluate: Parallel bandwidth represent the # of available compute resources Statistical Model More Exploration, More Exploitation: Faster Wall Clock
- 22. Parallelism Use Case ● Category: NLP ● Task: Sentiment Analysis ● Model: CNN ● Data: Rotten Tomatoes Movie Reviews ● Analysis: Predicting Positive vs. Negative Sentiment ● Result: 400x speedup Learn more https://aws.amazon.com/blogs/machine-learning/fast-cnn-tuni ng-with-aws-gpu-instances-and-sigopt/ Use Case: Fast CNN Tuning with AWS GPU Instances
- 23. SigOpt. Confidential. How to tune models with expensive training costs 4
- 24. SigOpt. Confidential. How to efficiently minimize time to optimize any function SigOpt’s multitask feature is an efficient way for modelers to tune model with an expensive training cost with the benefit of: • Faster time-to-market — The ability to bring expensive models into production faster • Reduction in infrastructure cost — Intelligently leverage infrastructure while reducing cost Through novel research SigOpt helps the user lower the overall time-to-market, while reducing the overall compute budget. 24 Expensive Training Cost
- 25. SigOpt. Confidential. Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters Expensive Training Cost
- 26. SigOpt. Confidential. Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Expensive Training Cost
- 27. SigOpt. Confidential. Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Expensive Training Cost
- 28. SigOpt. Confidential. Using cheap or free information to speed learning 28 SigOpt allows to the user to define lower-cost functions in order to quickly optimize expensive functions • Cheaper-cost functions can be flexible (fewer epochs, subsampled data, other custom features) • Use cheaper tasks earlier in the tuning process to explore • Inform more expensive tasks later by exploiting what we learn • In the process, reduce the full time required to tune an expensive model Expensive Training Cost
- 29. SigOpt. Confidential. Using cheap or free information to speed learning We can build better models using inaccurate data to help point the actual optimization in the right direction with less cost. • Using a warm start through multi-task learning logic [Swersky et al, 2014] • Combining good anytime performance with active learning [Klein et al, 2018] • Accepting data from multiple sources without priors [Poloczek et al, 2017] 29 Expensive Training Cost
- 30. Use Case: Image Classification on a Budget Use Case ● Category: Computer Vision ● Task: Image Classification ● Model: CNN ● Data: Stanford Cars Dataset ● Analysis: Architecture Comparison ● Result: 2.4% accuracy gain with a much shallower model Learn more https://mlconf.com/blog/insights-for-building-high-performing- image-classification-models/
- 31. SigOpt. Confidential. Next Talk: Efficient Approaches to Training Automated Early Stopping, Convergence Monitoring
- 32. SigOpt. Confidential. Register for this talk https://tuning.sigopt.com/tuning-for-systematic-trading
- 33. SigOpt. Confidential. Tobias Andreasen | tobias@sigopt.com For more information visit: https://sigopt.com/research/ Questions?
- 34. SigOpt. Confidential. Next talk: How should one think about convergence? 5 - and other approaches to efficient model training techniques
- 35. SigOpt. Confidential. How should one think about convergence? 5
- 36. SigOpt. Confidential. Future talks Convergence Implementation Infrastructure Implementation Use cases Experiment transfor Visit us at OpMl’20 Metric Management Parameter Importance
- 37. SigOpt. Confidential. The best model is found through convergence 37 Think about convergence
- 38. SigOpt. Confidential. Think about convergence The best model is found through convergence