SlideShare a Scribd company logo

Tuning for Systematic Trading: Talk 3: Training, Tuning, and Metric Strategy

SigOpt, profile picture
SigOpt

The document discusses advanced features of SigOpt, focusing on training, tuning, and metric strategies to enhance model performance through systematic experimentation. It outlines a comprehensive experimentation recipe, including project hygiene, data handling, evaluation strategies, and optimization loop processes. Key use cases highlight the platform's ability to operate with various modeling techniques while maximizing organizational efficiency.

Technology
1 of 38
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
SigOpt. Confidential. Talk #3 Training, Tuning, and Metric Strategy SigOpt Talk Series Tuning for Systematic Trading Tobias Andreasen — Machine Learning Engineer Tuesday, May 19, 2020
SigOpt. Confidential. Abstract SigOpt provides an extensive set of advanced features, which help you, the expert, save time while increasing model performance via experimentation. Today, we will continue this talk series by discussing training, tuning, and metric strategies in the context of SigOpt, and use this to come up with a recipe for how this can be used for model experimentation.
SigOpt. Confidential. Overview of SigOpt
SigOpt. Confidential. Accelerate and amplify the impact of modelers everywhere
SigOpt. Confidential. Experiment Insights Optimization Engine Track, analyze and reproduce any model to improve the productivity of your modeling Enterprise Platform Automate hyperparameter tuning to maximize the performance and impact of your models Standardize experimentation across any combination of library, infrastructure, model or task On-Premise Hybrid/Multi Solution: Experiment, optimize and analyze at scale 5
SigOpt. Confidential. SigOpt Features Enterprise Platform Optimization Engine Experiment Insights Reproducibility Intuitive web dashboards Cross-team permissions and collaboration Advanced experiment visualizations Usage insights Parameter importance analysis Multimetric optimization Continuous, categorical, or integer parameters Constraints and failure regions Up to 10k observations, 100 parameters Multitask optimization and high parallelism Training Monitor and Automated Early Stopping Infrastructure agnostic REST API Parallel Resource Scheduler Black-Box Interface Tunes without accessing any data Libraries for Python, Java, R, and MATLAB
SigOpt. Confidential. Experimentation Recipe
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases Black Box Optimization Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff 8
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 9 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff class SIGOPT: APITOKEN = 'PPPPPOJUZPFSAAAAADSSEYPCPNPGGGGGLNLRYKSRBWYZTTFI' CLIENTID = 1 PROJECT = 'quant_series' Creating SigOpt Project SigOpt Access Class
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters 10 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Load Data
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 11 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff class Data: X, Y = datasets.load_breast_cancer(return_X_y=True) def __init__(self, test_size=0.3): self.x_train, self.x_test, self.y_train, self.y_test = train_test_split( self.X, self.Y, test_size=test_size) def train_data(self,): return [self.x_train, self.y_train] def test_data(self,): return [self.x_test, self.y_test] def num_features(self,): return self.X.shape[1] def num_classes(self,): return len(np.unique(self.Y)) Load Data Data Loader Class
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters 12 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Create Evaluation Framework
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 13 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff class Evaluation: METRICS = [{'name': 'accuracy', 'objective': 'maximize', 'strategy': 'optimize'}, {'name': 'f1', 'objective': 'maximize', 'strategy': 'store'}, {'name': 'precision', 'objective': 'maximize', 'strategy': 'store'}, {'name': 'recall', 'objective': 'maximize', 'strategy': 'store'}, ] @classmethod def return_object(cls, y): y_true = y[0] y_hat = y[1] # needs to be replaced by data input and score calculations scores = [] scores.append(accuracy_score(y_true, y_hat)) scores.append(f1_score(y_true, y_hat, average='macro')) scores.append(precision_score( y_true, y_hat, average='macro', zero_division=1)) scores.append(recall_score(y_true, y_hat, average='macro')) return [{'name': m['name'], 'value': s} for m, s in zip(cls.METRICS, scores)] Create Evaluation Framework Model Evaluation Class
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters 14 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Create Model
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 15 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff class LogisticRegression: NAME = 'Logistic Regression' BUDGET = 10 PARAMETERS = [{'name': 'log_c', 'type': 'double', 'bounds': {'min': -11, 'max': 0}, 'grid': BUDGET}] def __init__(self, assignment): c = np.exp(assignment['log_c']) self.model = LogisticRegression(C=c) self.y_hat = None self.y_true = None def train(self, data): x = data[0] y = data[1] self.model.fit(x, y) def test(self, data): x = data[0] y = data[1] self.y_hat = self.model.predict(x) self.y_true = y def evaluate(self,): return [self.y_true, self.y_hat] Create Model Baseline Model Class
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters 16 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Optimization Loop
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 17 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff conn = Connection(client_token=SIGOPT.APITOKEN) experiment_meta = { 'type': 'grid', 'project': SIGOPT.PROJECT, 'name': Model.NAME, 'parameters': Model.PARAMETERS, 'metrics': Evaluation.METRICS} experiment = conn.experiments().create(**experiment_meta) while experiment.progress.observation_count < experiment.observation_budget: suggestion = conn.experiments(experiment.id).suggestions().create() model = Model(suggestion.assignments) model.train(data.train_data()) model.test(data.test_data()) conn.experiments(experiment.id).observations().create( suggestion=suggestion.id, values=Evaluation.return_object(model.evaluate()),) experiment = conn.experiments(experiment.id).fetch() Optimization Loop Experiment Create Call Optimization Loop
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 18 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff conn = Connection(client_token=SIGOPT.APITOKEN) experiment_meta = { 'type': 'grid', 'project': SIGOPT.PROJECT, 'name': Model.NAME, 'parameters': Model.PARAMETERS, 'metrics': Evaluation.METRICS} experiment = conn.experiments().create(**experiment_meta) while experiment.progress.observation_count < experiment.observation_budget: suggestion = conn.experiments(experiment.id).suggestions().create() model = Model(suggestion.assignments) model.train(data.train_data()) model.test(data.test_data()) conn.experiments(experiment.id).observations().create( suggestion=suggestion.id, values=Evaluation.return_object(model.evaluate()),) experiment = conn.experiments(experiment.id).fetch() Optimization Loop Experiment Create Call Optimization Loop
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 19 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff conn = Connection(client_token=SIGOPT.APITOKEN) experiment_meta = { 'type': 'grid', 'project': SIGOPT.PROJECT, 'name': Model.NAME, 'parameters': Model.PARAMETERS, 'metrics': Evaluation.METRICS} experiment = conn.experiments().create(**experiment_meta) while experiment.progress.observation_count < experiment.observation_budget: suggestion = conn.experiments(experiment.id).suggestions().create() model = Model(suggestion.assignments) model.train(data.train_data()) model.test(data.test_data()) conn.experiments(experiment.id).observations().create( suggestion=suggestion.id, values=Evaluation.return_object(model.evaluate()),) experiment = conn.experiments(experiment.id).fetch() Optimization Loop Experiment Create Call Optimization Loop
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 20 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Metric Strategy - Storing up to 50 non-optimized metrics - Gain insights through additional model information Objective - Storing business specific metrics to drive the best decisions - Hit business targets through modeling Adding Metrics
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 21 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Adding Metrics
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 22 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Investigating Metrics
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 23 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Investigating Metrics
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 24 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Metric Constraints - Guardrail your modeling by applying constraints to non-optimized metrics Objective - Use business requirements as constraints for your modeling - Maximize modeling impact Adding Metric Constraints
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 25 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Adding Metric Constraints
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters 26 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Create Model
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 27 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff class NeuralNetwork: NAME = 'Neural Network' BUDGET = 30 EPOCHS = 40 MAXCHECKPOINTS = {'max_checkpoints': EPOCHS, } NUMFEATURES = None NUMCLASSES = None PARAMETERS = [{'name': 'num_nodes', 'type': 'int', 'bounds': {'min': 4, 'max': 40}}, {'name': 'log_learning_rate', 'type': 'double', 'bounds': {'min': -11, 'max': 0}}, {'name': 'batch_size', 'type': 'int', 'bounds': {'min': 4, 'max': 10}}] def __init__(self, assignment): self.num_nodes = assignment['num_nodes'] self.learning_rate = np.exp(assignment['log_learning_rate']) self.batch_size = assignment['batch_size'] self.model = keras.Sequential( [layers.Dense(self.num_nodes, activation='relu', input_shape=(self.NUMFEATURES,)), layers.Dense(self.NUMCLASSES, activation='softmax'), ]) opt = keras.optimizers.Adam(learning_rate=self.learning_rate) self.model.compile( loss='sparse_categorical_crossentropy', optimizer=opt) Create Model Advanced Model Class
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters 28 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Optimization Loop
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 29 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff conn = Connection(client_token=SIGOPT.APITOKEN) experiment_meta = { 'project': SIGOPT.PROJECT, 'name': Model.NAME, 'parameters': Model.PARAMETERS, 'observation_budget': Model.BUDGET, 'training_monitor': Model.MAXCHECKPOINTS, 'metrics': Evaluation.METRICS,} experiment = conn.experiments().create(**experiment_meta) while experiment.progress.observation_count < experiment.observation_budget: suggestion = conn.experiments(experiment.id).suggestions().create() model = Model(suggestion.assignments) training_run = conn.experiments( experiment.id).training_runs().create(suggestion=suggestion.id) for _ in range(model.EPOCHS): model.train(data.train_data()) model.test(data.test_data()) checkpoint = conn.experiments(experiment.id).training_runs( training_run.id).checkpoints().create(values=Evaluation.return_object(model.evaluate()),) Optimization Loop Experiment Create Call Optimization Loop with Training Monitor
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 30 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff conn = Connection(client_token=SIGOPT.APITOKEN) experiment_meta = { 'project': SIGOPT.PROJECT, 'name': Model.NAME, 'parameters': Model.PARAMETERS, 'observation_budget': Model.BUDGET, 'training_monitor': Model.MAXCHECKPOINTS, 'metrics': Evaluation.METRICS,} experiment = conn.experiments().create(**experiment_meta) while experiment.progress.observation_count < experiment.observation_budget: suggestion = conn.experiments(experiment.id).suggestions().create() model = Model(suggestion.assignments) training_run = conn.experiments( experiment.id).training_runs().create(suggestion=suggestion.id) for _ in range(model.EPOCHS): model.train(data.train_data()) model.test(data.test_data()) checkpoint = conn.experiments(experiment.id).training_runs( training_run.id).checkpoints().create(values=Evaluation.return_object(model.evaluate()),) Optimization Loop Experiment Create Call Optimization Loop with Training Monitor
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases Optimization Loop with Training Monitor 31 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff conn = Connection(client_token=SIGOPT.APITOKEN) experiment_meta = { 'project': SIGOPT.PROJECT, 'name': Model.NAME, 'parameters': Model.PARAMETERS, 'observation_budget': Model.BUDGET, 'training_monitor': Model.MAXCHECKPOINTS, 'metrics': Evaluation.METRICS,} experiment = conn.experiments().create(**experiment_meta) while experiment.progress.observation_count < experiment.observation_budget: suggestion = conn.experiments(experiment.id).suggestions().create() model = Model(suggestion.assignments) training_run = conn.experiments( experiment.id).training_runs().create(suggestion=suggestion.id) for _ in range(model.EPOCHS): model.train(data.train_data()) model.test(data.test_data()) checkpoint = conn.experiments(experiment.id).training_runs( training_run.id).checkpoints().create(values=Evaluation.return_object(model.evaluate()),) Optimization Loop Experiment Create Call
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 32 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Training Monitor - Monitor convergence for better model performance - Increase generalization by seeing best convergence Objective - Decide on the best model for production by performance as well as convergence Looking at Convergence
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 33 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Looking at Convergence
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 34 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff while experiment.progress.observation_count < experiment.observation_budget: suggestion = conn.experiments(experiment.id).suggestions().create() model = Model(suggestion.assignments) training_run = conn.experiments( experiment.id).training_runs().create(suggestion=suggestion.id) for _ in range(model.EPOCHS): model.train(data.train_data()) model.test(data.test_data()) checkpoint = conn.experiments(experiment.id).training_runs( training_run.id).checkpoints().create(values=Evaluation.return_object(model.evaluate()),) if checkpoint.should_stop: conn.experiments(experiment.id).observations().create( training_run=training_run.id) break Looking at Convergence Optimization Loop with Training Monitor and Early Stopping
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 35 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff while experiment.progress.observation_count < experiment.observation_budget: suggestion = conn.experiments(experiment.id).suggestions().create() model = Model(suggestion.assignments) training_run = conn.experiments( experiment.id).training_runs().create(suggestion=suggestion.id) for _ in range(model.EPOCHS): model.train(data.train_data()) model.test(data.test_data()) checkpoint = conn.experiments(experiment.id).training_runs( training_run.id).checkpoints().create(values=Evaluation.return_object(model.evaluate()),) if checkpoint.should_stop: conn.experiments(experiment.id).observations().create( training_run=training_run.id) break Looking at Convergence Optimization Loop with Training Monitor and Early Stopping
SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 36 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff class SIGOPT: APITOKEN = 'DBOKLMJUZPFSYYCCEDSSEYPNPNPPLOWHLNLRYKSRBDYZIIFI' CLIENTID = 1 PROJECT = 'quant_series' Finishing up Project SigOpt Access Class
SigOpt. Confidential.SigOpt. Confidential. Key Use Casesclass SIGOPT: APITOKEN = 'DBOKLMJUZPFSYYCCEDSSEYPNPNPPLOWHLNLRYKSRBDYZIIFI' CLIENTID = 1 PROJECT = 'quant_series' 37 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Finishing up Project SigOpt Access Class
SigOpt. Confidential. Tobias Andreasen | tobias@sigopt.com For more information visit: https://sigopt.com/research/ Questions?

More Related Content

PDF
Tuning for Systematic Trading: Talk 2: Deep Learning
SigOpt
 
PDF
Tuning for Systematic Trading: Talk 1
SigOpt
 
PDF
SigOpt for Hedge Funds
SigOpt
 
PDF
SigOpt at O'Reilly - Best Practices for Scaling Modeling Platforms
SigOpt
 
PDF
SigOpt at Ai4 Finance—Modeling at Scale
SigOpt
 
PDF
Modeling at scale in systematic trading
SigOpt
 
PDF
SigOpt at GTC - Reducing operational barriers to optimization
SigOpt
 
PDF
Advanced Optimization for the Enterprise Webinar
SigOpt
 
Tuning for Systematic Trading: Talk 2: Deep Learning
SigOpt
 
Tuning for Systematic Trading: Talk 1
SigOpt
 
SigOpt for Hedge Funds
SigOpt
 
SigOpt at O'Reilly - Best Practices for Scaling Modeling Platforms
SigOpt
 
SigOpt at Ai4 Finance—Modeling at Scale
SigOpt
 
Modeling at scale in systematic trading
SigOpt
 
SigOpt at GTC - Reducing operational barriers to optimization
SigOpt
 
Advanced Optimization for the Enterprise Webinar
SigOpt
 

What's hot (15)

PDF
Tuning the Untunable - Insights on Deep Learning Optimization
SigOpt
 
PDF
Common Problems in Hyperparameter Optimization
SigOpt
 
PPTX
Machine Learning Fundamentals
SigOpt
 
PDF
MLOps Using MLflow
Databricks
 
PPTX
Pydata presentation
Thomas Huijskens
 
PDF
Pentaho World 2017: Automated Machine Learning (AutoML) and Pentaho (Thursday...
Caio Moreno
 
PDF
Model-Based Optimization for Effective and Reliable Decision-Making
Bob Fourer
 
PDF
PyData London 2018 talk on feature selection
Thomas Huijskens
 
PDF
Agile Machine Learning for Real-time Recommender Systems
Johann Schleier-Smith
 
PDF
“Houston, we have a model...” Introduction to MLOps
Rui Quintino
 
PDF
Recent advances on large scheduling problems in CP Optimizer
Philippe Laborie
 
PDF
Shortening the time from analysis to deployment with ml as-a-service — Luiz A...
PAPIs.io
 
PPTX
MLOps - The Assembly Line of ML
Jordan Birdsell
 
PDF
Nesma autumn conference 2015 - Is FPA a valuable addition to predictable agil...
Nesma
 
PDF
Model-Based Optimization / CP 2018
Bob Fourer
 
Tuning the Untunable - Insights on Deep Learning Optimization
SigOpt
 
Common Problems in Hyperparameter Optimization
SigOpt
 
Machine Learning Fundamentals
SigOpt
 
MLOps Using MLflow
Databricks
 
Pydata presentation
Thomas Huijskens
 
Pentaho World 2017: Automated Machine Learning (AutoML) and Pentaho (Thursday...
Caio Moreno
 
Model-Based Optimization for Effective and Reliable Decision-Making
Bob Fourer
 
PyData London 2018 talk on feature selection
Thomas Huijskens
 
Agile Machine Learning for Real-time Recommender Systems
Johann Schleier-Smith
 
“Houston, we have a model...” Introduction to MLOps
Rui Quintino
 
Recent advances on large scheduling problems in CP Optimizer
Philippe Laborie
 
Shortening the time from analysis to deployment with ml as-a-service — Luiz A...
PAPIs.io
 
MLOps - The Assembly Line of ML
Jordan Birdsell
 
Nesma autumn conference 2015 - Is FPA a valuable addition to predictable agil...
Nesma
 
Model-Based Optimization / CP 2018
Bob Fourer
 
Ad

Similar to Tuning for Systematic Trading: Talk 3: Training, Tuning, and Metric Strategy (20)

PDF
Modeling at Scale: SigOpt at TWIMLcon 2019
SigOpt
 
PDF
Tuning 2.0: Advanced Optimization Techniques Webinar
SigOpt
 
PDF
Metric Management: a SigOpt Applied Use Case
SigOpt
 
PDF
Advanced MLflow: Multi-Step Workflows, Hyperparameter Tuning and Integrating ...
Databricks
 
PDF
SigOpt for Machine Learning and AI
SigOpt
 
PDF
Building successful and secure products with AI and ML
Simon Lia-Jonassen
 
PPTX
CNCF-Istanbul-MLOps for Devops Engineers.pptx
cansukavili1
 
PDF
SigOpt at GTC - Tuning the Untunable
SigOpt
 
PPSX
Selling commercial Solar +/or Energy Storage solutions? You need GridMAP!
Iain Beveridge
 
PDF
Building the Ideal Stack for Machine Learning
SingleStore
 
PPT
principle of six sigma
Lyceum of The Philippines University
 
PDF
How to Run a Cost-Efficient Optimization Program With a Limited Budget
VWO
 
PPTX
MLOps and Reproducible ML on AWS with Kubeflow and SageMaker
Provectus
 
PDF
Failure is an Option: Scaling Resilient Feature Delivery
Optimizely
 
PPT
Strata CA 2019: From Jupyter to Production Manu Mukerji
Manu Mukerji
 
PPTX
PureApp Presentation
Prolifics
 
PDF
How to Lower the Cost of Deploying Analytics: An Introduction to the Portable...
Robert Grossman
 
PPT
OTS - Everything you wanted to know but didn't ask
Jeff Hackney
 
PPT
Design for six_sigma
mangal das
 
PPTX
Simulacion luis garciaguzman-21012011
lideresacademicos
 
Modeling at Scale: SigOpt at TWIMLcon 2019
SigOpt
 
Tuning 2.0: Advanced Optimization Techniques Webinar
SigOpt
 
Metric Management: a SigOpt Applied Use Case
SigOpt
 
Advanced MLflow: Multi-Step Workflows, Hyperparameter Tuning and Integrating ...
Databricks
 
SigOpt for Machine Learning and AI
SigOpt
 
Building successful and secure products with AI and ML
Simon Lia-Jonassen
 
CNCF-Istanbul-MLOps for Devops Engineers.pptx
cansukavili1
 
SigOpt at GTC - Tuning the Untunable
SigOpt
 
Selling commercial Solar +/or Energy Storage solutions? You need GridMAP!
Iain Beveridge
 
Building the Ideal Stack for Machine Learning
SingleStore
 
principle of six sigma
Lyceum of The Philippines University
 
How to Run a Cost-Efficient Optimization Program With a Limited Budget
VWO
 
MLOps and Reproducible ML on AWS with Kubeflow and SageMaker
Provectus
 
Failure is an Option: Scaling Resilient Feature Delivery
Optimizely
 
Strata CA 2019: From Jupyter to Production Manu Mukerji
Manu Mukerji
 
PureApp Presentation
Prolifics
 
How to Lower the Cost of Deploying Analytics: An Introduction to the Portable...
Robert Grossman
 
OTS - Everything you wanted to know but didn't ask
Jeff Hackney
 
Design for six_sigma
mangal das
 
Simulacion luis garciaguzman-21012011
lideresacademicos
 
Ad

More from SigOpt (14)

PDF
Optimizing BERT and Natural Language Models with SigOpt Experiment Management
SigOpt
 
PDF
Experiment Management for the Enterprise
SigOpt
 
PDF
Efficient NLP by Distilling BERT and Multimetric Optimization
SigOpt
 
PDF
Detecting COVID-19 Cases with Deep Learning
SigOpt
 
PDF
Tuning Data Augmentation to Boost Model Performance
SigOpt
 
PDF
Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...
SigOpt
 
PDF
Machine Learning Infrastructure
SigOpt
 
PDF
SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...
SigOpt
 
PDF
Lessons for an enterprise approach to modeling at scale
SigOpt
 
PDF
SigOpt at MLconf - Reducing Operational Barriers to Model Training
SigOpt
 
PDF
Machine Learning Infrastructure
SigOpt
 
PPTX
Tips and techniques for hyperparameter optimization
SigOpt
 
PDF
MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...
SigOpt
 
PDF
Using Optimal Learning to Tune Deep Learning Pipelines
SigOpt
 
Optimizing BERT and Natural Language Models with SigOpt Experiment Management
SigOpt
 
Experiment Management for the Enterprise
SigOpt
 
Efficient NLP by Distilling BERT and Multimetric Optimization
SigOpt
 
Detecting COVID-19 Cases with Deep Learning
SigOpt
 
Tuning Data Augmentation to Boost Model Performance
SigOpt
 
Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...
SigOpt
 
Machine Learning Infrastructure
SigOpt
 
SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...
SigOpt
 
Lessons for an enterprise approach to modeling at scale
SigOpt
 
SigOpt at MLconf - Reducing Operational Barriers to Model Training
SigOpt
 
Machine Learning Infrastructure
SigOpt
 
Tips and techniques for hyperparameter optimization
SigOpt
 
MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...
SigOpt
 
Using Optimal Learning to Tune Deep Learning Pipelines
SigOpt
 

Recently uploaded (20)

PDF
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PPTX
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
AVTRON Technologies LLC
 
PDF
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
IAESIJAI
 
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
PPTX
Cloud computing and distributed systems.
kkkkkhan
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
PDF
NewMind AI Monthly Chronicles - July 2025
NewMind AI
 
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
PDF
GDG Cloud Iasi [PUBLIC] Florian Blaga - Unveiling the Evolution of Cybersecur...
athlonica
 
PDF
Advanced IT Governance
University of Hertfordshire
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PDF
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
AVTRON Technologies LLC
 
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
IAESIJAI
 
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
Cloud computing and distributed systems.
kkkkkhan
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
NewMind AI Monthly Chronicles - July 2025
NewMind AI
 
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
GDG Cloud Iasi [PUBLIC] Florian Blaga - Unveiling the Evolution of Cybersecur...
athlonica
 
Advanced IT Governance
University of Hertfordshire
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 

Tuning for Systematic Trading: Talk 3: Training, Tuning, and Metric Strategy

  • 1. SigOpt. Confidential. Talk #3 Training, Tuning, and Metric Strategy SigOpt Talk Series Tuning for Systematic Trading Tobias Andreasen — Machine Learning Engineer Tuesday, May 19, 2020
  • 2. SigOpt. Confidential. Abstract SigOpt provides an extensive set of advanced features, which help you, the expert, save time while increasing model performance via experimentation. Today, we will continue this talk series by discussing training, tuning, and metric strategies in the context of SigOpt, and use this to come up with a recipe for how this can be used for model experimentation.
  • 4. SigOpt. Confidential. Accelerate and amplify the impact of modelers everywhere
  • 5. SigOpt. Confidential. Experiment Insights Optimization Engine Track, analyze and reproduce any model to improve the productivity of your modeling Enterprise Platform Automate hyperparameter tuning to maximize the performance and impact of your models Standardize experimentation across any combination of library, infrastructure, model or task On-Premise Hybrid/Multi Solution: Experiment, optimize and analyze at scale 5
  • 6. SigOpt. Confidential. SigOpt Features Enterprise Platform Optimization Engine Experiment Insights Reproducibility Intuitive web dashboards Cross-team permissions and collaboration Advanced experiment visualizations Usage insights Parameter importance analysis Multimetric optimization Continuous, categorical, or integer parameters Constraints and failure regions Up to 10k observations, 100 parameters Multitask optimization and high parallelism Training Monitor and Automated Early Stopping Infrastructure agnostic REST API Parallel Resource Scheduler Black-Box Interface Tunes without accessing any data Libraries for Python, Java, R, and MATLAB
  • 8. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases Black Box Optimization Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff 8
  • 9. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 9 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff class SIGOPT: APITOKEN = 'PPPPPOJUZPFSAAAAADSSEYPCPNPGGGGGLNLRYKSRBWYZTTFI' CLIENTID = 1 PROJECT = 'quant_series' Creating SigOpt Project SigOpt Access Class
  • 10. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters 10 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Load Data
  • 11. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 11 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff class Data: X, Y = datasets.load_breast_cancer(return_X_y=True) def __init__(self, test_size=0.3): self.x_train, self.x_test, self.y_train, self.y_test = train_test_split( self.X, self.Y, test_size=test_size) def train_data(self,): return [self.x_train, self.y_train] def test_data(self,): return [self.x_test, self.y_test] def num_features(self,): return self.X.shape[1] def num_classes(self,): return len(np.unique(self.Y)) Load Data Data Loader Class
  • 12. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters 12 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Create Evaluation Framework
  • 13. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 13 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff class Evaluation: METRICS = [{'name': 'accuracy', 'objective': 'maximize', 'strategy': 'optimize'}, {'name': 'f1', 'objective': 'maximize', 'strategy': 'store'}, {'name': 'precision', 'objective': 'maximize', 'strategy': 'store'}, {'name': 'recall', 'objective': 'maximize', 'strategy': 'store'}, ] @classmethod def return_object(cls, y): y_true = y[0] y_hat = y[1] # needs to be replaced by data input and score calculations scores = [] scores.append(accuracy_score(y_true, y_hat)) scores.append(f1_score(y_true, y_hat, average='macro')) scores.append(precision_score( y_true, y_hat, average='macro', zero_division=1)) scores.append(recall_score(y_true, y_hat, average='macro')) return [{'name': m['name'], 'value': s} for m, s in zip(cls.METRICS, scores)] Create Evaluation Framework Model Evaluation Class
  • 14. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters 14 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Create Model
  • 15. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 15 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff class LogisticRegression: NAME = 'Logistic Regression' BUDGET = 10 PARAMETERS = [{'name': 'log_c', 'type': 'double', 'bounds': {'min': -11, 'max': 0}, 'grid': BUDGET}] def __init__(self, assignment): c = np.exp(assignment['log_c']) self.model = LogisticRegression(C=c) self.y_hat = None self.y_true = None def train(self, data): x = data[0] y = data[1] self.model.fit(x, y) def test(self, data): x = data[0] y = data[1] self.y_hat = self.model.predict(x) self.y_true = y def evaluate(self,): return [self.y_true, self.y_hat] Create Model Baseline Model Class
  • 16. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters 16 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Optimization Loop
  • 17. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 17 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff conn = Connection(client_token=SIGOPT.APITOKEN) experiment_meta = { 'type': 'grid', 'project': SIGOPT.PROJECT, 'name': Model.NAME, 'parameters': Model.PARAMETERS, 'metrics': Evaluation.METRICS} experiment = conn.experiments().create(**experiment_meta) while experiment.progress.observation_count < experiment.observation_budget: suggestion = conn.experiments(experiment.id).suggestions().create() model = Model(suggestion.assignments) model.train(data.train_data()) model.test(data.test_data()) conn.experiments(experiment.id).observations().create( suggestion=suggestion.id, values=Evaluation.return_object(model.evaluate()),) experiment = conn.experiments(experiment.id).fetch() Optimization Loop Experiment Create Call Optimization Loop
  • 18. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 18 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff conn = Connection(client_token=SIGOPT.APITOKEN) experiment_meta = { 'type': 'grid', 'project': SIGOPT.PROJECT, 'name': Model.NAME, 'parameters': Model.PARAMETERS, 'metrics': Evaluation.METRICS} experiment = conn.experiments().create(**experiment_meta) while experiment.progress.observation_count < experiment.observation_budget: suggestion = conn.experiments(experiment.id).suggestions().create() model = Model(suggestion.assignments) model.train(data.train_data()) model.test(data.test_data()) conn.experiments(experiment.id).observations().create( suggestion=suggestion.id, values=Evaluation.return_object(model.evaluate()),) experiment = conn.experiments(experiment.id).fetch() Optimization Loop Experiment Create Call Optimization Loop
  • 19. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 19 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff conn = Connection(client_token=SIGOPT.APITOKEN) experiment_meta = { 'type': 'grid', 'project': SIGOPT.PROJECT, 'name': Model.NAME, 'parameters': Model.PARAMETERS, 'metrics': Evaluation.METRICS} experiment = conn.experiments().create(**experiment_meta) while experiment.progress.observation_count < experiment.observation_budget: suggestion = conn.experiments(experiment.id).suggestions().create() model = Model(suggestion.assignments) model.train(data.train_data()) model.test(data.test_data()) conn.experiments(experiment.id).observations().create( suggestion=suggestion.id, values=Evaluation.return_object(model.evaluate()),) experiment = conn.experiments(experiment.id).fetch() Optimization Loop Experiment Create Call Optimization Loop
  • 20. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 20 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Metric Strategy - Storing up to 50 non-optimized metrics - Gain insights through additional model information Objective - Storing business specific metrics to drive the best decisions - Hit business targets through modeling Adding Metrics
  • 21. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 21 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Adding Metrics
  • 22. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 22 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Investigating Metrics
  • 23. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 23 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Investigating Metrics
  • 24. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 24 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Metric Constraints - Guardrail your modeling by applying constraints to non-optimized metrics Objective - Use business requirements as constraints for your modeling - Maximize modeling impact Adding Metric Constraints
  • 25. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 25 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Adding Metric Constraints
  • 26. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters 26 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Create Model
  • 27. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 27 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff class NeuralNetwork: NAME = 'Neural Network' BUDGET = 30 EPOCHS = 40 MAXCHECKPOINTS = {'max_checkpoints': EPOCHS, } NUMFEATURES = None NUMCLASSES = None PARAMETERS = [{'name': 'num_nodes', 'type': 'int', 'bounds': {'min': 4, 'max': 40}}, {'name': 'log_learning_rate', 'type': 'double', 'bounds': {'min': -11, 'max': 0}}, {'name': 'batch_size', 'type': 'int', 'bounds': {'min': 4, 'max': 10}}] def __init__(self, assignment): self.num_nodes = assignment['num_nodes'] self.learning_rate = np.exp(assignment['log_learning_rate']) self.batch_size = assignment['batch_size'] self.model = keras.Sequential( [layers.Dense(self.num_nodes, activation='relu', input_shape=(self.NUMFEATURES,)), layers.Dense(self.NUMCLASSES, activation='softmax'), ]) opt = keras.optimizers.Adam(learning_rate=self.learning_rate) self.model.compile( loss='sparse_categorical_crossentropy', optimizer=opt) Create Model Advanced Model Class
  • 28. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters 28 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Optimization Loop
  • 29. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 29 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff conn = Connection(client_token=SIGOPT.APITOKEN) experiment_meta = { 'project': SIGOPT.PROJECT, 'name': Model.NAME, 'parameters': Model.PARAMETERS, 'observation_budget': Model.BUDGET, 'training_monitor': Model.MAXCHECKPOINTS, 'metrics': Evaluation.METRICS,} experiment = conn.experiments().create(**experiment_meta) while experiment.progress.observation_count < experiment.observation_budget: suggestion = conn.experiments(experiment.id).suggestions().create() model = Model(suggestion.assignments) training_run = conn.experiments( experiment.id).training_runs().create(suggestion=suggestion.id) for _ in range(model.EPOCHS): model.train(data.train_data()) model.test(data.test_data()) checkpoint = conn.experiments(experiment.id).training_runs( training_run.id).checkpoints().create(values=Evaluation.return_object(model.evaluate()),) Optimization Loop Experiment Create Call Optimization Loop with Training Monitor
  • 30. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 30 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff conn = Connection(client_token=SIGOPT.APITOKEN) experiment_meta = { 'project': SIGOPT.PROJECT, 'name': Model.NAME, 'parameters': Model.PARAMETERS, 'observation_budget': Model.BUDGET, 'training_monitor': Model.MAXCHECKPOINTS, 'metrics': Evaluation.METRICS,} experiment = conn.experiments().create(**experiment_meta) while experiment.progress.observation_count < experiment.observation_budget: suggestion = conn.experiments(experiment.id).suggestions().create() model = Model(suggestion.assignments) training_run = conn.experiments( experiment.id).training_runs().create(suggestion=suggestion.id) for _ in range(model.EPOCHS): model.train(data.train_data()) model.test(data.test_data()) checkpoint = conn.experiments(experiment.id).training_runs( training_run.id).checkpoints().create(values=Evaluation.return_object(model.evaluate()),) Optimization Loop Experiment Create Call Optimization Loop with Training Monitor
  • 31. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases Optimization Loop with Training Monitor 31 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff conn = Connection(client_token=SIGOPT.APITOKEN) experiment_meta = { 'project': SIGOPT.PROJECT, 'name': Model.NAME, 'parameters': Model.PARAMETERS, 'observation_budget': Model.BUDGET, 'training_monitor': Model.MAXCHECKPOINTS, 'metrics': Evaluation.METRICS,} experiment = conn.experiments().create(**experiment_meta) while experiment.progress.observation_count < experiment.observation_budget: suggestion = conn.experiments(experiment.id).suggestions().create() model = Model(suggestion.assignments) training_run = conn.experiments( experiment.id).training_runs().create(suggestion=suggestion.id) for _ in range(model.EPOCHS): model.train(data.train_data()) model.test(data.test_data()) checkpoint = conn.experiments(experiment.id).training_runs( training_run.id).checkpoints().create(values=Evaluation.return_object(model.evaluate()),) Optimization Loop Experiment Create Call
  • 32. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 32 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Training Monitor - Monitor convergence for better model performance - Increase generalization by seeing best convergence Objective - Decide on the best model for production by performance as well as convergence Looking at Convergence
  • 33. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 33 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Looking at Convergence
  • 34. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 34 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff while experiment.progress.observation_count < experiment.observation_budget: suggestion = conn.experiments(experiment.id).suggestions().create() model = Model(suggestion.assignments) training_run = conn.experiments( experiment.id).training_runs().create(suggestion=suggestion.id) for _ in range(model.EPOCHS): model.train(data.train_data()) model.test(data.test_data()) checkpoint = conn.experiments(experiment.id).training_runs( training_run.id).checkpoints().create(values=Evaluation.return_object(model.evaluate()),) if checkpoint.should_stop: conn.experiments(experiment.id).observations().create( training_run=training_run.id) break Looking at Convergence Optimization Loop with Training Monitor and Early Stopping
  • 35. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 35 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff while experiment.progress.observation_count < experiment.observation_budget: suggestion = conn.experiments(experiment.id).suggestions().create() model = Model(suggestion.assignments) training_run = conn.experiments( experiment.id).training_runs().create(suggestion=suggestion.id) for _ in range(model.EPOCHS): model.train(data.train_data()) model.test(data.test_data()) checkpoint = conn.experiments(experiment.id).training_runs( training_run.id).checkpoints().create(values=Evaluation.return_object(model.evaluate()),) if checkpoint.should_stop: conn.experiments(experiment.id).observations().create( training_run=training_run.id) break Looking at Convergence Optimization Loop with Training Monitor and Early Stopping
  • 36. SigOpt. Confidential.SigOpt. Confidential. Key Use Cases 36 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff class SIGOPT: APITOKEN = 'DBOKLMJUZPFSYYCCEDSSEYPNPNPPLOWHLNLRYKSRBDYZIIFI' CLIENTID = 1 PROJECT = 'quant_series' Finishing up Project SigOpt Access Class
  • 37. SigOpt. Confidential.SigOpt. Confidential. Key Use Casesclass SIGOPT: APITOKEN = 'DBOKLMJUZPFSYYCCEDSSEYPNPNPPLOWHLNLRYKSRBDYZIIFI' CLIENTID = 1 PROJECT = 'quant_series' 37 Experimentation Recipe: Project Hygiene Data Handling Evaluation Strategy Baseline Modeling Metric Strategy Advanced Modeling Optimization Feedback Loop - Convergence - Early Stopping Project Handoff Finishing up Project SigOpt Access Class
  • 38. SigOpt. Confidential. Tobias Andreasen | tobias@sigopt.com For more information visit: https://sigopt.com/research/ Questions?