COSMOS1_Scitech_2014_Ali
- 1. A Novel Approach to Simultaneous Selection of Surrogate Models, Constitutive Kernels, and Hyper-parameter Values Ali Mehmani*, Souma Chowdhury#, and Achille Messac# * Syracuse University, Department of Mechanical and Aerospace Engineering # Mississippi State University, Bagley College of Engineering 10th Multi-Disciplinary Design Optimization Conference AIAA Science and Technology Forum and Exposition January 13 – 17, 2014 National Harbor, Maryland
- 2. Surrogate model • Surrogate models are commonly used for providing a tractable and inexpensive approximation of the actual system behavior in many routine engineering analysis and design activities: 2
- 3. Surrogate model • Surrogate models are commonly used for providing a tractable and inexpensive approximation of the actual system behavior in many routine engineering analysis and design activities: 풘풊 흍( 풙 − 풙풊 ) 3 Model Type Kriging RBF SVR . . . Linear Exponential Gaussian Cubic Multiquadric . . . Kernel / basis function Correlation parameter Shape parameter . . . Hyper-parameter 풇 풙 = 풏 풊=ퟏ 흍 풓 = (풓ퟐ + 풄ퟐ) ퟏ/ퟐ 풓= 풙 − 풙풊 풄풍풐풘풆풓 < 풄 < 풄풖풑풑풆풓
- 4. Research Objective Develop a new model selection approach, which simultaneously select the best model type, kernel function, and hyper-parameter. Types of model Types of basis/kernel Hyper-parameter(s) 4 • RBF, •Kriging, • E-RBF, • SVR, •QRS, • … • Linear •Gaussian • Multiquadric • Inverse multiquadric •Kriging • … • Shape parameter in RBF, • Smoothness and width parameters in Kriging, •Kernel parameter in SVM, • …
- 5. Presentation Outline 5 • Surrogate model selection • REES-based Model Selection • 3-Level model selection • Regional Error Estimation of Surrogate (REES) • Numerical Examples • Concluding Remarks
- 6. Surrogate model selection 6 Experienced-based model selection • Dimension and nature of sample points, • Level of a noise, • Application domain, • … Suitable Surrogate Automated model selection Error measures are used to select the best surrogate • RMSE, • Cross-validation, • REES, • … Hyper-parameter selection (Kriging-Guassian) using cross validation and maximum likelihood estimation (Martin and Simpson) Model type and basis function selection using cross validation (Viana and Haftka) Model type selection using leave-one-out cross validation (Drik Gorisson et al.)
- 7. 3-Level model selection In 3-level model selection, the selection criteria could depend 7 on the user preference. Standard surrogate-based analysis Structural optimization applications lower median error lower maximum error
- 8. 3-Level model selection 8 Median error Maximum error Two model selection criteria evaluated using advanced surrogate error estimation method presented in REES Depending on the problem and the available data set, the median and maximum errors might be mutually conflicting mutually promoting Pareto models A single optimum model
- 9. 3-Level model selection To implement a 3-level model selection, two approaches are 9 proposed: (i) Cascaded technique, and (ii) One-Step technique.
- 10. 3-Level model selection 10 Cascaded technique For each candidate kernel function, hyper-parameter optimization is performed to minimize the median and maximum error. Hyper-parameter optimization is the process of quantitative search to find optimum hyper-parameter value(s). Post hyper-parameter optimizations, Pareto filter is used to reach the final Pareto models.
- 11. 3-Level model selection 11 Cascaded technique Solutions of the hyper-parameter optimization in the cascaded technique for multiquadric basis function of RBF surrogate for Baranin-hoo function
- 12. 3-Level model selection The three-level model selection could also be performed by solving a single uniquely formulated mixed integer nonlinear programming (MINLP) problem. 12 One-Step technique To escape the potentially high computational cost of the cascaded technique Subjected to model type basis function hyper-parameter(s)
- 13. Regional Error Estimation of Surrogate (REES) The REES method is derived from the hypothesis that the accuracy of approximation models is related to the amount of data resources leveraged to train the model. • Mehmani, A., Chowdhury, S., Zhang, Jie, and Messac, A., “Quantifying Regional Error in Surrogates by Modeling its Relationship with Sample Density,” 54th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference, Paper No. AIAA 2013-1751, Boston, Massachusetts, April 8-11, 2013. • Mehmani, et. al., “Model Selection based on Generalized-Regional Error Estimation for Surrogate,” 10th World Congress on Structural and Multidisciplinary Optimization, Paper No. 5447, Orlando, Florida, May 19-24, 2013. • Mehmani, et. al.., “Regional Error Estimation of Surrogates (REES),” 14th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, Paper No. AIAA 2012-5707, Indianapolis, Indiana, September 17-19, 2012. 13
- 14. REES Surrogate Model Training Sample Point Point
- 15. 1st Iter. 2nd Iter. 3rd Iter. 4th Iter. Error Surrogate ε1 ε2 ε3 ε4 ? Training Point Test Point 8 16 12 12 16 8 20 4 24 0 1st 2nd 3rd 4th REES
- 16. Training Point Test Point 1st Median of RAEs Intermediate Actual model surrogate model ε = 풎풆풅 | 풇풊 − 풇풊 1st 풇풊 | , 퐢 = ퟏ, … , ퟏퟔ ... REES 16
- 17. 17 Median of RAEs Momed It. 1 REES t1 t2 t3 t4 Number of Training Points
- 18. 18 Median of RAEs t1 t2 t3 t4 Number of Training Points Momed It. 1 It. 2 REES
- 19. 19 Median of RAEs REES t1 t2 t3 t4 Number of Training Points Momed It. 1 It. 2 It. 3
- 20. 20 It. 1 It. 3 Median of RAEs It. 2 t1 t2 t3 t4 Number of Training Points Momed It. 4 Model of Median REES
- 21. 21 It. 1 It. 3 Median of RAEs It. 2 t1 t2 t3 t4 Number of Training Points Momed It. 4 Model of Median Predicted Median Error REES
- 22. Predicted Maximum Error 22 It. 1 It. 3 Median of RAEs It. 2 t1 t2 t3 t4 Number of Training Points Momed It. 4 Model of Median Momax Mode of maximum error distribution at each iteration Predicted Median Error REES
- 23. The effectiveness of the new 3-level model selection method is investigated by considering the following three candidate surrogates: Model type Kernel function Hyper-parameter RBF Kriging The methods are implemented on three benchmark problems and an engineering design problem are tested. 23 Gaussian basis function Multiquadric basis function Gaussian correlation function Exponential correlation function Radial basis kernel function Sigmoid kernel function SVR Numerical Examples
- 24. 24 Numerical Examples Numerical Setting The numerical settings for the implementation of REES-based model selection for the benchmark problems The numerical settings for the hyper-parameter optimization 24
- 25. Hyper-parameter optimization of Cascaded technique in different surrogate type and Kernel functions for Branin-Hoo function with 2 design variables 25 Numerical Examples
- 26. 26 Numerical Examples Numerical Setting The numerical settings for One-Step technique Integer design variables
- 27. 27 Results; Branin-hoo 2 design variables Computational cost One-Step Technique RBF-Multiquadric Cascaded technique
- 28. 28 Results; Branin-hoo 2 design variables One-Step Technique Cascaded technique Actual error RBF-Multiquadric RBF-Multiquadric
- 29. Results; Hartmann function with 6 design variables 29 Computational cost One-Step Technique SVR-Radial basis Cascaded technique
- 30. Results; Hartmann function with 6 design variables 30 One-Step Technique SVR-Radial basis Cascaded technique Actual error SVR-Radial basis
- 31. Results; Dixon & Price function with 18 design variables 31 Computational cost One-Step Technique RBF-Multiquadric Cascaded technique
- 32. Results; Dixon & Price function with 18 design variables 32 One-Step Technique RBF-Multiquadric Cascaded technique Actual error RBF-Multiquadric
- 33. 33 Numerical Examples Emed Emed Emed Emax Emax Emax Initial Value Final Solution RBF-Multiquadric (C=0.9)
- 34. 34 Concluding Remarks A new 3-level model selection approach is developed to select the best surrogate among available surrogate candidates based on the level of accuracy. (i) model type selection, (ii) kernel function selection, and (iii) hyper-parameter selection. This approach is based on the model independent error measure given by the Regional Error Estimation of Surrogates (REES) method. The preliminary results on problems indicate at least 60% reduction in maximum and median error values.
- 35. 35 Future Work Implementation of One-Step technique with larger pool of surrogates with different number of kernels higher dimensional and more computationally intensive problems Develop an open online platform called Collaborative Surrogate Model Selection (COSMOS) to allow users to submit - training data for identifying an ideal model from existing pool of surrogate models, and - their own new surrogate into the pool of surrogate candidates. If interested in COSMOS please contact me at amehmani@syr.edu
- 36. Acknowledgement I would like to acknowledge my research adviser Prof. Achille Messac, and my co-adviser Dr. Souma Chowdhury for their immense help and support in this research. I would also like to thank my friend and colleague Weiyang Tong for his valuable contributions to this paper. Support from the NSF Awards is also acknowledged. 36
- 37. Questions and Comments 37 Thank you
- 38. Quantifying the mode of median and maximum errors 38
- 39. 39 A chi-square (χ2) goodness-of-fit criterion is used to select the type of distribution from a list of candidates such as lognormal, Gamma, Weibull, logistic, log logistic, inverse Gaussian, and generalized extreme value distribution.