Modular Machine Learning for Model Validation
0 likes463 views
The document presents a presentation on modular machine learning for model validation, hosted by Dr. Joseph Simonian from Autonomous Investing, showcasing a methodology that integrates traditional econometrics and data science techniques. This framework aims to enhance the evaluation of investment signals through a structured approach consisting of three main modules: sub-sample classification, signal quality assessment, and signal diversification. The methodology emphasizes the benefits of combining both econometric models and data science methodologies to improve confidence in model validation processes.
![MODULAR MACHINE LEARNING
A defining characteristic of MML is its blending of what Breiman [2001] calls the “data modeling culture”
represented by econometrics, with the “algorithmic modeling culture” represented by data science.
The data modeling culture has hitherto been the dominant culture in traditional statistical practice and is
primarily concerned with providing information about the relationships that exists between input and response
variables and assumes that the data are generated by a specific stochastic process.
In contrast, the algorithmic modeling culture assumes that the relationships between input and response
variables are essentially too complex to uncover, and consequently is primarily concerned with devising
methods to successfully predict responses from inputs.](https://image.slidesharecdn.com/quspeakerseries-2-200722153308/85/Modular-Machine-Learning-for-Model-Validation-14-320.jpg)
Modular Machine Learning for Model Validation
- 1. Qu Speaker Series Modular Machine Learning for Model Validation An Afternoon with Dr. Joseph Simonian Autonomous Investing 2020 Copyright QuantUniversity LLC. Hosted By: Sri Krishnamurthy, CFA, CAP sri@quantuniversity.com www.qu.academy 07/22/2020 Online https://quspeakerseries2.spla shthat.com/
- 2. 2 QuantUniversity • Boston-based Data Science, Quant Finance and Machine Learning training and consulting advisory • Trained more than 1000 students in Quantitative methods, Data Science and Big Data Technologies using MATLAB, Python and R • Building a platform for AI and Machine Learning Exploration and Experimentation
- 3. For registration information, go to https://QuSummerSchool.splashthat.com 3
- 5. 5
- 6. 6 • Joseph Simonian is the Founder and CEO of Autonomous Investing Solutions. Prior to that, Joseph was an Investment Strategist at Acadian Asset Management. Before joining Acadian, Joseph was the director of Quantitative Research for the Portfolio Research and Consulting Group at Natixis Investment Managers. Prior to that, he was a principal research analyst in the Global Institutional Solutions Group at Fidelity Investments. He was also previously a vice president at JPMorgan Asset Management and PIMCO. • Joseph is currently the co-editor of the Journal of Financial Data Science and Advisory Board member for the Financial Data Professional Institute. • Joseph holds a Ph.D. from the University of California, Santa Barbara; an M.A. from Columbia University; as well as a B.A. from the University of California, Los Angeles. Modular Machine Learning for Model Validation
- 7. Modular Machine Learning for Model Validation: An Application to The Fundamental Law of Active Management Joseph Simonian, PhD
- 8. Intro Implementing model validation through a set of interdependent modules that utilizes both traditional econometrics and data science techniques can produce robust assessments of the predictive effectiveness of investment signals in an economically intuitive manner.
- 9. Intro The proposed methodology, modular machine learning, also answers a number of practical questions that arise when applying block time series cross-validation such as what number of folds to use and what block size to use between folds.
- 10. Intro It is possible to re-interpret the Fundamental Law of Active Management into a model validation framework by expressing its fundamental concepts, information coefficient and breadth, using the formal language of data science.
- 11. Intro We introduce an approach towards model validation which we call modular machine learning (MML) and use it to build a methodology that can be applied to the evaluation of investment signals within the conceptual scheme provided by the FL. Our framework is modular in two respects: (1) It is comprised of independent computational components, each using the output of another as its input, and (2) It is characterized by the distinct role played by traditional econometric and date science methodologies.
- 12. Intro where σ_IC^2 represents the variance of the IC. Although there are various ways to express the variability of the IC, in this article we will build our framework using the formula in (1), as it extends the original FL in a formally elegant manner with few underlying assumptions.
- 13. MODULAR MACHINE LEARNING Model validation using MML is implemented through three distinct modules: • Sub-sample classification module: An econometric model is used to classify the sub-sets of a time series into distinct regimes. • Signal quality module: The regime-specific sub-sets of data from (1) are used as inputs to block time- series cross-validated regressions, to derive regime-specific measures of signal quality. In the context of the FL, these would be values for the IC and IC variance. • Signal diversification module: Using signal quality measurements from (2) as inputs, clustering is used to determine the level of diversity present in a set of investment signals. In the context of the FL, the number of clusters is used as the value for BR.
- 14. MODULAR MACHINE LEARNING A defining characteristic of MML is its blending of what Breiman [2001] calls the “data modeling culture” represented by econometrics, with the “algorithmic modeling culture” represented by data science. The data modeling culture has hitherto been the dominant culture in traditional statistical practice and is primarily concerned with providing information about the relationships that exists between input and response variables and assumes that the data are generated by a specific stochastic process. In contrast, the algorithmic modeling culture assumes that the relationships between input and response variables are essentially too complex to uncover, and consequently is primarily concerned with devising methods to successfully predict responses from inputs.
- 15. MODULAR MACHINE LEARNING Data science and traditional econometrics each have unique strengths that can prove valuable to the model validation process. Data science is directed towards prediction, testing, and pattern recognition. In contrast, traditional econometrics has a fairly weak track record with regards to producing effective forecasting tools, but has been more successful in producing formal frameworks that provide ex-post insight into the structure of economic data.
- 16. Advantages There are distinct advantages in using an econometrically rigorous foundation upon which to base model validation, both in terms of methodological clarity and technical utility.
- 17. Advantages Using regime-switching models to classify time series provides the formal means for the model validation process to speak to the particular “mental models” of the economic and financial world that portfolio managers use when building investment strategies.
- 18. Advantages The second advantage of using regime-switching models for sub-sample classification is that when their output is used as the input to a cross-validation procedure, the length and number of folds is objectively determined by the regime-switching model being employed. While the various folds belonging to a particular regime will almost invariably be of different length, they will all be of like kind, representing specific economic and market states. Accordingly, in the context of the FL, this regime-based cross-validation, allows for the derivation of predictive IRs corresponding to different regimes, which can then be considered alone or combined through various averaging procedures.
- 19. Advantages The third advantage of using regime-switching models in model validation is in relation to the block size in block time-series cross-validation. Time series cross-validation is intended to address the “memory” of the past that is embedded in data exhibiting chronological dependencies. It differs from standard types of cross- validation in its requirement that training data temporally precede test data. Block time series cross-validation has been introduced. The latter procedure is defined by the omission of some observations (blocks) lying between training and test samples during cross-validation, in order to reinforce the “memoryless” nature of the operation. When blocking is employed, the question of block size naturally arises, but time-series cross validation by itself does not provide a clear-cut way to choose a value. Traditional regime-switching models on the other hand can provide some guidance in this respect as they provide a specification for the order of the autoregressive process that drives regimes. This order value seems to be a natural choice for the choice of the block value for the blocks that are inserted between training and test samples.
- 21. 21 Demos, slides and video available on QuAcademy Go to www.qu.academy 21
- 22. 22 Stay tuned for more speakers in the Qu Speaker Series this summer! https://qusummerschool.splashthat.com/ Next week! Model Validation for Machine learning models
- 23. Thank you! Sri Krishnamurthy, CFA, CAP Founder and CEO QuantUniversity LLC. srikrishnamurthy www.QuantUniversity.com Contact Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not be distributed or used in any other publication without the prior written consent of QuantUniversity LLC. 23