Synthetic data in finance
1 like279 views
The presentation discusses the challenges faced with real datasets in quantitative finance and offers insights into synthetic data generation tools, including proprietary and open-source options. It highlights various difficulties such as missing values, access issues, and imbalanced data that can complicate analysis. Additionally, several tools and demos for generating synthetic datasets are presented, aimed at enhancing machine learning applications in finance.
Synthetic data in finance
- 1. 2nd Annual Machine Learning in Quantitative Finance Synthetic Data Generation for Machine Learning in Finance 2020 Copyright QuantUniversity LLC. Presented By: Sri Krishnamurthy, CFA, CAP QuantUniversity 9/1/2020 Powered by:
- 2. 2 Speaker • Quant, Data Science & ML practitioner • Prior Experience at MathWorks, Citigroup and Endeca and 25+ financial services and energy customers. • Columnist for the Wilmott Magazine • Author of forthcoming book “Pragmatic Machine Learning in Finance ” • Teaches Data Science/AI at Northeastern University, Boston • Reviewer: Journal of Asset Management Sri Krishnamurthy Founder and CEO QuantUniversity
- 3. 3 About QuantUniversity • Boston-based Data Science, Quant Finance and Machine Learning training and consulting advisory • Trained more than 1000 students in Quantitative methods, Data Science, ML and Big Data Technologies • Building a platform for operationalizing AI and Machine Learning in the Enterprise
- 4. 4 1. Challenges with Real Datasets 2. Synthetic Dataset generation tools ▫ Proprietary ▫ Open Source – Faker – Data Synthesizer – SDV – Synthpop – GANs 3. Demos ▫ VIX Data Generator Agenda
- 5. Challenges with Real Datasets
- 6. 6
- 7. 7 Not be feasible to get samples for all categories • Lighting conditions • Modifications (Glasses/No glasses, Moustache/ No Moustache etc.) • Positions Coverage Challenges with real datasets
- 8. 8 All scenarios haven’t played out • Stress scenarios • What-if scenarios Challenges with real datasets Figure ref: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf
- 9. 9 Missing values • Missing at random • Missing sequences • Need data to fill frames Challenges with real datasets
- 10. 10 • Access ▫ Hard to find ▫ Rare class problems ▫ Privacy concerns making it difficult to share Challenges with real datasets
- 11. 11 Imbalanced • Need more samples of rare class • Need proxies for data points that were not observed or recorded Challenges with real datasets
- 12. 12 Labels • Human labeling is hard • Synthetic label generators Challenges with real datasets
- 13. Tools for Synthetic Data Generation
- 14. 14 Proprietary Tools Company Core Technology Tonic.ai All-in-one platform for data anonymization, subsetting, and synthesis integrated with databases (hadoop, oracle, mysql, MS sql server, mongo db, amazon aurora/redshift, and google big query) - Uses Condenser and Masquerade Mostly.ai Tablular data using generative deep neural networks (no image data) CVEDIA - Sensor modeling and algorithm training - Handle image using SynCity as a custom pocket laboratory to generate highly entropic scenes, conditions, and metadata. Enable real-time Hardware-In-the-Loop (HWIL), Human-In-the-Loop (HITL) or Software-In- the-Loop (SIL) simulations even with complex sensor configurations Deep vision data Image creation Synthetic training data Synthesis.ai The data generation platform for computer vision
- 15. 15 López de Prado, Marcos, Machine Learning for Asset Managers, Cambridge University Press 2020
- 19. 19 Synthpop
- 22. 22 WGAN
- 23. 23
- 24. 24
- 25. 25
- 26. 26
- 28. 28
- 29. 29 Demo 1 – Loan Data Synthesizer
- 30. 30 Demo 2: Synthetic Sales data generation
- 31. 31 Demo 3 : Synthetic VIX generation
- 32. 32 If you want to be a part of QuSandbox private Beta Contact us: info@qusandbox
- 33. Sri Krishnamurthy, CFA, CAP Founder and Chief Data Scientist sri@quantuniversity.com srikrishnamurthy www.qu.academy Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not be distributed or used in any other publication without the prior written consent of QuantUniversity LLC. 33