Managing uncertainty in ai performance target setting
Download as PPTX, PDF1 like57 views
This document discusses methods for setting performance targets and evaluating uncertainty for AI models. It recommends using Monte Carlo simulations to project how different levels of accuracy would impact product performance before developing complex algorithms. This allows determining if baseline accuracy from simple models is sufficient or if higher accuracy targets are needed. Simulations can also estimate if gathering more data would significantly improve performance. Calibration of confidence scores is important for applications requiring per-instance decisions or risk assessments.
Managing uncertainty in ai performance target setting
- 1. MANAGING UNCERTAINTY IN AI PERFORMANCE TARGET SETTING Key methods • Monte Carlo Simulation • AI model ‘calibration’
- 2. HOW MUCH ACCURACY DOES YOUR PRODUCT REQUIRE?
- 3. WORKING BACKWARD FROM PRODUCT REQUIREMENTS •In classical AI applications, such as CFAR and MNIST, Kaggle contests and other online challenges, we are accustomed to hearing about algorithms in the 95-99 percentile + of accuracy •Frequently predictive solutions have built upon one another over a period of years, sometimes decades, with new state of the art models improving performance by a fraction of a percentage point
- 6. FOR MANY APPLICATIONS WITH NEW DATASETS THE RETURN ON INVESTMENT AND WAITING PERIOD TO REACH THE 99TH PERCENTILE IN ACCURACY IS PROHIBITIVE • New datasets require long periods of data exploration to determine and eliminate errors in the dat The data collection process • Frequently, basic models can reach acceptable baseline levels of accuracy with in a time period tha acceptable for early prototype development • Not all applications require the highest level of accuracy • Models with higher and higher levels of accuracy can have diminishing returns as they take longer and are more expensive to train, especially when using cloud services such as AWS • Simple models such as MLP and KNN can be implemented quickly using tools like scikit-learn and decent results
- 7. RESULTS OF SIMPLE MODELS APPLIED TO MNIST
- 8. HOW CAN YOU DETERMINE WHEN YOU NEED TO INVEST IN DEVELOPING MORE COMPLEX METHODS?
- 9. HIGHEST RISK VS. LOWER RISK APPLICATIONS •For many applications the risks are clear and the lowest levels of error possible are desirable • Self driving cars • Medical applications •For many applications, the risk associated with an error is not fatal, and the costs associated with 99+ percentile accuracy are large. In some cases, decision boundaries are not clear to human observers and/or labels (such as appraisal values) are not agreed upon. These types of applications are frequent when business, financial or economic subject matter is the target of a prediction problem, but can also appear in other low risk applications such as chat-bots, where occasional errors may not dissuade prospects from converting to sales.
- 10. IDEA – USE MONTE CARLO SIMULATION • Use Monte Carlo to simulate algorithm performance on real data before developing algorithm • For example, you can assess the impact of different levels of accuracy on your product performan investing time and money into developing an AI algorithm
- 12. SIMULATION •Select percentage of known labels, in this hypothetical case “buy” recommendations for hypothetical stocks with returns above a threshold, and create an “AI” selected data set by randomly sampling 1 – p negative examples to be mislabeled by hypothetical AI •Create a model of your product or business performance •Simulate the performance of the product or business using Monte Carlo trials. In this case a portfolio of 50 hypothetical stocks were chosen by the AI and compared to those chosen by a hypothetical human, with some information, from the same universe of stocks •Probability distribution of false identification in feature space can be specified and tested for distributions with same mean precision •In this hypothetical example, an algorithm with 99% accuracy would be a good target, but should consider whether or not 80-90% would be sufficient •Also: should consider what level of accuracy is possible (for example, by considering variability between human experts, in light of Big Data) and
- 13. INCOMPLETE DATA •Another question companies frequently face is whether or not the cost and time required to gather additional data will significantly improve model performance •Concept: utilize simulation on existing data to estimate performance improvements •Can sub-sample from data to simulate missing data, either in feature or label space •Eliminate field entries or entire examples and track degradation of algorithm performance •If performance does not decrease significantly, than more data is unlikely to be helpful
- 14. NEW DATASome population data may be available for target populations at a high level, but predicting labels for individuals from the population requires data to be gathered and significant Predictive features. Companies need to decide if the investment is worth it. For example: should we gather data to predict income in the U.S. or Canada first? Can simulate perfo determine which country would be more profitable to predict on a per-capita basis, given product or For example, targeted advertising based on predicted income. Different distribution assumptions can
- 15. ALL OF THIS CAN BE ACCOMPLISHED BEFORE AN ALGORITHM IS DEVELOPED OR DURING EARLY STAGES OF ALGORITHM DEVELOPMENT As we can see from the error rates of these simple algorithms on MNIST data, which can be rapidly prototyped using existing packages, a product prototype can be built while considering the added benefits of further development on the dataset we need to work with by simulating the performance within our business or product model
- 16. MODEL CALIBRATION Guo, C., Pleiss, G., Sun,Y., Wienberger, K., (2017) On Calibration of Modern Neural Networks Proceedings of the 34th International Conference on Machine Learning, 70, pp 1321-1303
- 17. IMPORTANCE OF CALIBRATION • Useful when decisions need to be made or risks need to be assessed at the level of single predictio • For example, in human-ai collaboration paradigms in which human assistance is requested for cas Machine confidence falls below a threshold • Investors buying single art works require risk assessments on a per-item basis • Current calibration methods as reviewed in the referenced article asses calibration across all featur • However, there is no reason to assume that an algorithm equally well calibrated across all subsets • For example, there have been many cases in which facial recognition, sentiment analysis fail for pro subgroups