3. Workshop Responsible Data Science - Discussion on Accuracy in data science by Mykola Pechenizkiy
1 like314 views
The JADS workshop on responsible data science discusses critical aspects of ensuring accuracy in data science models while addressing fairness, confidentiality, and transparency. It highlights the challenges of achieving 100% accuracy due to various trade-offs, including precision-recall and bias-variance. The workshop emphasizes the need for understanding model limitations, performance guarantees, and the implications of uncertain data.
3. Workshop Responsible Data Science - Discussion on Accuracy in data science by Mykola Pechenizkiy
- 1. JADS Workshop on Responsible Data Science “Data science without guesswork – How to answer questions with a guaranteed level of accuracy?” Moderated by prof.dr. Mykola Pechenizkiy RDS: • Fairness • Accuracy • Confidentiality • Transparency
- 2. www.jads.nl Data science with(out) guesswork • Why? • What? • How?
- 3. www.jads.nl Open questions • Accuracy Guarantees, accountability and responsibility • Accuracy What are the right measures? • Accuracy Do we understand the trade-offs well?
- 4. www.jads.nl When should you trust your model’s prediction? • What are the limits of an algorithm / model and when is it known to (have high chances to) fail? • Beyond “simple” error bounds: errors generated by uncertain data, noisy labels, biased data, concept drift, … • Guarantees on average vs. per case performance • cf. worst case analysis • What are we really trying to guarantee? • what are the right accuracy measures • in connection to accountability
- 5. www.jads.nl What are (not so) well understood tradeoffs? Models are hard to make 100% accurate many trade-offs: • Well formulated and well studied: • precision-recall; bias-variance; robustness-adaptivity; • Well formulated, but not so well studied: • accuracy-computational resources; accuracy-human resources (inducing, applying and maintaining does not come for free) • Not so well formulated and not so well studied: • Accuracy-fairness, accuracy-confidentiality, accuracy-privacy, accuracy-transparency, …