Optimizing AI and human collaboration in mammography

The most important aspect of designing processes that leverage human-AI symbiosis is task design and allocation. Cost aspect has been secondary, and for some industries, like healthcare, I did not consider cost (savings) at all since the objective there is well well-being of humans. Then I came across this paper. https://lnkd.in/g986Z2PN It brings the cost element in the mix as well. I think our thought processes still sync since they are not worried about cost savings, but rather the cost of errors. The authors examine how to optimally share diagnostic tasks between AI and human radiologists in mammography screening, in order to minimize costs and maintain or improve performance. They compare three strategies: Expert-alone: radiologists do all interpretations. Automation: AI alone does all interpretations. Delegation (hybrid): AI first assesses mammograms, and only those above some risk threshold are passed on to radiologists. The researchers have built an optimization model that takes into account costs like follow-ups for false positives, litigation costs for false negatives, the cost of using AI, cost of expert evaluation, prevalence of disease, and the performance (sensitivity, specificity / AUC) of both AI algorithms and human experts. Then they validate by backtesting on real mammography datasets from a crowdsourced competition. Key findings here are: Delegation (AI + human) often wins: Under many realistic parameter settings, the hybrid/delegation strategy gives the lowest expected cost compared to human-only or full automation. In their backtests, the delegation strategy reduced costs by about 17.5% to 30.1% relative to the expert-alone strategy. Critical role of disease prevalence and cost trade-offs: What strategy is optimal depends strongly on how common breast cancer is in the screened population, and how severe (in cost) false negatives are relative to false positives.For example, when prevalence is higher or litigation costs for missing a cancer are large, strategies that reduce false negatives (even at the cost of more false positives) become more favorable. Effect of algorithm and litigation costs: Lower costs for AI use and lower litigation costs tend to expand the range of conditions under which delegation is optimal. If AI is cheap and risk of false negatives (or liability) is lower, then full automation becomes more plausible. But high costs push decision-making toward human expert involvement. Performance thresholds matter: The relative performance of AI vs radiologists (e.g., AUC, true/false positive rates) determines when one strategy overtakes another. As AI performance increases, there’s a shift: from expert-alone → delegation → automation. Liability asymmetry: If AI systems are held to stricter liability (e.g., legal standard, product liability) compared to humans, this increases the cost of errors by AI and makes full automation less attractive. #artificialintelligence

  • No alternative text description for this image

To view or add a comment, sign in

Explore content categories