Reimagining Medical Image Segmentation: Efficient Prompting, Adaptive Ranking, and Scalable Use

By Taha Kass-Hout, MD, MS, Erhan Bas, and Zhijian (George) Yang

Excited to share our latest research into AI-driven medical imaging—a purely homebrew solution developed in-house, leveraging the creativity and expertise of our team and our brilliant intern, Aishik Konwer. This work, accepted at CVPR 2025, explores enhancements to the Segment Anything Model (SAM) with the goal of improving efficiency in medical image segmentation—all while aiming to minimize the need for human annotations, while boosting segmentation accuracy and efficiency. By combining efficient prompting with preference optimization, our research method produces high-quality organ and tumor segmentations with minimal expert input. In fact, it achieved state-of-the-art results on lung and breast tumors, with organ segmentation across X-rays, ultrasounds, and CT scans using only a fraction of the usual labeled data. This is a major leap for AI in healthcare imaging. Let’s dive deeper.

This research aims to advance medical image segmentation by reducing reliance on human annotations while striving to maintain high accuracy. Ultimately, this research aims to contribute to faster diagnoses, reduced radiologist workload, and broader applicability across medical imaging.

Why This Research Matters

Medical AI continues to be bottlenecked by the need for meticulously labeled datasets—a process that is both time-consuming and expensive. Our research aims to explore ways to reduce annotation efforts while maintaining segmentation accuracy, ultimately working toward making AI-assisted imaging more efficient, scalable, and accessible across different healthcare settings.

Key Innovations in This Work

I. Unsupervised Smart Prompting: Traditional AI models require radiologists to input manual prompts (like points, bounding boxes, or contours) to guide segmentation. Our research investigates ways for the model to automatically generate its own prompts by leveraging vision-language intelligence (BiomedCLIP, VQA, and GPT-based processing) to locate and define anatomical structures without direct human input. This approach aims to reduce the burden of manual labeling, shorten turnaround times, and enable more scalable AI tools. Let me explain. The model generates its own prompts to locate structures in an image – no human clicks needed. It leverages vision-language intelligence to capture semantic, location, and shape information from the scan automatically. In simple terms, the AI “understands” what to look for (e.g. a lung outline or a tumor region) without explicit guidance, which means far fewer hand-drawn labels are required.

I. Efficient Prompting – The AI is designed to generate its own prompts instead of depending on expert-provided markers, with the goal of making segmentation faster and more scalable.

II. AI Preference Optimization (learning from feedback): Instead of relying on fully annotated segmentation masks as ground truth, this approach explores ranking-based feedback. A virtual annotator assigns ratings to different segmentation outputs, allowing the model to refine its predictions based on relative rankings. This could lead to AI that adapts over time with minimal human oversight, reducing the need for explicit reward functions or extensive manual corrections. Let us explain. Instead of relying on dense annotations, the model learns from simple feedback signals. A virtual annotator gives each predicted segmentation a rating or ranking, like saying “this one’s better than that one.” The model then adjusts itself to improve, effectively simulating a human-in-the-loop who rewards good outputs and penalizes weaker ones. This preference-based learning fine-tunes the segmentation quality without extensive human intervention or complex reward models. The result is an AI that continuously self-improves as if a specialist were guiding it – but with much less effort.

II. Intuitive Ranking – Instead of requiring full segmentation masks, the model seeks to improve by learning from simple rankings, potentially reducing the need for costly manual annotations.

III. Diverse Use: Our research demonstrates promising performance in reducing annotation efforts while maintaining segmentation accuracy. In our internal testing, our approach achieved a Dice score of 89.68 in lung segmentation with 50% of the training data, outperforming traditional supervised models or other foundation models in limited-data settings. Additionally, for abdominal CT organ segmentation, our method reached a mean Dice score of 85.70, showcasing its effectiveness across multiple modalities. Notably, we also observed strong performance in ultrasound-based breast tumor segmentation, achieving a Dice score of 88.15, further emphasizing the model’s generalization across diverse imaging techniques. Importantly, this framework maintained high accuracy despite using significantly fewer manually labeled images, reinforcing its potential to improve workflow efficiency for radiologists while reducing reliance on expert annotations.

III. Diverse Use – The approach is being explored across X-rays, ultrasounds, and CT scans, with preliminary results showing potential segmentation performance of 89.68% for lungs, 85.70% for abdominal organs, and 88.15% for breast tumors in ultrasound images.

Homebrew and Lightweight

Unlike large-scale AI frameworks that often require significant computational resources and extensive labeled datasets, our approach is designed to be efficient and adaptable. The goal is to develop a solution that could one day be practical in real-world clinical settings.

Potential Impact on Patients & Providers

• Faster Diagnoses → AI-assisted segmentation could help accelerate workflows, reducing time-to-diagnosis for conditions like lung disease and tumors.

• Reduced Radiologist Workload → Manual annotation is time-consuming. An AI-driven approach like this aims to let experts focus more on interpretation rather than pixel-by-pixel labeling.

• Improved Treatment Planning → More accurate AI-assisted segmentation could support better decision-making for surgical planning and therapy selection.

Looking Ahead

As we are deeply invested in the intersection of AI and healthcare, this research is exciting because it explores ways to address one of the biggest challenges in medical AI—the reliance on large, manually annotated datasets. By investigating methods to reduce annotation efforts while maintaining accuracy, we move toward a future where AI could become an increasingly practical tool in healthcare. This is just the beginning. Huge kudos to the team, our intern Aishik Konwer, and everyone involved in making this vision a reality. Looking forward to discussing this at CVPR 2025 and seeing how these ideas shape the future of AI in medicine!

Preprint is available here

Aishik Konwer Zhijian Yang Erhan Bas Cao (Danica) Xiao Prateek Prasanna Parminder Bhatia