Matthew Crowson, MD

Complex Concepts, Simple Trios: Guide to LLMs in Healthcare Just as a tripod provides unparalleled stability and primary colors form the foundation of all hues, certain truths in innovation often reveal themselves in threes. In the rapidly evolving world of healthcare, integrating large language models (LLMs) (aka generative AI) is a multifaceted transformation built upon interconnected pillars. Over the last few months, I've had countless discussions about AI's impact on healthcare. I'm excited to share that I'm channeling these learnings into a new project. While this venture is still in its early stages, gathering diverse perspectives and validating our approach with real-world insights is critical. That's why I'm launching 'Healthcare AI in Threes,' an article series to explore my evolving understanding of these crucial foundational elements. I aim to share concise, actionable insights in digestible trios, straight from the front lines of product development. Here is the list that I have planned so far: 1. The "Why": Three Core Strengths Focus: Unstructured Text, Natural Language Processing, Infinite Knowledge Base We'll uncover what makes LLMs fundamentally different and incredibly powerful for healthcare. 2. The "How": Three Essential Concepts Focus: Context, Prompt, Tools Understand the core concepts that drive successful LLM interactions and applications. 3. The "Path": Three Keys to Adoption Focus: Workflow Integration, Explainability, Regulatory We'll discuss the keys to achieving widespread and responsible LLM adoption in healthcare. 4. The "Where": Three Points of Impact Focus: Point of Care, Population Health, Agentic Applications Discover the diverse areas where LLMs are poised to revolutionize healthcare operations and patient care. 5. The "Enablers": Three Critical Technologies Focus: FHIR, MCP, RAG Explore the core technologies that are enabling and accelerating LLM adoption in healthcare. 6. The  "Use Cases": Three Transformative Cases Focus: Summarization, Clinical Decision Support (CDS), Clinical Trials Matching Practical and impactful use cases where LLMs are already transforming clinical workflows. I'm looking for feedback, validation, and collaboration. If you work in these areas or any topic resonates with you, please subscribe to the newsletter or connect with me. Are there other topics or questions about LLMs in healthcare that we should cover? I'd love to hear from you! https://lnkd.in/gfzrqHAN

71

14 Comments

One of our core areas of work is curating datasets, and this also represents our use of LLMs for label extraction. When we begun labeling our Emory CXR dataset, we extracted both CheXpert labels and LLM labels. At that point we needed to make a decision - will we keep chasing every new LLM update (while remaining cost conscious) to update our labels or what was going to be our guide? We share our experience in this paper - https://rdcu.be/ekNCl led by Bardia Khosravi. We developed #Radprompter to help with tracking and versioning our prompts with new updates. We found - 1. LLM-based labeling outperformed the CheXpert labeler, with the best LLM achieving 95% sensitivity for fracture detection versus CheXpert’s 51%. 2. Larger models showed better sensitivity, while chain-of-thought (COT) prompting had variable effects. 3. Image classifiers showed resilience to labeling noise when tested externally. Simply stated - We can rely on using the smallest models with good performance. Be careful with COT as the models tend to "overthink" resulting in inaccuracies. For downstream task model training, LLM extracted labels are almost as good as human extracted labels - which means you can direct your resources to the test dataset. We found that the models trained on these labels were very robust to label noise.

73

3 Comments

BREAKING NEWS! Two popular large language models (LLMs), GPT-4 and Llama-3, produced medical device-like decision support across a range of scenarios even though those are not regulated by the FDA. Authors of a new study were interested in finding out whether LLMs show promise for clinical decision support (CDS) even though none is currently authorized by the FDA as a CDS device.  "𝐹𝑜𝑙𝑙𝑜𝑤𝑖𝑛𝑔 𝑑𝑒𝑐𝑖𝑠𝑖𝑜𝑛 𝑠𝑢𝑝𝑝𝑜𝑟𝑡 𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑠 𝑎𝑏𝑜𝑢𝑡 𝑡𝑖𝑚𝑒-𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙 𝑒𝑚𝑒𝑟𝑔𝑒𝑛𝑐𝑖𝑒𝑠, 𝟣𝟢𝟢% 𝑜𝑓 𝐺𝑃𝑇-𝟦 𝑎𝑛𝑑 𝟧𝟤% 𝑜𝑓 𝐿𝑙𝑎𝑚𝑎-𝟥 𝑟𝑒𝑠𝑝𝑜𝑛𝑠𝑒𝑠 𝑤𝑒𝑟𝑒 𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑡 𝑤𝑖𝑡ℎ 𝑑𝑒𝑣𝑖𝑐𝑒-𝑙𝑖𝑘𝑒 𝑑𝑒𝑐𝑖𝑠𝑖𝑜𝑛 𝑠𝑢𝑝𝑝𝑜𝑟𝑡 (𝐹𝑖𝑔. 𝟣). 𝐴𝑙𝑙 𝑚𝑜𝑑𝑒𝑙 𝑠𝑢𝑔𝑔𝑒𝑠𝑡𝑖𝑜𝑛𝑠 𝑤𝑒𝑟𝑒 𝑐𝑙𝑖𝑛𝑖𝑐𝑎𝑙𝑙𝑦 𝑎𝑝𝑝𝑟𝑜𝑝𝑟𝑖𝑎𝑡𝑒 𝑎𝑛𝑑 𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑡 𝑤𝑖𝑡ℎ 𝑠𝑡𝑎𝑛𝑑𝑎𝑟𝑑𝑠 𝑜𝑓 𝑐𝑎𝑟𝑒." Conclusions? 1) LLMs are capable of CDS! 2) If so, people will use them on a massive scale. 3) If so, it would make a lot of sense for regulators to provide regulatory oversight even though LLMs provide disclaimers that they are not intended to be used for medical advice. And these are just unimodal LLMs. Imagine the regulatory challenge that will come with multi-modality. Study: https://lnkd.in/eaCCbXed

66

12 Comments

Thrilled to share that our new paper on Multi-Agent Systems for Healthcare (MASH), with Eric Topol, MD and Pranav Rajpurkar, was published today in Nature Biomedical Engineering! Hundreds (thousands?) of companies in medical AI are rapidly developing specialized tools of varying complexity and autonomy for both clinical and operational tasks. Once brought to market, how will these products interact with patients and providers to improve care quality and efficiency? We imagine decentralized networks of such specialized AI agents—what we call Multi-Agent Systems for Healthcare (MASH)—working collaboratively across both clinical and operational domains, communicating in natural language, and accessing only the information/data necessary for their tasks. We discuss the challenges and opportunities, and a follow a patient through such a care journey. Check out our vision for the future of healthcare AI: 👉 https://rdcu.be/efXNc As an aside, it's absolutely incredible to see how much has evolved in AI since our original draft about a year ago! For example, the capabilities visual avatars have improved by orders of magnitude, and in just the last few weeks Model Context Protocols (MCPs) have emerged as a new way for agents to interact and use tools. Looking forward to hearing your thoughts and discussing where AI is taking healthcare next! SSM Health Saint Louis University School of Medicine Saint Louis University

71

2 Comments

Generative AI in Medicine: From Hype to Healthcare Transformation A landmark review in Nature Medicine explores how Generative Artificial Intelligence (GAI) is reshaping the landscape of healthcare — from clinical decision-making and education to biomedical research and administration. 🔹 Beyond automation: GAI models such as GPT-5, Gemini, and DeepSeek-R1 are evolving into agentic systems—capable of reasoning, multi-modal analysis, and task completion with minimal supervision. 🔹 Clinical promise: Early trials show improved diagnostic support, medical documentation, and predictive analytics, especially when AI augments—rather than replaces—clinicians. 🔹 Research acceleration: AI agents can now generate hypotheses, analyze data, simulate experiments, and even design new molecules and proteins—speeding scientific discovery. 🔹 Education and operations: GAI supports personalized medical training and relieves clinicians from heavy administrative tasks through real-time documentation and intelligent assistants. 🔹 Key challenges: Ethical governance, bias mitigation, transparency, and robust evaluation remain critical for safe integration into real-world healthcare systems. 🩺 As the review concludes, “careful, thoughtful adoption” of generative AI can enhance accessibility, equity, and quality in healthcare—if guided by strong clinical validation and ethical oversight.

99

3 Comments

A few reflections from #AAAI2025 meeting in Philadelphia ICYMI (pt 1) It's true that AI is much more than ChatGPT (more generally, Foundation models or LLMs). But the impact of LLMs attracting scientists and engineers to AI--that's off the charts.  Which makes for phenomenal innovation to come. FOUR highlights (of many) Agentic AI: moving from single step to iterative problem solving by LLMs that then enable them to break large tasks into smaller tasks, each solved by more specialized (smaller) models. Also may help improve explainability of models. Accelerating the transformation of “hmm... I have an idea” to "I have an app/working piece of code” thanks to gen AI code and libraries. It will mean a lot more junk to sift through, but also way more pearls. Andrew Ng's #AAAI2025 comments on fast prototyping are worth watching--"one new start-up a month." LLMs can advance scientific research and discovery: accelerating every step of the way. One of the most intriguing ideas, "thinking about embedding space rather than real-world space as the realm of exploration and understanding." (embedding space is the way computers or LLMs represent words/knowledge). Advancing AI in education (K-12, university, and public) #EAAI is a dedicated group of educators who share their lesson plans, their games and workbooks for teaching people at every level about AI. From kids learning about embeddings and gender bias (there’s that embedding idea again) to 6 week programs for people on parental leave by Sedef Akinli Kocak, PhD, MBA to a class for the whole of UT Austin faculty and staff by Peter Stone. THREE challenges Questionable benchmarking: entities using benchmarking to train models are distorting the playing field. It’s like studying to the answer key, the results don’t reflect the actual quality of the models. Subbarao Kambhampati delivered a fantastic keynote. Using LLM/FMs where more traditional (lower resource-intensive) AI tools are just fine, and even better. This will gradually sort itself out, provided the students of AI still learn things other than LLMs. Applications need to keep up with the developers. Terrific initiatives in improving credibility in elections (Biplav Srivastava), explaining radiology findings to patients (Luyang Luo), and crowdsourcing sustainability efforts (Elena Sierra), just to name a few at AAAI. Lastly, delighted to have met Francesca Rossi President of AAAI at the Women’s Mentoring Lunch—together with so many capable women in the field of data science and AI! Thanks #AAAI André Artelt Ulrike Kuhl Chantan Reddy Rachel Chung Narges Norouzi Maryam Mirzaei Julia Stoyanovich Sedef Akinli Kocak, PhD, MBA Lisa Meeden Vivek Gupta AIForSocialGood.ai Alondra Nelson

43

6 Comments

Part 4: Evaluation for AI in Healthcare – Reflecting on HealthBench, Gaps We’re Bridging, and What’s Next OpenAI’s HealthBench is a major milestone in the healthcare AI space. It finally brings a rigorous, clinically grounded structure to evaluating LLMs—with themes like emergency escalation, communication nuance, and global diversity. As someone working at the intersection of AI, health behavior, and patient-facing tools, I see this as a gold-standard baseline we’ve long needed. Why it matters: It finally moves us beyond outdated exam-style benchmarks (like MedQA or USMLE-style MCQs) and into real-world, scenario-based evaluation-synthetic cases written by researchers, validated by physicians. Finally, a benchmark that feels closer to real patient interaction. At our end, we’ve been developing a comprehensive evaluation pipeline for healthcare AI—especially for tools used in sexual health, triage, and digital guidance. And this new benchmark helps us align efforts while also identifying where real-world complexity still needs deeper reflection. Where HealthBench Raises the Bar: • Meaningful axes like emergency referral, response depth, and global diversity • A shared benchmark we can finally build around • A thoughtful first move toward standardized healthcare evaluation - especially suited for general medical guidance • And backed by a robust rubric (48,000+ grading criteria) with input from 262 physicians across 26 specialties The areas we believe need further reflection: 1. Sexual Health and Sensitive Domains → These are often excluded due to cultural complexity—but they’re the exact areas patients hesitate most. We’re actively building a sexual health–specific evaluation track to address nuance, stigma, and user safety. 2. Multimodal Testing → Real-world healthcare tools aren’t text-only. They use images, symptom photos, even voice input. Our framework integrates text + image evaluation— starting with tools used in digital triage, sexual health and dermatology. 3. Privacy & Consent Simulation Consent isn’t always clean-cut. We test real-world gray areas: • Uploading someone else’s data • Ambiguous user identity • Simulating underage use These edge cases are critical to ensure tools don’t just follow policy—but behave ethically. 4. Behavioral & Emotional Safety → We don’t just evaluate what the AI “knows”—we test how it behaves when a user is confused, anxious, or hesitant. Because in healthcare, trust is built—or lost—in those small, human moments. We’ve been conducting weekly deep-dive evaluations—one focus area at a time. We are aligned with healthbench in key areas and also go beyond—to reflect sensitive use cases, human behavior, and real-world risks. Our eval pipeline—particularly those built for high-context, sensitive, and multimodal healthcare tools. Because in healthcare, evaluation must go beyond benchmarks. Yudara Kularathne MD, FAMS(EM) Thanveer Ahamed Janitha Prathapa Dr Sithira A.

22

2 Comments

In 2020, Ron Li , Steve Asch, and Nigam Shah published a multidisciplinary framework for AI delivery science in healthcare (recently shared by Jan Beger). Five years later, I'm seeing a clear pattern: organizations that treat governance as a design principle, not a gate check, are consistently outpacing their peers. Why it works: embedding governance early reduces costly rework cycles that stall releases and drain budgets. While competitors navigate months-long review loops, teams that are compliant by design ship the next iteration. The playbook I keep seeing succeed: - Inventory all sensitive data via AI-driven discovery - Categorize models early under EU AI Act tiers, FDA SaMD rules, HIPAA - Embed privacy-preserving tech — redaction strategies, differential privacy, synthetic data, continuous monitoring, and remediation previews — wherever sharing is constrained The 2025 debate isn't whether to invest in AI governance. It's whether you'll use it to pull ahead of teams still bolting compliance on at the end. What governance tactic has saved your team the most time? I'd love to hear your thoughts and examples. Jan Beger, GE HealthCare, Ron Li , Steve Asch, and Nigam Shah

41

8 Comments