HIPAA in the Age of LLMs: What Founders Need to Know About PHI and Generative AI

The AI Gold Rush Meets HIPAA: Are You Covered?

HealthTech founders are racing to integrate generative AI into care delivery, coding, claims, and patient engagement. But there’s a hard truth: if your AI touches Protected Health Information (PHI), you may already be in violation of HIPAA—even pre-launch.

AI can scale your impact, but it can also scale your liability.

As someone who advises high-growth SaaS and healthcare startups on Health Insurance Portability and Accountability Act (HIPAA) Security + Privacy, HITRUST, ISO - International Organization for Standardization 27001, and AI governance, I’ve seen what happens when innovation gets ahead of regulation. Here’s what you need to know to stay compliant—and win trust—in the GenAI era.

What Is PHI Under HIPAA?

HIPAA protects any health information linked to an identifiable individual, including names, MRNs, diagnoses, treatment dates, biometric data, and more (there are 18 HIPAA Identifiers for PHI!). Even a symptom combined with a phone number can qualify as PHI.

HIPAA’s Privacy Rule limits how PHI can be used or shared. The Security Rule mandates controls over electronic PHI (ePHI). The Breach Notification Rule requires disclosure if that data is compromised.

If your AI system processes, stores, or transmits PHI, you’re officially in the compliance zone—and so are your vendors.

Where GenAI and HIPAA Collide

1. Prompt Injections: Breach by Manipulation

Generative AI models are vulnerable to prompt injection attacks, where users manipulate the model into leaking information or behaving unsafely.

Researchers have already shown that medical Large Language Models (LLMs- the type of Generative AI that most folks are used to dealing with) can be coerced into flipping diagnoses or divulging PHI. A manipulated chatbot could potentially disclose another patient’s records or send unauthorized transcripts.

HIPAA risk: Uncontrolled prompts = uncontrolled PHI disclosure.

2. Using Non-Compliant Vendors

Many startups unknowingly violate HIPAA by using popular AI services that refuse to sign Business Associate Agreements (BAAs).

For example:

• ChatGPT (non-enterprise): Not HIPAA compliant.

• Google Med-PaLM 2: Available under Google’s HIPAA BAA.

• Microsoft Azure OpenAI: HIPAA-eligible with proper configurations.

If your AI vendor won’t sign a BAA or uses your inputs to train future models—you’re exposing PHI unlawfully.

Treat your AI vendor like an Electronic Health Records (HER) vendor. Vet accordingly.

3. Training on PHI Without Controls

Training or fine-tuning a model on identifiable health data without explicit authorization or safeguards is a HIPAA violation. You’ll need:

• A valid legal basis for use (authorization or allowed operations).

• Secure infrastructure.

• Clear audit trails and access controls.

Some generative models have been shown to memorize and regurgitate PHI from training sets. That’s a regulatory and reputational nightmare.

Don’t treat your training data like development scraps. It’s regulated.

4. Hallucinated PHI and Medical Errors

Even when not trained on PHI, AI can hallucinate facts—like allergies, medications, or medical histories.

An AI drafting a clinical note that wrongly includes “penicillin allergy” can lead to improper treatment. It also risks inserting PHI from elsewhere, potentially exposing unrelated patient data.

HIPAA’s “minimum necessary” standard applies to outputs too.

5. Shadow AI: Staff Bypassing Security

Clinicians often turn to ChatGPT or other tools for summaries, patient instructions, or translations. Netskope reports that 71% of healthcare workers still use unauthorized AI tools at work.

That’s a compliance minefield:

• No BAA = automatic HIPAA violation.

• Copy-pasting PHI = untracked disclosure.

• Logs or prompts may be stored externally.

Shadow AI is the new Shadow IT. You need policies now—not after a breach.

HIPAA Rules Most Impacted by AI

Here’s how existing HIPAA regulations apply directly to generative AI systems:

• Privacy Rule: Requires legal justification for PHI use. AI doesn’t get a pass.

• Security Rule: Requires safeguards (encryption, access control, audit logs) around PHI—including where AI systems operate.

• Breach Notification Rule: If AI causes or contributes to a PHI exposure, notification obligations apply—regardless of whether the breach was human or algorithmic.

• Emerging State Laws: California, among others, is drafting laws specifically targeting AI in care settings. Expect more.

Building HIPAA-Compliant GenAI Workflows

You don’t need a 200-page policy manual. But you do need structure. Here’s a step-by-step approach:

1. Perform an AI-Specific HIPAA Risk Assessment

Map every place PHI touches (i.e. conduct a data flow diagram exercise) your AI:

• Inputs (prompts, logs)

• Outputs (text generation, summaries)

• Vendors (storage, training)

Update your security risk analysis (requires to occur under HIPAA) to include AI workflows.

2. Choose HIPAA-Ready Vendors

Ask:

• Will they sign a BAA?

• Do they store or train on your data? What assurance do you have?

• Are there strong encryption and access controls?

Use solutions like Azure OpenAI or Med-PaLM 2 under HIPAA terms—or build in-house.

3. Apply “Minimum Necessary” and De-Identify When Possible

Strip identifiers from prompts and outputs when feasible.

• Use de-identified or synthetic data for training.

• Avoid unnecessary details in outputs (e.g., full names, exact dates).

• Validate anonymization techniques. (Anonymization is KEY when utilizing LLMs!)

4. Implement Technical Controls

Lock down your AI pipeline:

• Encrypt data at rest and in transit. (Encryption can slow things down and add costs, but in healthcare, protecting patient data isn’t optional—you just need to design smart so security doesn’t kill performance.)

• Use role-based access controls.

• Maintain audit logs.

• Segment and monitor networks.

• Sandbox AI environments and outputs. (!)

AI models deserves the same protections as your EHR or production database.

5. Write Clear AI Use Policies

Define:

• Approved AI tools.

• Forbidden practices (e.g., no PHI in consumer apps).

• Output validation procedures.

Train all staff—including engineers—on these policies. Include real-world examples, as your people will learn better from stories than from Key Risk Indicators (KRIs) and stats.

6. Monitor and Review AI Outputs

Use:

• Human-in-the-loop review for clinical tasks. (Humans should still very much be part of processes and final QA)

• Tools to scan AI outputs for PHI leaks or errors. (examples include Microsoft Azure Content Safety + Azure OpenAI Content Filters, Amazon Macie, Google DLP API, and Private AI)

• Prompt engineering to reduce hallucinations.

Quality control work best when being built in—not bolted on.

7. Detect and Block Shadow AI

Use Data Loss Prevention (DLP) tools and Cloud Access Security Brokers (CASB) to:

• Detect when staff try to upload PHI to unapproved tools.

• Block risky destinations (e.g., public ChatGPT).

And critically—offer approved internal alternatives to meet the same productivity needs.

8. Prepare for Breaches

Have a documented incident response plan for AI-related events:

• Prompt injection?

• Vendor breach?

• Hallucinated disclosure?

Run tabletop exercises. Know who notifies the U.S. Department of Health and Human Services (HHS – enforcement arm of HIPAA in the US). Know when to notify patients.

HIPAA Compliance as a Growth Strategy

Founders often see compliance as red tape. But in healthcare, trust is currency.

Being HIPAA-compliant will assist in:

• Closing enterprise deals faster.

• Accelerating security reviews.

• Building credibility with payers, providers, and regulators.

• Increasing valuation during due diligence.

Investors notice when a startup has airtight governance (or at least they should with some due diligence). They also notice when it doesn’t.

Startups that bake in privacy by design—versus scrambling post-breach or fine—will ultimately win.

In Closing

Generative AI is already transforming healthcare. But PHI doesn’t care if your product is still in beta.

If you’re not clear on how AI intersects with HIPAA, you’re playing with fire—and regulators, investors, and enterprise customers will notice first.

Lock down your prompts. Secure your vendors. Govern your models.

HIPAA compliance isn’t a blocker—it’s a differentiator. It’s the signal that you’re ready to scale.

P.S. If you're looking for insights on cyber risk management, security compliance, and practical ways to protect your business, you're in the right place. I help organizations build security strategies that work. Follow me for actionable content or reach out to discuss how we can strengthen your cybersecurity posture!