The Evolving Landscape of GenAI Regulation

Amid the rapid proliferation of Large Language Models (LLMs) and Generative AI, along with their growing influence on regulatory policies and downstream applications, the EU AI Act seeks to define General-Purpose AI (GPAI) models by distinguishing them from AI systems and applications. This distinction primarily considers their broad applicability and ability to perform a wide range of tasks. An alternative differentiation focuses on their underlying architecture and the significant computational power these models require for unsupervised learning from publicly available and increasingly multimodal data. Since a model’s full scope of capabilities often becomes apparent only after deployment, regulators require capability thresholds to serve as predictors of a model’s potential systemic impact.

Being risk-oriented, the AI Act highlights this aspect as a source of systemic risk, meaning that highly capable AI models can potentially destabilize entire systems, such as banking or financial markets. Risk is further compounded by the widespread applicability of such models, as they can be deployed in various ways, including via APIs and local installations. They may also be modified downstream, fine-tuned, or integrated into new GPAI models or AI systems. To prevent risk from escalating into systemic risk, the AI Act—aligned with its tiered risk-based approach—establishes specific rules for GPAI models, particularly those that pose systemic risks. These rules apply not only to the models themselves but also when they are integrated into an AI system placed on the market, at which point both GPAI and AI system obligations come into effect.

For proponents of softer AI regulation, the fundamental question is: Why regulate GPAI models before they are applied to a specific use case?

This approach differs from existing regulatory models, such as Software as a Medical Device (SaMD), where classification depends on intended use. Risk-based regulation implies that risk assessment should depend on the declared intent behind an application. However, while randomized clinical trials have shown that LLMs do not significantly influence physicians’ diagnostic reasoning, large-scale discrimination often goes undetected for extended periods, raising concerns about bias propagation and model collapse as synthetic data is repeatedly reused.

Risk Mitigation and Regulatory Compliance

Recognizing this challenge, the AI Act stipulates that GPAI providers, given their foundational role in downstream AI systems, have distinct responsibilities within the AI value chain. To facilitate integration and ensure regulatory compliance, proportionate transparency measures are mandated. Additionally, for models that pose systemic risks, evaluations must be conducted to assess and mitigate these risks, using methods such as structured Q&A sets and adversarial testing.

However, as GPAI models are increasingly applied to specialized, domain-specific language environments, generalist evaluations may fail to detect critical risks, such as hallucinations (false positives) and ignored evidence (false negatives). In high-stakes applications like healthcare, these failures can significantly impact model inference accuracy and reinforce discriminatory outcomes. For example, if a model excludes demographic subpopulations in a disease phenotype, it may lead to systematic exclusion from accurate diagnosis or treatment recommendations—effectively redlining these populations.

Systemic risks demand proactive governance, both internal and external, through stringent oversight of training data sources and quality. These risks must also be transparently communicated to downstream users via comprehensive technical documentation. The EU GPAI Code of Practice (CoP) is being meticulously developed to strike the right balance between enabling innovation and enforcing guardrails. While some may see this as a constraint on the productivity potential of GPAI and LLMs, it is, in fact, a critical opportunity—one that advances transparency in training data quality and reliability, its implications for equity and inclusion, and, ultimately, the responsible deployment of AI.

For GPAI models released under free and open-source licenses—where parameters, weights, and architectural details are publicly accessible—exceptions to transparency requirements apply under the AI Act, unless the model poses a systemic risk. In such cases, open-source status does not exempt it from compliance. However, these transparency waivers explicitly do not extend to training data. Since open-sourcing a GPAI model does not necessarily disclose details about the datasets used for training or fine-tuning—nor ensure compliance with copyright law—GPAI models remain subject to transparency obligations regarding training data content and adherence to copyright regulations, particularly in identifying and respecting reserved rights.

Ensuring Ethical and Safe AI (Data) Deployments

In many ways, the AI Act serves as a preamble to the AI Liability Directive (AILD), which governs liability for damages caused by AI products. While the AI Act seeks to minimize risks to safety and fundamental rights, it does not outright prohibit AI systems with residual risks from being placed on the market. Consequently, harm remains a possibility when AI systems are deployed in the EU, necessitating the AILD’s liability provisions to enable damage claims.

However, between potential harm and the widespread use of unchecked GPAI and LLMs lies a vast mitigation space—particularly for preventing discrimination and manipulation. Beyond infringing on human rights, these risks can also undermine the cost-effectiveness of AI-driven solutions in critical applications such as cancer diagnosis and treatment. Historically, medical devices regulated by the FDA, EU MDR, and UK MDR have been approved without fully accounting for potential discriminatory biases. Furthermore, excessive reliance on unsupervised learning—without a human in the loop—can generate erroneous meta-associations between tokens, leading to flawed inferences and harmful clinical decisions.

Regulatory oversight is essential to prevent these risks from materializing. However, it must extend beyond AI systems themselves to the rapidly evolving real-world data (RWD) ecosystems that shape them. The quality and reliability of both RWD and real-world evidence (RWE) are critical to ensuring that unsupervised learning remains grounded in reality rather than drifting into fabricated correlations or misleading inferences.

These data ecosystems are dynamic, not merely built on existing datasets but continuously shaped by lived experiences, contextual nuances, and evolving knowledge. Without rigorous validation and oversight, AI models risk learning from distorted or incomplete representations of reality, ultimately compromising their safety, efficacy, and fairness—especially in high-stakes applications like healthcare.

For AI regulation to be effective, it must prioritize transparency in data sources—ensuring they meet rigorous standards for equity, diversity, and inclusion (EDI). Without clear provenance and accountability, training datasets risk detaching from truth, leading to AI models that reinforce bias or hallucinate outputs due to excessive reliance on synthetic data.

Future Developments in AI Regulation

Access to domain-specific real-world datasets is critical to ensuring the responsible and compliant use of Generative AI and LLMs in healthcare. RWD serves as the foundation for a clinically meaningful, adaptable, and reliable AI ecosystem for diverse medical devices and AI applications.

It is essential to clearly define and distinguish RWD from big data, population datasets, and other fragmented data sources. A definition of RWD that merely describes it as non-RCT-generated data is insufficient unless it explicitly incorporates ecological validity—ensuring that the data reflects real-world conditions and is not detached from ground truth.

A compelling non-medical example of the necessity of real-life validation can be drawn from F-16 fighter jet range estimations. Imagine developing a system to predict an F-16’s operational range. If the dataset misclassifies an aircraft variant—treating an F-16C as an F-16XL—the expected range will be significantly off. Additionally, multiple external variables influence real-world range, including:

• Weather conditions (headwinds, temperature, altitude)

• Payload weight (missiles, fuel tanks, ECM pods)

• Aerial refueling capability (some F-16 variants can refuel mid-air, others cannot)

If a dataset fails to identify aerial refueling, it will systematically underestimate endurance, leading to flawed mission planning. In fragmented RWD, if key real-world factors are missing- such as the fact that the jet was refueled mid-air, results can be skewed unpredictably—depending on which data sources are used.

Just as fighter jet range estimations must be validated against real-world performance, AI models must be validated against real-world conditions to ensure accuracy, reliability, and fairness in high-stakes applications like healthcare.

Regulatory bodies recognize the urgent need to:

• Define robust RWE methodologies.

• Engage key stakeholders across healthcare and AI ecosystems.

• Develop standardized approaches for capturing, validating, and applying real-world data effectively.

Several governance frameworks are shaping this landscape. In the EU, the European Health Data Space (EHDS) regulation and the Health Data Access Bodies Community of Practice are anticipated as a pivotal step toward harmonized health data governance. In the U.S., key initiatives include:

• The Assurance Standards Guide and Reporting Checklist (ARC), by the Coalition for Health AI (CHAI).

• The Health Equity Across the AI Lifecycle (HEAAL) framework by the Health AI Partnership.

• STANDING Together, a consensus-driven initiative tackling algorithmic bias and health dataset transparency.

Yet, regulatory sandbox environments—critical for validating RWE and ensuring EDI in AI models—remain underdeveloped. Without controlled environments for testing GPAI outputs against real-world ground truths, systemic bias and model drift remain significant risks. Integrative Data Governance (IDG) is the only sustainable solution.

IDG refers to:

• Technical standards and policies ensuring high-quality data collection, storage, and processing throughout its lifecycle.

• Multi-stakeholder controls aligned with the quintuple aim: 1) Enhancing patient experience and safety, 2) Improving population health, 3) Reducing costs, 4) Improving healthcare workforce well-being, 5) Advancing health equity.

At the same time, emerging methodologies—such as N-of-1 trials and adaptive, just-in-time interventions—are using mobile health apps to generate real-world insights while simultaneously testing device functionality and clinical outcomes in sandbox environments.

Generative AI has the potential to enhance clinical practice and public health by enabling:

✅ Agile learning curves for clinicians and health systems.

✅ Personalized and dynamic decision support based on evolving real-world data.

✅ Scalable healthcare innovations that improve outcomes while mitigating bias.

Ultimately, regulatory frameworks must evolve alongside AI itself, ensuring transparent and equity-driven deployment of Generative AI in healthcare.