Inside Amsterdam’s high-stakes experiment to create fair welfare AI

Amsterdam tried to do things fairly when it created an AI model to evaluate welfare recipients, but it ended up with an algorithm that was just as biased as humans. In this edition of What’s Next in Tech, find out what went wrong.

⚡ What's Next in Tech Readers: Claim your special, FREE 30-day trial subscription today.

The Dutch city thought it could break a decade-long trend of implementing discriminatory algorithms. Its failure raises the question: can these programs ever be fair?

There’s an ongoing debate about whether algorithms can ever be fair when tasked with making decisions that shape people’s lives. Over the past several years of efforts to use artificial intelligence in this way, examples of collateral damage have mounted: nonwhite job applicants weeded out of job application pools in the US, families being wrongly flagged for child abuse investigations in Japan, and low-income residents being denied food subsidies in India. 

Proponents of these assessment systems argue that they can create more efficient public services by doing more with less and, in the case of welfare systems specifically, reclaim money that is allegedly being lost from the public purse. In practice, many were poorly designed from the start. They sometimes factor in personal characteristics in a way that leads to discrimination, and sometimes they have been deployed without testing for bias or effectiveness. In general, they offer few options for people to challenge—or even understand—the automated actions directly affecting how they live. 

When Amsterdam set out to create an AI model to detect potential welfare fraud, officials thought it could break a decade-plus trend of discriminatory algorithms that had harmed people all over the world.

City officials in the welfare department believed they could build technology that would prevent fraud while protecting citizens’ rights. They followed these emerging best practices and invested a vast amount of time and money in a project that eventually processed live welfare applications. But in their pilot, they found that the system they’d developed was still not fair and effective. Why? 

Lighthouse Reports, MIT Technology Review, and the Dutch newspaper Trouw have gained unprecedented access to the system to try to find out. In response to a public records request, the city disclosed multiple versions of the algorithm and data on how it evaluated real-world welfare applicants, offering us unique insight into whether, under the best possible conditions, algorithmic systems can deliver on their ambitious promises.  

The answer to that question is far from simple. Read the story.

Sign up for The Algorithm, our exclusive AI newsletter, to demystify the latest AI breakthroughs and to cut through the hype.

Get ahead with these related stories:

1. Responsible AI has a burnout problem — Companies say they want ethical AI. But those working in the field say that ambition comes at their expense.
2. Are we ready to hand AI agents the keys? — We’re starting to give AI agents real autonomy, and we’re not prepared for what could happen next.
3. It’s pretty easy to get DeepSeek to talk dirty — Most mainstream AI chatbots can be convinced to engage in sexually explicit exchanges, even if they initially refuse.

Image: Chantal Jahchan