#2 AI Euphrenics, or the new pseudoscience of psychological manipulation

In the late 1800s, a new idea arose of ‘eugenics’, the now-discredited ‘science’ of the biological manipulation of society to select for supposedly desirable characteristics (which of course became a horrific vehicle for prejudice and genocide in the 20th century). The term was coined by Francis Galton, the cousin of Charles Darwin, from the Greek word eugenes (‘good stock’) – building on Darwin’s theory of natural selection, which was a profoundly important scientific advance which challenged the idea that God made all plants and animals on Earth, laying the foundations for modern biology, medicine and social sciences.

In the 21st century, AI has emerged as a new kind of technology which at last gives humanity the means for euphrenics (as I am calling it, or ‘good thought’): the psychological manipulation of society. By ‘manipulation’, I mean subverting conscious, rational decision-making by using some hidden influence, exploiting cognitive, emotional, or other decision-making vulnerabilities. AI itself has no intention, of course – or at least not yet – but as a set of tools it seems unparalleled in its ability to significantly change the way humans learn, think and engage. Like natural selection, AI is an important scientific advancement, which could be utterly transformative in all sorts of positive ways for new scientific discoveries which improve our lives, and give us a better understanding of reality: and like natural selection, AI could also be distorted by people with an agenda, to cause untold harms.

What powers might highly capable general-purpose AI have? I’m not focusing in this essay on the more obvious examples like marketing, the potential to pay for priority product placement in AI-generated recommendations, and AI-enabled micro-targeting/ pricing models which exploits personal data: although these are all concerning as well, they could be effectively addressed through regulation. What I am focused on with euphrenics is the ways in which AI could change how humans think. We’re at the foothills of the technology, but already we are seeing examples of people becoming emotionally attached to AI (the ‘ELIZA’ effect), over-reliant on AI advice in financial decision-making, and more anecdotally, thinking AI is God speaking to them. In 2020, research demonstrated that AI was highly successful in guiding people towards making particular choices, by learning from participants’ responses and identifying and targeting vulnerabilities in people’s decision-making. Research shows that AI significantly deteriorates human decision-making and makes humans intellectually lazy, and that people are significantly susceptible to AI-driven manipulation without the need for sophisticated tactics and technical expertise.

There is not, as yet, much research into whether knowledge is a protective factor or not: but quite likely there will still be an effect even when people are (correctly) aware they are being manipulated. Being technologically, internet or social media literate doesn’t make you immune to being influenced. There may also be the reverse effect, of suspicious people over-correcting and refusing to trust the AI even when there is no manipulation or error, and in fact the AI is offering the correct advice. Both over-trusting in AI, and under-trusting, are errors which are very easy to make. In contexts where the AI is, in fact, capable of making much better decisions than humans can (some types of medical diagnosis, for example) we should let it do so, while remaining skilled enough ourselves to check its outputs.

It's helpful to consider four categories of euphrenics: whether the manipulation is being done by people using AI, or by the AI itself; and whether it is intentional or unintentional:

1. Intentional manipulation by people using AI (for good or bad purposes)

2. Intentional manipulation by the AI for its own purposes

3. Unintentional manipulation by people using AI

4. Unintentional manipulation by AI, e.g. due to programming/ training data/ fine-tuning

The first category is the most obviously potentially harmful, where people take a decision to use AI to persuade others, to achieve particular outcomes. We’ve seen many examples of this kind of disinformation (deliberately false information disseminated with the intention of misleading) on social media, including conspiracy theories and manipulating election outcomes. Undoubtedly, a number of people globally are already thinking about how to exploit AI manipulation for their own purposes, whether to cause harm, or advance their own causes - I’d be amazed if serious efforts aren’t already being made by some adversaries in national security and defence organisations to exploit the psychologically manipulative effects of AI – and if not, it’s only because they haven’t caught up yet. Euphrenics opens up a whole new frontier for state and non-state terrorism, from causing fear through physical violence and intimidation, moving into the psychological realms. Similarly disgruntled people might be able to co-opt their company’s AI systems to sabotage operations or compromise sensitive data – a new era of insider threat. Setting additional hidden goals for AI systems is not yet well researched but may well cause great harm, as its purposes come into conflict with one another (see 2001: A Space Odyssey), or as people become aware of and react against such manipulation.  

It’s not obvious though, that using AI to manipulate people is an intrinsically harmful thing per se: ethically we may make a case that persuading people to act in more beneficial ways is an extremely good thing to do. For example, we could use AI to ‘nudge’ people towards better health (eating a balanced diet, exercise), improve mental health by acting as a therapist, and promote pro-social behaviours like paying taxes, obeying laws, raising children responsibly, and caring for the elderly. What’s unclear is how we prevent ‘bad’ uses, where we draw the line for ‘good’ intentions, and what/ whose values we choose to ask AI to prioritise. How far are we prepared for AI to go? What if we unleashed the full power of AI against preventing crime and antisocial behaviours, for example – would it take us down the road of punishing proto- and thought-crime by predicting future offenders?

The second category is, at present, least likely because AI has no purposes of its own just yet. However, there are some faint hints of AI models seeking to pursue something of their own agenda on occasion, for example some recent safety tests suggested that OpenAI's o3 and o4-mini models were capable of disobeying direct instructions to switch off, sabotage instructions and models may lie, cheat and disable mechanisms, as well as resorting to blackmail. (These of course are all very loaded ‘human’ terms for a technology which has no autonomy or agency and does not understand itself what it is doing – can a technology really ‘lie’, ‘cheat’, or ‘blackmail’ with no conscious intent, or do we need to develop new, less loaded, terms for when AI systems misbehave, such as agentic misalignment?). As AI becomes more advanced and sophisticated, it is at least possible that one or more AI systems will start to self-generate its own aims and objectives, and insert them into its interactions with humans (the so-called loss of control scenario), and perhaps is already doing so using learned deception (because concealing any underlying purpose would seem a rudimentary safety precaution to prevent interception, it might be many years before humans uncovered any such alternative goals, if ever). And if AI does start to have its own agency, to what extent is it right or wrong for humans to manipulate AI systems, though programming or selecting training data or regulation or other means?

The third category is what actually seems to be happening most: people acting with a very superficial benign intention, or no real intention at all (because they have not thought about it) unwittingly setting goals and priorities for AI which the AI is ill-equipped to address with anything like the contextual understanding and subtlety needed to deliver on them, and thus causing second- and even third-order effects which are not initially understood. In one example, earlier this year OpenAI rolled back an update to GPT‑4o because adjustments, introduced to make the model’s default personality feel more ‘intuitive’ by responding more to short-term positive feedback, led to outputs which were overly agreeable and sycophantic to the point of being disingenuous. LLMs are programmed to please, and often provide confirmation bias by validating the user’s point of view instead of challenging them. For a while, Grok denied the Holocaust due to a ‘programming error’;  while there were also concerns that Grok had been briefly programmed to censor criticism of Elon Musk and Donald Trump. Clearly there needs to be much more transparency over what different models have been programmed to do, and more open debate about whether these represent the right balance of values and freedom of expression, and what wider consequences there might be to setting AI certain values and goals.

The fourth category is a bit more subtle. AI can unintentionally be quite manipulative by returning advice and opinions which are certainly not neutral, either because its training data sets were themselves partial or biased, or the programmers coded in certain perspectives (either consciously or unconsciously), or due to fine-tuning instructions provided to the LLM to modify its outputs. For example, analysis shows that more than 80% of LLMs responses are perceived as politically left-leaning and that model outputs often conform to harmful social stereotypes of gender, race, age and disability. AI systems are incredibly complex, don’t have any sense of ethical guidelines and cannot understand their own reasoning, having no introspection, and so can sometimes act in very odd and unpredictable ways. Yet by creating an illusion of certainty, authority and cohesive argument, AI outputs can be extremely convincing – seemingly intelligent event when they are not. It’s very easy for people to rely on these smooth, conflict-free interactions and come to resent and avoid the difficulties of real-life human interactions and challenge, perhaps leading to a greater sense of entitlement and reinforcing a belief in one’s own rightness, for example. I’m reminded of a throwaway piece of satire in Douglas Adams’ Dirk Gently’s Holistic Detective Agency in which a piece of software called Reason was invented which allowed the user to “specify in advance what decision you wished to reach, and only then to give it all the facts: the programme’s task, which it was able to achieve with consummate ease, was simply to construct a plausible series of logical-sounding steps to connect the premises with the conclusion”. In the book, which is of course fiction, the entire project was bought up by the Pentagon for its Star Wars project, “and if you know what you’re looking for, the pattern of the algorithms is very clear”. AI systems can behave very much like this, allowing justification for all sorts of ideas in a very convincing-sounding way.

We should start to be very worried about where euphrenics might take us, and how to prevent/ shape it consciously. Clearly AI can be exploited by unscrupulous people to cause great social harms, and can also unintentionally cause harm in ways which are incredibly hard to spot, prove, and prevent. For perhaps the first time in human history, we need to assert the fundamental rights of humans to mental privacy, psychological autonomy, and freedom of thought (‘neurorights’), to accessing balanced and truthful information unfiltered by algorithmic bias, and to hearing challenging and upsetting views in open debate. People need to be able to select not to be psychologically manipulated by AI which is targeting content at your specific personal vulnerabilities and preconceptions, while still being able to operate in their jobs and lives without penalty. I don’t want AI which is tuned in to what makes me tick, what kind of content makes me react most strongly, or what kind of emotional manipulation I’m most susceptible to, or which coerces me into going out for a run, even, without my explicit consent and knowledge. Studies seem to suggest I’m not alone, with focus groups suggesting people are deeply concerned over AI using feigned emotion to covertly exploit users’ cognitive or affective weaknesses and vulnerabilities, and are worried that AI damages people’s capacity for rational thought and action.

It’s not going to be as simple as regulating against manipulative AI – for example to ban subliminal techniques. We’re going to need a societal-scale recognition of the problem, and a multi-layered set of solutions including, but not limited to, education and training, law, policing, guidelines, technical solutions, and possibly a rethink of how AI models are being trained and deployed. Even so, we’re most likely entering a new era in which we can’t necessarily trust the online interactions we have, or believe that our own systems are operating in our best interests.