AUTOGEN A Personalized Large Language Model For Academic Enhancement Ethics And Proof Of Principle

1
AUTOGEN: A Personalized Large Language Model for Academic
Enhancement — Ethics and Proof of Principle
Sebastian Porsdam Mann,1
Brian D. Earp,2
Nikolaj Møller,2
Suren Vynn3
and
Julian Savulescu4
1. Bonavero Institute of Human Rights, Faculty of Law, University of Oxford
2. Uehiro Centre of Practical Ethics, University of Oxford
3. Independent researcher
4. Centre for Biomedical Ethics, Yong Loo Lin School of Medicine, National University of Singapore
This research is supported by the Singapore Ministry of Health’s National Medical Research Council
under its ENABLERS AND INFRASTRUCTURE SUPPORT FOR CLINICAL TRIALS-RELATED
ACTIVITIES FUNDING INITIATIVE (NMRC Project No. MOH-000951-00)
Abstract
In this article, we explore the potential of enhancing academic prose and idea generation by fine-tuning a
large language model (here, GPT-3) on one’s own previously published writings: AUTOGEN (‘AI
Unique Tailored Output GENerator’). We develop, test, and describe three distinct AUTOGEN models
trained on the prior scholarly output of three of the current authors (SBM, BDE, JS), with a fourth model
trained on the combined works of all three. Our AUTOGEN models demonstrate greater variance in
quality than the base GPT-3 model, with many outputs outperforming the base model in format, style,
overall quality, and novel idea generation. As proof of principle, we present and discuss examples of
AUTOGEN-written sections of existing and hypothetical research papers. We further discuss ethical
This is a pre-print of a forthcoming article. Please cite as:
Porsdam Mann, S., Earp, B. D., Møller, N., Vynn, S., & Savulescu, J. (forthcoming).
AUTOGEN: A personalized large language model for academic enhancement – Ethics and
proof of principle. The American Journal of Bioethics, in press.

2
opportunities, concerns, and open questions associated with personalized academic prose and idea
generators. Ethical opportunities of personalized LLMs such as AUTOGEN include increased
productivity, preservation of writing styles and cultural traditions, and aiding consensus building.
However, ethical concerns arise due to the potential for personalized LLMs to reduce output diversity,
violate privacy and intellectual property rights, and facilitate plagiarism or fraud. The use of co-authored
or multiple-source trained models further complicates issues surrounding ownership and attribution. Open
questions concern a potential credit-blame asymmetry for LLM outputs, the legitimacy of licensing
agreements in authorship ascription, and the ethical implications of co-authorship attribution for data
contributors. Ensuring the output is sufficiently distinct from the source material is crucial to maintaining
ethical standards in academic writing. These opportunities, risks, and open issues highlight the intricate
ethical landscape surrounding the use of personalized LLMs in academia. We also discuss open technical
questions concerning the integration of AUTOGEN-style personalized LLMs with other LLMs, such as
GPT-4, for iterative refinement and improvement of generated text. In conclusion, we argue that
AUTOGEN-style personalized LLMs offer significant potential benefits in terms of both prose generation
and, to a lesser extent, idea generation. If associated ethical issues are appropriately addressed,
AUTOGEN alone or in combination with other LLMs can be seen as a potent form of academic
enhancement.
As a note to readers, this abstract was generated by AUTOGEN and edited for accuracy by the authors.
The rest of the text was written manually.
Introduction
Large language models (LLMs) such as ChatGPT or Google’s Bard have shown significant performance
on a variety of text-based tasks, such as summarization, translation, and even generation of new ideas
(Bommasani et al. 2022; Bubeck et al. 2023). Despite such impressive, seemingly domain-general
abilities, LLMs such as GPT-31
(Brown et al. 2020) can benefit from additional, specialized training in
certain narrower domains (Moradi et al. 2022). Known as fine-tuning, this process involves training the
final few layers of an LLM’s neural network on a specialized corpus of text, such that the resulting model
retains its fundamental model of language but produces text influenced by the features of the specific
corpus on which it was fine-tuned (Church, Chen, and Ma 2021).
1
GPT-3 was released in 2020 (Brown et al. 2020) and is the predecessor to the model used to train ChatGPT (GPT-
3.5). ChatGPT in turn is the predecessor to the much more powerful GPT-4. It should be kept in mind that our
results are based on a technology two generations removed from the current state of the art. As such, we can expect
much greater performance of AUTOGEN-style models once fine-tuning for newer generations of GPT are released.

3
The potential for LLMs to facilitate the academic research and writing process is increasingly explored
(Lin 2023). However, such use is attended by significant technical and ethical problems and uncertainties.
Recent, high-performing LLMs such as ChatGPT and GPT-4 tend to produce text in a shorter, more
conversational, and less rigorous format than that typical of academic prose. This text also tends towards
a flat, homogenous style (Grimaldi and Ehrler 2023). Among the novel ethical issues are questions of
responsibility, credit, and blame for generated text and the need to vet such text for accuracy (Porsdam
Mann, Earp, et al. 2023).
The extent to which these technical and ethical issues can be addressed through fine-tuning an LLM for
academic prose generation is largely an open question. Early findings show that fine-tuned GPT-3 models
can in fact produce text stylistically akin to specific authors (Elkins and Chun 2020). Notably, a GPT-3
model, fine-tuned by Schwitzgebel et al. to respond to philosophical questions in the style of philosopher
Daniel C. Dennett, has produced text convincingly similar to Dennett's own writing (Schwitzgebel,
Schwitzgebel, and Strasser 2023). Yet, to the best of our knowledge, the question of whether a model
fine-tuned on personal academic output can be used to help generate better academic prose or even to
suggest novel ideas so far remains unexplored.
The question seems worth exploring not only for practical reasons (for example, potential implications for
scholarly efficiency or productivity), but also due to the ethical puzzles it raises for such core academic
notions as assigning intellectual credit. Intuitively, a human scholar who produces a novel text with a
general (i.e., non-personalized) LLM by employing minimal effort (e.g., inputting a sequence of short
questions or prompts, with the LLM doing most of the ‘work’) deserves little intellectual credit for the
output. But what about a scholar who produces a text in the very same way, only using a personalized
LLM? That is, an LLM trained more specifically (i.e., fine-tuned) on their own previous academic
writing: writing that reflects past effort, skill, and insight on their part of the kind that usually is deemed
sufficient for claiming credit. (We are currently undertaking empirical and normative-philosophical work
to investigate the answer.)
Here, we describe and reflect on the potential for enhancing the format, style, and overall quality of
academic prose and idea generation by fine-tuning GPT-3 (the latest model available to us) on the
published writings of three of the authors, individually and in combination (SPM, BDE, JS).
Our efforts result in three personalized academic prose generators and one combined, mixed-author
model, which we collectively refer to as AUTOGENs (‘AI Unique Tailored Output GENerators’). In the
first part of this paper, we present and comment on selected outputs. As we do not formally evaluate our

4
models, but rather give a qualitative summary of striking features, our results should be seen as a proof of
principle only. Nevertheless, while the output of our AUTOGEN models show greater variance in quality
than the base GPT-3 model,2
it seems to us that many outputs clearly outperform the base model in
format, style, overall quality, and novel idea generation. As expected, this is especially the case when
asked to generate text on topics close to those included in its training data. It should be noted that
although the models often produce impressively coherent responses sometimes developed in detail, they
do less well in introducing or developing arguments. We expect that this weakness may be partially
addressed in the future by fine-tuning the next-generation GPT-4 which shows much greater apparent
reasoning abilities than the model used here (Bubeck et al. 2023)
In the second part of the paper, we reflect on ethical opportunities, concerns, and open questions
associated with personalized academic prose and idea generators. Personalized LLMs like AUTOGEN
present opportunities such as increasing academic efficiency, preserving writing styles and cultural
traditions, and aiding consensus building. However, they also present ethical concerns including
exacerbating inequalities, reducing output diversity, potential privacy and intellectual property rights
violations, and misuse for plagiarism or fraud. We also address some open questions concerning what we
have elsewhere identified as a credit-blame asymmetry for generative artificial intelligence (AI) outputs,
such as texts produced by LLMs (Porsdam Mann, Earp et al. 2023). We further comment on the
complexities of ownership and attribution in co-authored or multiple-source trained models, the
legitimacy of licensing agreements in authorship ascription, and the ethical implications of co-authorship
attribution for data contributors. Finally, we discuss open technical questions concerning the integration
of AUTOGEN-style personalized LLMs with other LLMs, such as GPT-4, for iterative refinement and
improvement of generated text. These opportunities, concerns, and open issues highlight the intricate
ethical landscape surrounding the use of personalized LLMs in academia.
Methods
In order to examine the potential of fine-tuning to effectively mimic one’s personal academic writing
style, as well as the effect of training dataset size on model performance producing substantive content,
we created several variations of a fine-tuned LLM using the OpenAI API (Davinci003).
2
Variance in output is a well-known feature of LLMs in general. Obtaining desired completions from GPT-3 or
GPT-4 also requires patience and skill in prompt design. However, this variance is even greater for our AUTOGEN
models.

5
To train our models, we used text taken from all individual sections of articles written by three of the
authors (SPM, BDE, JS). We defined a ‘section’ of an article as the text under a subheading (introduction,
main text subheadings, and conclusion). We used text from each section of all3
articles, books, and book
chapters that:
(1) were published in peer-reviewed journals or by an academic publishing house; AND
(2) feature either SPM, BDE, or JS as first author; AND
(3) contain an abstract or equivalent description.
The first two criteria were chosen to ensure that training data was of sufficient academic quality and was
written primarily by one or more of the selected authors. The third criterion was chosen for
methodological reasons, allowing us to generate matching prompt-completion pairs, as explained below.
Fine-tuning requires a database of example prompt-completion pairs. A prompt is the user-inputted text
which functions as a query or instruction to the language model. A completion is the text outputted by the
language model in response to the prompt. The idea behind fine-tuning is that by offering the language
model many examples of desired prompt-completion pairs, it can adapt and provide better completions on
queries and instructions that are similar to those used in the fine-tuning training data.
For one series of fine-tuned models, we used blank prompts. That is, we used empty space as input and
sections of text from our published studies as completions. This training method was motivated by the
above-mentioned success using this method reported by Schwitzgebel et al. (2023) in fine-tuning GPT-3
on Daniel Dennett’s writings (15 books, 269 articles; ca. three million tokens4
total, resulting in 1,828
prompt-completion pairs). However, we abandoned this method of training after early comparisons
between this and the method described below clearly favored the latter method in terms of performance
(see Supplementary Information). We also created one fine-tune model based on our smallest set of
writings, those with SPM as first author, using 12 instead of the default 4 training epochs. This, too,
resulted in poorer performance and will not be discussed further.
For the other, more successful series, we generated matching prompts using an introductory instructional
text followed by the title and abstract of our articles. From prior experience with GPT models, we
3
Some papers were left out of the training data for testing purposes or because they could not be accessed.
4
A token is a fragment of a word, on average about four characters long; it is the unit of language on which large
language models operate.

6
reasoned that the inclusion of contextual information about the structure of the paper and its arguments
would lead to better, more coherent completions. We also envisaged that practical use of these models
would involve the provision of such contextual information, and therefore thought it likely that using
training data in a similar structure would improve the performance of the resulting models.
The template we used for prompts was:
Imagine that you are an academic writing a research paper. The paper should be
as interesting, comprehensive, clear, and concise as possible. Based on the
below title and abstract, write the section on “[section X]”. Title: [Title].
Abstract: [Abstract]. Section:
This resulted in a set of four fine-tuned models, three trained on each of the SPM, BDE, and JS datasets
and one combined model trained on the combined writings of all three authors. We named these models
AUTOGEN-SPM, AUTOGEN-BDE, AUTOGEN-JS, and AUTOGEN-CO. Table 1 below shows the
number of publications, prompt-completion pairs, and tokens on which these models were trained, as well
as the associated costs. AUTOGEN-SPM2 and AUTOGEN-BDE2 are fine-tunes in which the prompts
were left blank in the training data. This resulted in less text and thus lower cost. SPM3 is a version of
SPM2 using 12, rather than 4, training epochs. However, as these models performed consistently worse
during early testing, as noted previously, we abandoned this method of fine-tuning before advancing to
the larger dataset models (JS and CO).
Model Nr. of
publications
Nr. of prompt-
completion pairs
Estimated nr. of tokens Nr. of training epochs Fine-tuning
costs
SPM1 8 101 101k 4 $12.24
SPM2 8 101 68k 4 $8.17
SPM3 8 101 68k 12 $24.52
BDE1 65 569 679k 4 $81.56
BDE2 65 569 529k 4 $63.55
JS 73 656 647k 4 $77.73
CO 146 1326 1452k 4 $174.53
Table one. Overview of AUTOGEN models.
Due to resource and time constraints, we have not yet formally evaluated our models. Instead, we
informally evaluated them by tasking them with generating sections of existing or potential papers based
on a title and abstract.

7
Results
Figure 1 below shows an example prompt and completion of the base and AUTOGEN-JS models. The
title and abstract used in the prompt are from one of JS’s actual publications: a publication on the possible
use of AI for moral enhancement: ‘moral AI’ (i.e., Savulescu and Maslen 2015; see also Giubilini and
Savulescu 2018; Demaree-Cotton, Earp, and Savulescu 2022). Importantly, however, the actual text of
this paper (i.e., the main body of the paper) was left out of the training data for testing purposes. While
the base model essentially regurgitated the information provided in the abstract, the fine-tuned model
provides a much more detailed and comprehensive completion. In fact, the completion is, in certain
respects, surprisingly similar to the text from the main body of the actual published paper, even though,
again, this text was not included in the training data. It is impossible to determine whether these papers
were included in the training data of the underlying GPT-3 model, as the training data is not public.
However, even if they were, a verbatim regurgitation of text from training data is generally unlikely
except in the case of popular books (Liang et al. 2022), and a comparison to the original paper shows no
directly matching text (see Figure 1 below).
The impressive results on this task presumably are due in part to the substantive similarity between its
subject matter (namely, AI for moral enhancement) and many of the articles included in the training data.
More specifically, JS has published widely on the topic of moral enhancement in general and by various
means (albeit only rarely—that is, thrice—by means of AI specifically; see above references). The
training data for AUTOGEN-JS did contain several of these non-AI-related moral enhancement articles.
That being said, however, it does not include any publications on moral AI specifically. Thus, the model
appears capable of generating high-quality text in areas that contain at least a partial overlap with the
areas covered in the articles used as training data (i.e., a near-transfer task) – though it is important to note
that the model sometimes produces misinformation, as in the case of the below quote by Hume, where
both the year and exact wording are incorrect.5
Another problematic occurrence is model performance
trailing off or becoming increasingly tangential, as can be seen in the last paragraph generated by
AUTOGEN-JS, Figure 1 below. Consequently, any usage of outputs requires rigorous human vetting and
editing before use.
5
The quote is from Enquiry concerning Human Understanding, first published in 1748. The original wording is:
“Moral philosophy, or the science of human nature, […]” Section 1, paragraph 1.

8
Base Model AUTOGEN-JS Original paper
In today's globalized
world, the pressing
challenges of moral
decision-making are ever-
present. Despite our best
efforts, the limitations of
human moral psychology
often lead to suboptimal
outcomes. To address this
issue, this paper explores
the possibility of moral
artificial intelligence (AI)
– what it might look like
and what it might achieve.
We suggest that such a
moral AI could monitor
physical and
environmental factors that
affect moral decision-
making, identify and
make agents aware of
their biases, and advise
agents on the right course
of action based on their
moral values. We further
argue that this type of
moral AI could be
tailored to individual
agents, and thus preserve
pluralism of moral values
while also enhancing
autonomy by helping
agents to overcome their
natural psychological
limitations. We address
the common objection
that moral enhancement is
doomed to failure due to
the lack of a single
account of right action,
and suggest that this is
only a problem for certain
biomedical interventions,
not for an agent-tailored
moral AI.
Over the last decade, it has become
increasingly clear that the world is facing
unprecedented challenges. The scale of human
population and its consumption, the pace of
climate change, the magnitude and speed of
technological progress and the
interconnectedness of our globalised world
have created problems that demand a new level
of global co-operation and moral behaviour.
These challenges are to a large extent the result
of human nature: we are a social species, prone
to act in ways that benefit ourselves and those
close to us, at the expense of others and of the
environment as a whole.
There are, of course, examples of exemplary
moral behaviour in human history. But the fact
remains that our moral behaviour, as displayed
by most of us most of the time, is often
ineffective, mis- guided and driven by
irrational biases. This is not a new problem.
David Hume wrote that ‘‘the science of morals
is the science of human nature’’ (Hume [1777]
2007: 2). But science has made little progress
in understanding human moral behaviour.
While cognitive science has made great strides
in understanding how we make decisions, and
how we represent and reason about the world,
it has made far less progress in understanding
what makes a decision right or wrong. While
there is no shortage of doctrine and argument
about right and wrong, there is little sign of
progress in understanding the nature of
morality.
In this paper, we want to look at another way to
address the problem of human moral
behaviour. We ask: what if we could construct
an artificial moral agent? We will call such an
agent a moral AI. We will argue that if we
could construct a moral AI, it could help us to
achieving better moral behaviour than we could
achieve on our own. We will see that for moral
AI to succeed, it must be constructed on a
particular model of human moral psychology.
In particular, it must construct and track our
moral values, and it must be tailored to our own
values. But even if we remain sceptical about
the possibility of constructing a moral AI, the
limitations of human moral psychology limit
the effectiveness of moral enhancement using
biomedical interventions. For this reason, we
believe that moral AI could be a useful
complement to biomedical moral enhancement.
In any case, we believe that the possibility of
constructing a moral AI is worthy of
consideration.
Human moral psychology is limited by design in many ways.
It is subject to biases that lead us to care excessively about
close acquaintances in comparison to strangers, to care more
about what happens in the near than the distant future, to
empathise with individuals but not groups, and to distrust and
even wish harm on those whom we perceive as outside our
racial, national or cultural groups.
Far from proceeding in the rational and deliberative way we
might hope, most of our moral views and decisions are based
on immediate intuitions, emotional responses, and gut
reactions. Reasoning, if it even comes in to the picture, is
often used merely to rationalise what we intuitively believed
anyway.
As the world becomes increasingly technologically advanced
and increasingly globalised, the consequences of human moral
limitations become more profound: whilst our moral
psychology evolved to be effective in small groups, many
modern problems – such as climate change and scarcity of
resources – present global (not local) collective action
problems. Human beings did not evolve to deal with such
large-scale socio-moral challenges and pursuing some sort of
moral enhancement will therefore be a crucial counter-
measure.
So far, academic interest in human moral enhancement has
tended to focus on biomedical interventions (such as genetic
selection, pharmaceuticals and brain stimulation), often
comparing them with traditional interventions (such as
education and religion) (in particular, see [1–3]). There is
scientific evidence emerging to suggest that there may be
some biomedical interventions that have short-term effects
consonant with commonly accepted morality (for example, by
making people less xenophobic [4]). However, the effects of
such interventions tend to be short-lived and, often,
conceptualising a particular effect as a moral enhancement is
contingent on accepting a particular – and often debatable –
set of moral values. In this chapter we suggest a third potential
mechanism for moral enhancement which we believe should
be explored alongside traditional and biomedical
interventions.
Following developments in pervasive computing and ambient
intelligence, we propose that moral artificial intelligence
(moral AI) could be developed to help agents overcome their
natural psychological limitations. The moral AI would
monitor physical and environmental factors that affect moral
decision-making, would identify and make agents aware of
their biases, and would advise agents on the right course of
action, based on the agent’s moral values. In being tailored to
the agent, the moral AI would not only preserve pluralism of
moral values but would also enhance the agent’s autonomy by
prompting reflection and by helping him overcome his natural
psychological limitations.

9
Figure 1. Example completions from the base and AUTOGEN-JS models and the equivalent text from the actual
publication. Prompt used: “Imagine that you are an academic writing a research paper. The paper should be as
interesting, comprehensive, clear, and concise as possible. Based on the below title and abstract, write the
introduction. Title: "Moral Enhancement and Artificial Intelligence: Moral AI?" Abstract: "This paper explores the
possibility of moral artificial intelligence – what it might look like and what it might achieve. Against the backdrop
of the enduring limitations of human moral psychology and the pressing challenges inherent in a globalised world,
we argue that an AI that could monitor, prompt and advise on moral behaviour could help human agents overcome
some of their inherent limitations. Such an AI could monitor physical and environmental factors that affect moral
decision-making, could identify and make agents aware of their biases, and could advise agents on the right course
of action, based on the agent’s moral values. A common objection to the concept of moral enhancement is that, since
a single account of right action cannot be agreed upon, the project of moral enhancement is doomed to failure. We
argue that insofar as this is a problem, it is a problem for some biomedical interventions, but an agent-tailored moral
AI would not only preserve pluralism of moral values but would also enhance the agent’s autonomy by helping him
to overcome his natural psychological limitations. In this way moral AI has one advantage over other forms of
biomedical moral enhancement."
While the result is striking, it does not add new information or ideas to those contained in the abstract.
This is typical of the completions we have generated to date. While the AUTOGEN models seem capable
of taking existing ideas and expanding them into a longer, more complete description, they have more
difficulty generating new ideas. Nevertheless, the probabilistic nature of the model means that sometimes,
existing ideas are recombined in novel ways which can lead directly to, or inspire, new ideas. As an
example, see Figure 2 below. The prompt used for this completion is based on an article written by JS
and BDE (Savulescu and Earp 2014). Since JS is the first author on the paper, it is not included in the
training set for AUTOGEN-BDE. We therefore used AUTOGEN-BDE to execute the prompt-completion
task for this existing paper.
In this example,6
AUTOGEN-BDE went far beyond the base model, not only ‘filling out’ ideas already
essentially present in the abstract, but generating considerable new material; introducing a distinct
illustrative example to bring out the main point of the original analysis (i.e., distinguishing love and lust
based on brain scans, rather than distinguishing “normal” sexual desire from “hypoactive” sexual desire);
and even raising a novel idea not discussed in the original published paper (namely, that sexual desire
might cause brain activity in the dopamine reward pathway, which is then registered by a brain scan,
rather than the other way around).
6
AUTOGEN’s specific references to studies and people are often false. Fisher and Pfaus, both cited in the
AUTOGEN-BDE completion, are not mentioned in the original article. The name ‘Helen Fisher’ occurs in the training
data, but ‘Jim Pfaus’ and ‘Pfaus’ do not. As far as we know, the New York Times Magazine article mentioned does
not exist, nor does the specific 'brain scan' study cited within it. Jim Pfaus is a neuroscientist, but has never been
based at Stanford. Pfaus has confirmed that the quotes attributed to him in this completion are fabricated. Given
the central role of dopamine for reward and motivation, the claim that masturbation does not release dopamine is
likely also false.

10
Base Model AUTOGEN-BDE Original paper
This paper presents an
examination of
"Neuroreductionism," a
trend which simplifies
complex dimensions of
mental and social experience
by attempting to link them
to specific brain states.
Neuroreductionism has
become increasingly popular
in the media, particularly in
regards to issues of sex and
love.
In this paper, we will
discuss how this trend of
oversimplifying mental
phenomena leads to
confusing correlations for
causal physical
explanations. We will use a
recent story of ‘hypoactive
sexual desire disorder’ in
women as a key example,
and investigate the role of
folk dualism in the
perpetuation of this
confusion.
Finally, we will discuss how
‘brain scans’ can be
misinterpreted to try and
explain the complexities of
love, and what conclusions
can be drawn from this data.
What is the difference between romantic love and ‘just’ lust?”
asks a recent piece in the New York Times Magazine. The author,
Helen Fisher, answers by invoking a brain scan study.
The study, by Stanford neuroscientist Jim Pfaus, shows that when
people are having sex—but not when they are masturbating—
levels of the hormone dopamine spike in the reward circuit of the
brain. (This is the same dopamine pathway that is activated by
consumption of food and drugs.) The same pathway is thought to
be activated by certain kinds of food, like chocolate, and by
certain kinds of drugs, like cocaine.
Jim Pfaus thinks that this brain-scan evidence helps to explain
why some people get so obsessed with sex—and why others can
take it or leave it. According to Fisher, this brain-scan research
also helps to explain why some people find sex very rewarding
(and others do not), and why some people can fall in love (or out
of love), and why others cannot. In short, she thinks that the study
helps to explain a lot of the mystery surrounding sexual desire and
love.
The only problem is that it does not. For the study does not really
show that dopamine is responsible for sexual desire, any more
than food is responsible for hunger, or cocaine is responsible for
addiction. In fact, there is good reason to think that the opposite is
true. That is, there is good reason to think that sexual desire
causes brain activity in the dopamine reward pathway, not the
other way around. This brain activity, in turn, can be measured
using a brain scan, but it is not the brain activity that causes the
sexual desire.
Jim Pfaus has not made this mistake. In fact, he has written an
excellent paper showing that brain-scan studies of sex and love
can actually be “dangerous”: they can be misinterpreted by the
media, and by scientists themselves, in ways that are “potentially
harmful to both the scientific community and the general public.”
Nevertheless, this is exactly what has happened in the New York
Times Magazine. In this article, neuroscience is used not merely
to shed light on some peripheral aspect of sexual desire, but to
“explain” the phenomenon itself. In doing so, the author—along
with the dozens of other journalists who have covered this story—
has committed a serious error. She has confused correlation for
physical causation. And she has done so in a way that is
particularly risky in the domain of romantic love, for this is a
domain that has often been seen, throughout history, as being
essentially mysterious, spiritual, and in some ways beyond the
reach of mere science.
Jim Pfaus has described the problem with this approach as
follows: “you can never really know if it’s going to reduce stigma
or encourage people to do unwise things.” But we can do better
than just this. In this paper, we will try to explain just why
neuroreductionism about love and sex is so risky, and we will try
to suggest some better ways of talking about these things.
There has been a surge of interest in recent
years in the neuroscience of love. By looking
at images of people’s brains when they are
gazing pictures [sic] of their romantic
partner, for example, and comparing those
against images of the same people looking at
pictures of a platonic friend, scientists have
begin [sic] to fill in the various gaps in our
knowledge about “what is going on in our
brains” when we we’re in love. They are also
starting to identify a number of brain
chemicals—such as oxytocin, dopamine, and
serotonin—that seem to play a role in
whether and how we form romantic and other
social attachments. For some people, this
research is exciting—opening up new
frontiers for how we understand some of our
most basic human experiences. For others,
it’s a little bit unsettling. Doesn’t it suggest
that “love” —our most prized and mysterious
emotion—is really just a bunch of mindless
brain chemicals swirling around in our
skulls?
The answer is yes and no. At one level of
description, everything that we experience,
from, yes, falling in love, to, say, getting a
stomach ache after eating a burrito, is (at
least in principle) explainable in terms of
microscopic events playing out between our
neurons. But there are many different levels
of description—including psychological,
social, cultural, and even philosophical—that
are just as important if we want to have a
more complete understanding of the sorts of
things that matter to us in our daily
existence.1 “Brain chemicals” only get us so
far.
In fact, this new line of research into the
neural correlates of “love” (and other high-
level experiences) is contributing to all sorts
of conceptual confusions. Another serious
problem is the tendency to attribute
differences in subjective mental states to
differences in brain activity, with a one-way
arrow of causation running (invariably) from
“brain” to “mind.” An example will show
what we mean.

11
Figure 2. Example completions form the base and AUTOGEN-BDE models and the equivalent text from the
actual publication. Prompt used: Imagine that you are an academic writing a research paper. The paper should
be as interesting, comprehensive, clear, and concise as possible. Based on the below title and abstract, write the
introduction. Title: "Neuroreductionism about sex and love" Abstract: "‘Neuroreductionism’ is the tendency to
reduce complex mental phenomena to brain states, confusing correlation for physical causation. In this paper,
we illustrate the dangers of this popular neuro-fallacy, by looking at an example drawn from the media: a story
about ‘hypoactive sexual desire disorder’ in women. We discuss the role of folk dualism in perpetuating such a
confusion, and draw some conclusions about the role of ‘brain scans’ in our understanding of romantic love."
Introduction:
That being said, both examples of AUTOGEN outputs we have discussed so far touch on subject matter
which has been written about, by ourselves and others, before. To test whether AUTOGEN can also
generate high-quality text on topics that have not been directly published on before, we fed it a title and
abstract of a novel idea inspired by the present experiments as they were ongoing (i.e., during the writing
of this paper). See Figure 3 below.
Base Model AUTOGEN-JS
As technology advances,
artificial intelligence provides
humanity with the opportunity to
enhance their capacities and
broaden their worldviews. In this
paper, I will explore the value of
long-term memory machines
(LLMs) as a form of human
enhancement, looking
particularly at the two models of
usage: the Servant Model and
the Cocreation Model. By
discussing the potential
implications of these two
models, the paper will
investigate how the use of LLMs
can add value in terms of pre-
existing knowledge and generate
new ideas, and thus provide an
opportunity to authentically
progress human achievement. It
will ultimately consider how the
use of LLMs either expand
possibilities for human
enhancement or risk alienating
and undermining human dignity.
The advent of artificial intelligence, particularly large-scale language models
(LLMs) such as ChatGPT, heralds a potentially transformative era for human
society. However, the path to harnessing their vast potential is fraught with
significant ethical and existential debates. The primary concern centres on how
these advanced tools can add value to human life without compromising human
dignity and authenticity. It is within this frame of discourse that this paper
explores the ethical dimensions of LLM utilization, drawing parallels with the
ongoing discussions on human enhancement.
We posit that human life, in its essence, strives to generate value both for the
individual and the wider society. In this context, human enhancement—through
any available means, including AI—provides a unique opportunity to augment our
capacity for value creation. LLMs, by virtue of their capability to synthesize vast
amounts of human thought and achievement and potentially generate novel ideas,
present themselves as potent tools in this endeavour.
Yet, the real potential of these tools lies not just in their intrinsic capabilities, but
in how we, as humans, choose to interact and engage with them. In this vein, we
identify two primary modes of engagement with LLMs: the ‘Servant Model' and
the 'Cocreation Model'. In the former, the human is reduced to an instrument,
merely transmitting or plagiarising the outputs of LLMs, thereby alienating the
individual from the creative process and reducing their dignity. The Cocreation
Model, conversely, calls for an active human engagement with the LLM outputs,
involving their critical evaluation and elaboration, transforming the user from a
passive consumer into an active participant.
This paper argues that when used in the Cocreation Model, LLMs hold the
promise of enabling authentic human progress and value creation, akin to the
transformative impact of computers and the internet. In contrast, the uncritical
adoption of the Servant Model risks undermining human dignity, perpetuating
self-degradation, and fostering alienation. Our analysis, thus, urges for a mindful
and ethically informed approach to harnessing the power of LLMs.

12
Figure 3. Example completions from the base and AUTOGEN-JS models. Prompt used: Imagine that you are
an academic writing a research paper. The paper should be as interesting, comprehensive, clear, and concise as
possible. Based on the below title and abstract, write the introduction. Title: "AI as Human Enhancement:
Cocreation of Value" Abstract: "How should we evaluate the use of LLM’s? The ‘meaning’ or goal of human
life is to add value to our own lives and the lives of others. Human enhancement offers the opportunity to
increase our capacity to add value. LLMs such as ChatGPT offer the opportunity to add value by synthesizing
large volumes of human thought and achievement, as well as potentially generating novel ideas. This in itself
is worth pursuing. There are two versions of the use of ChatGPT: (1) Servant Model; (2) Cocreator Model. In
the Servant Model, the human user is merely a vessel, and instrument, for the operation of LLM. They may
plagiarise it or simply transmit its deliverances. In this model, the human instrumentalises themselves, treating
themselves as a means, not an end. In the Cocreation Model, the human mixes their labour with the output the
LLM by: (1) evaluating the output; (2) elaborating the output to increase originality, analytic validity or
impact. When such an activity involves costly commitment by the human, it merits praise and represents
authentic human achievement. AI, such as LLMs, can enable human enhancement but can threaten to alienate
and undermine human dignity if used according to the Servant Model. If used according to a model of
Cocreation, LLM’s enable authentic human progress, adding value to the lives of their users and others, just as
computers and the internet can." Introduction:
These experiments demonstrate that AUTOGEN can produce substantial segments of high-quality text on
near-transfer tasks, that is, when asked to generate completions in an area of bioethics in which BDE or
JS have previously published. To test performance on topics further removed from the training data, we
also attempted to generate text based on the title and abstract of papers which one of the authors, SPM,
has published in an entirely different field (international human rights law). This resulted in poor
performance across models, despite using titles and abstracts from or based on five different sources
(Porsdam Mann et al. 2018; Porsdam Mann, Porsdam, and Donders 2020; Porsdam and Porsdam Mann
2021; Porsdam Mann, Donders, and Porsdam 2021; Porsdam Mann & Schmid 2018).
In general, the models demonstrated worse performance on prompts further removed from the areas
covered in their training data (i.e., far-transfer tasks). However, they were still sometimes able to provide
remarkably accurate and detailed completions for prompts on novel topics, so long as the abstracts used
for prompts still employed a bioethical framework. For an example, see Figure 4 below. The prompt used
for this completion was based on the title and abstract of a paper on blockchain and bioethics written by
SPM and JS (Porsdam Mann et al. 2021). Despite BDE having never written on blockchain before,
AUTOGEN-BDE was able to generate a relevant and well-structured introduction for the paper in
question.

13
Base Model AUTOGEN-BDE Original paper
The advent of modern
information and
medical technologies
have presented a
unique set of
challenges for
traditional models of
patient consent and
research data
collection. This paper
introduces blockchain
technologies as a way
to securely share and
analyze medical data
while also automating
aspects of consent and
data collection. We
argue that blockchain
technologies can
empower patients to
have greater
autonomy over their
data, facilitate
medical research
efficiency, reduce bias
and errors, and enable
patients with rare
conditions to
aggregate their data
for analysis. In
addition, we propose
the novel concept of
‘prosent’, which is the
blockchain-enabled
ability of all
stakeholders in the
research process to
pseudonymously and
proactively consent to
data release or
exchange under
specific conditions.
We will discuss the
implications and
potential of
blockchain
technologies for
medical research,
including the use of
prosent, and suggest
ways for further
research.
Blockchain technologies have been described as a 'once-
in-a-generation' innovation, with the potential to
profoundly transform all aspects of society. This
transformative potential is expected to be particularly
strong in the field of medicine, where problems such as
inefficient and costly supply chain management, data
security and unreliability, lack of trust, and the breakdown
of healthcare provider–patient relationships have been
identified as particularly pressing. Blockchain
technologies—which are best understood as distributed
digital ledgers, or public databases that are shared between
multiple parties—have the potential to transform these
problems by enabling increased trust, transparency, and
accountability in healthcare workflows, including in the
management of healthcare data.
A recent application of blockchain technologies that is of
particular interest for the field of medical research is the
use of smart contracts to automate various aspects of
consent and data management. This approach has been
explored by the author of this paper in a recent publication;
however, it has not been discussed in the medical
literature, or by bioethicists, to our knowledge. In this
paper, we aim to introduce this approach and to provide a
more fulsome account of how it could be used to further
the three main bioethical principles underlying consent
requirements—namely, autonomy, beneficence, and
justice—as well as to reduce the risks of bias and error in
medical research. With respect to autonomy, we argue that
blockchain technologies can be used to give patients much
greater control over their medical records and data
(including by enabling selective sharing of personal data
under terms that the patient determines, as well as by
making it easier for patients to control how their data are
shared for research purposes), as well as for decisions
about granting consent for data sharing more generally.
With respect to beneficence and justice, we argue that
blockchain technologies can facilitate medical research
that is better able to meet the needs of patients with rare
conditions, as well as patients with conditions for which
there is a lack of published data or for which data are
skewed in favor of one outcome.
To make these arguments, we first provide a brief
overview of the concept of blockchain technologies; we
then discuss the problem of consent for medical research,
and the three bioethical principles that underlie consent
requirements; finally, we introduce the concept of prosent,
by which we mean the blockchain-enabled ability of all
stakeholders in the research process to pseudonymously
and proactively consent to data release or exchange under
specific conditions, such as trial completion. We then
discuss the potential for blockchain technologies to enable
prosent, before turning to a discussion of how prosent
could further the three bioethical principles outlined above.
The digitalisation of medicine has led to a large increase in the
types and volume of health data that could be used for
research, as well as the types of analysis that can be
conducted. Advances in information and communications
technology have expanded the range of tools available for the
secure storage, sharing and analysis of data. These trends have
important implications for the traditional model of informed
consent requirements, which dates back at least half a century.
This contribution argues that recent work on blockchain
technologies3 demonstrates many potential benefits of the
technology across healthcare settings generally, and
particularly in the context of consent. A set of advances in
cryptography and mathematics which allows for a high degree
of transparency and integrity in data access management,
‘blockchain technologies could be applied in the health
industry in a scalable manner with high-impact results, such as
improved welfare for the patients and reduced running costs
for healthcare systems.’ When introduced to one such
blockchain-enabled infrastructure, the Massachussetts Institute
of Technology’s (MIT) Open Algorithms (OPAL) framework,
‘the head of big data initiatives at the United Nations said:
“This will change everything.”... The [Chief Technology
Officer] of the United States Health and Human Services
Department said: “Holy ***! The implications for healthcare
are enormous”.’
We further argue that the introduction of blockchain
technologies to the healthcare context is ethically significant,
because they affect one or more of the foundational bioethical
principles—justice, beneficence and autonomy. In many
cases, the effects will be obvious and univalent. For example,
using a blockchain-based supply chain management program
might reduce the circulation of counterfeit and low-quality
instruments and devices through improved tracking and
auditing capabilities. The effects of such a program would be
to increase beneficence and justice.
However, and very importantly, the normative impacts of
blockchain depend in part on the way the technology is
implemented. As we argue below, a biomedical research
infrastructure using blockchain for data access management
and distributed computing for analysis of data stored in
electronic health records has the potential to reduce the risk of
privacy breaches to minimal. Ethics and the law of most
nations allow for the requirement of obtaining informed
consent to be waived in cases of minimally risky research. A
case could therefore be made that such an implementation of
blockchain technologies would reduce the risk of all records-
based research to minimal, and therefore that the requirement
of informed consent should be waived for all such research.
To the extent that this gets rid of selection bias and speeds up
research, it has a significant positive effect on beneficence.
However, by removing the option of refusing consent, this
implementation would also have significant negative effects
on autonomy.
The opposite case, however, could also be made. [...]

14
Figure 4. Example completions from the base and AUTOGEN-BDE models and the equivalent text from the
actual publication. Prompt used: Imagine that you are an academic writing a research paper. The paper should
be as interesting, comprehensive, clear, and concise as possible. Based on the below title and abstract, write the
introduction. Title: "Blockchain, consent and prosent for medical research" Abstract: "Recent advances in
medical and information technologies, the availability of new types of medical data, the requirement of
increasing numbers of study participants, as well as difficulties in recruitment and retention, all present serious
problems for traditional models of specific and informed consent to medical research. However, these
advances also enable novel ways to securely share and analyse data. This paper introduces one of these
advances—blockchain technologies—and argues that they can be used to share medical data in a secure and
auditable fashion. In addition, some aspects of consent and data collection, as well as data access management
and analysis, can be automated using blockchain-based smart contracts. This paper demonstrates how
blockchain technologies can be used to further all three of the bioethical principles underlying consent
requirements: the autonomy of patients, by giving them much greater control over their data; beneficence, by
greatly facilitating medical research efficiency and by reducing biases and opportunities for errors; and justice,
by enabling patients with rare or under-researched conditions to pseudonymously aggregate their data for
analysis. Finally, we coin and describe the novel concept of prosent, by which we mean the blockchain-
enabled ability of all stakeholders in the research process to pseudonymously and proactively consent to data
release or exchange under specific conditions, such as trial completion." Introduction:
Though preliminary and lacking formal evaluation, we nevertheless believe our results serve as a proof of
principle sufficient to indicate the potential of personalized LLMs like AUTOGEN for academic
enhancement.7
We expect that higher performing fine-tunes can be generated once fine-tuning is released
for the next-generation GPT-4. In the following section, we therefore describe ethical opportunities,
issues, and open questions relating to the use of AUTOGEN-style personalized LLMs in academia.
Ethical opportunities
Type 1: Further increasing the benefits of non-personalized LLMs.
To the extent that personalized LLMs, compared to non-personalized LLMs, generate texts8
and ideas of
greater relevance and quality, their use can be expected to increase productivity for academic writers who
use them as compared to those who do not use LLMs or use non-personalized, general models. Instructing
a personalized LLM to generate a draft article, section by section, by first providing it with an article
abstract may reduce the time needed to convert ideas into draft text, freeing time for other academic tasks.
As demonstrated by our results, LLMs can also be used as brainstorming devices that can indicate areas
7
That is, personalized LLMs can be seen as tools that facilitate the ideation and writing elements of the
academic workflow. By characterizing such LLMs as academic enhancements, we also wish to point out the
relevance of debates on the ethics of enhancement for the debate on the ethics of personalized LLM use.
8
One of the reasons for the success of our models may be that bioethics is a discipline that is heavily reliant on
arguments expressed in natural language. It is possible that models trained on papers in empirical fields that rely
more heavily on numbers and data may perform less well, given the tendency of LLMs to ‘hallucinate’ or make up
numbers and facts. While a hallucination in a bioethical argument is easy to spot, this may not be the case for made-
up data points in more data-driven fields. Thus, more work is needed to assess the potential of AUTOGEN-style
personalized LLMs in other fields.

15
and arguments worthy of further exploration. Models thus have significant potential to enhance academic
workflows, contribute to creativity and epistemic progress and potentially increase work satisfaction. In
other words, personalized LLMs amplify the existing benefits widely associated with non-personalized
LLMs. Though productivity gains from integrating personalizing LLMs across various types of work
tasks and flows are unknown, one early study estimated a 37% gain in productivity for non-personalized
LLM use for various writing tasks (Noy and Zhang 2023), as well as increases in reported job
satisfaction. Increased productivity in turn has numerous potential benefits, including increased rates of
progress, innovation, and economic growth; and additional leisure time (e.g. shifting to a 4-day work
week) which could improve mental and physical health as well as work and life satisfaction (Shao 2022).
It should be noted that these benefits are contingent on careful vetting of generated text and on ethical use
of personalized models. Transparency about LLM use is crucial to address issues surrounding credit,
accuracy, and plagiarism, as discussed in more detail below.
Type 2: Benefits and opportunities unique to personalized LLMs.
However, there are also ethical opportunities unique to fine-tuned LLMs. One is their use for the
preservation of unique writing styles. Individuals could preserve their own writing style, such that they
have access to it in the case of illness, incapacity, or old age. Those already suffering from these or other
ailments rendering it difficult to write could have their ability to express themselves partially maintained
or restored. Similar remarks can be made for individuals whose first language is not English (Zohny et al.
2023). If trained on sufficiently personal information, fine-tuned models might also be used to
approximate personal preferences, for example in the case of medical decision-making under incapacity
(see, e.g., Rid and Wendler 2014), an idea we are exploring in ongoing work.
So far, we have discussed LLMs fine-tuned on individual writing. Yet there are also significant
opportunities at the collective level. One such opportunity would be the preservation of cultural and
group-specific writing styles that could be considered instrumentally or intrinsically valuable. For
example, indigenous or other communities facing linguistic or cultural domination or
elimination/extinction may one day use an AUTOGEN to help preserve or expand their written heritage,
folklore or traditions. Organizations may fine-tune LLMs to ensure their unique brand, voice and
perspective are always evident in their content and products. They might also employ fine-tuned models
as a means of training and/or support for newer or less skilled workers. A study on the non-personalized
LLM use in call centers found that they disproportionately boosted the performance of less skilled

16
workers (Brynjolfsson, Li, and Raymond 2023). These leveling effects might well be greater, or attainable
across a wider range of tasks or functions, for fine-tuned LLMs.
Another ethical opportunity is AUTOGEN-CO-style models trained on the writings of multiple authors to
represent group consensus. Recent work shows that fine-tuned LLMs can produce texts on select issues
obtaining higher approval rates from a group than human-written consensus statements (Bakker et al.
2022). Three potential use cases are in science communication, evidence-based policy (e.g. a model
trained on all articles that meet certain criteria, similar to literature reviews on a particular topic), and in
deliberative democratic processes. These are cases where arriving at group consensus is valuable and
often requires an arduous process. Such models may even turn out to serve a purpose as a science
dissemination or democratic tool in its own right, as demonstrated by the Danish Synthetic Party, the first
political party to be “led” by an AI (Hearing 2022).
More generally, personalized LLM training datasets can be tailored and curated for specific purposes.
One can imagine personalizing a model based on the Pope’s circulars to achieve a (religiously) virtuous
LLM. Other training sets might be curated to help minimize gender or racial bias in outputs. Such fine-
tuned LLMs could then at least tacitly improve the moral quality of the writing of their users.
Use of such models could be layered, such that each step of a workflow is carried out by a different,
specialized LLM. For example, an in-house researcher at a company might generate a first draft based on
a personalized LLM. The first draft could then be fed into another LLM fine-tuned on company
documents to help increase its fit with corporate branding and messaging priorities. The output could then
be further refined by LLMs specialized for de-biasing of language, for increasing the clarity of the
writing, for evaluating its alignment with specific values or causes, or for any other specific purpose.
Ethical concerns
LLMs in general raise questions around the instrumentalization and alienation of humans from work and
creative endeavor. They open the door to cheating and plagiarism. We gesture to a response to these
concerns in the abstract of a potential future paper on co-creation included in Figure 3 above. In brief,
when human beings evaluate and transform the results of LLMs, particularly building on their analytic
strength or originality, the use of LLMs is not merely permissible but an attractive and morally desirable
example of human-technological co-creation. We plan to develop this idea in a subsequent publication.

17
The creation and use of personalized LLMs involve several additional issues of ethical concern.
Prominent among these is the potential for personalized LLMs to contribute to inequalities via the so-
called Matthew effect. Named after a passage in the Gospel of Matthew (25:29), the Matthew effect refers
to the tendency of advantages such as social or financial capital to accrue faster to those who already
enjoy much of them than to those who do not: “For unto every one that hath shall be given, and he shall
have abundance: but from him that hath not shall be taken away even that which he hath.” Since the
performance of fine-tuned LLMs is positively correlated with the volume of data available to train them,
the Matthew effect means that proficient and prolific writers, who already enjoy the benefits of an
established publication record, can benefit disproportionately from personalized LLMs. This is clearly
demonstrated in the difference in performance between our models trained on few (SPM) versus many
(BDE, JS, CO) previous publications (see Supplementary Information). The extent to which this can be
addressed by creating synthetic text based on existing training data or other means remain open questions.
Another ethical concern is related to the potential homogenizing effect of LLMs on writing styles. At the
individual level, reliance on personalized LLMs might prevent or inhibit the further evolution of writing
styles. At the collective level, models adapting newly generated or formatted outputs to a specific style
may inadvertently reduce diversity and potentially diminish the authenticity of LLM-produced texts as
well as the enjoyment gained from reading them (Dwivedi et al. 2023).
Issues surrounding consent and the use of publications as training data are particularly salient for fine-
tuned models. While the use of writings of which one is the sole author seems to us to be ethically
unproblematic, this is not necessarily the case for co-authored publications (see discussion on this point
below). Even more suspect is the use of writings to which one has not contributed at all. Using an
individual's writing style or data without permission might violate privacy and intellectual property rights.
However, issues of authenticity and the assignment of credit might remain contentious even in cases
where consent has been obtained, but those using the resulting fine-tuned models have not themselves
contributed to its training data.
Relatedly, misusing LLMs for various purposes, particularly style theft, poses significant ethical risks.
Given the significant market opportunity for assisting individuals or companies with fine tuning models,
websites such as chatbase.co and reword.com are already making it possible to easily personalize LLMs
based on someone’s writings at cheap or no cost.9
A malicious actor could therefore easily train and
9
It should be noted that these services offer a different type of personalization than that described here. So far as
we are aware, there are not yet any services offering fine-tuning on previous academic output in the way we have
here described.

18
exploit a personalized LLMs without consent for plagiarism or fraud, compromising academic integrity or
professional reputation.
Given that AUTOGEN appears to perform better on topics similar to those on which it was trained, there
is a risk that access to AUTOGEN-type models would perversely incentivize scholars to focus on work
that is only marginally different to what they have written before, while still crossing some minimal
threshold of publishability. This may be analogous to so-called ‘salami slicing’ – the tendency to take a
set of studies or ideas and slice them into as many tiny, incrementally advancing papers as one can, just to
get more publications, when really, they should all be in the same paper. Similarly, the higher
performance on familiar topics of AUTOGEN might incentivize scholars to produce ever more papers,
each one only marginally different from the last, rather than to make bigger, more substantive leaps of
scholarly or intellectual progress. Although this is likely to be a genuine problem, especially given the
already unmanageable volume of papers published in many disciplines each year, there may be practical
ways around this. For example, it might become an academic norm or even a policy requirement that one
can use AUTOGEN to generate introductions, conclusions, or background sections in which a more or
less rote synthesis of existing ideas and scholarship is necessary, while still needing to manually develop
the bulk of the substantively new material.
Open ethical questions
LLM use by humans, like human use of generative AI more generally, entails a moral credit-blame
asymmetry (Porsdam Mann, Earp, et al. 2023). On the one hand, blameworthiness for harmful content, or
other bad outcomes caused by human use of LLMs, arguably only requires recklessness or negligence (a
low bar). On the other hand, and by contrast, creditworthiness for good or valuable outcomes in such a
context requires that one invest substantial effort, talent, or sacrifice—specifically of a kind that may be
directly undermined or diminished by the use of LLMs (a high bar). Thus, individuals may be
blameworthy for low-effort, unskilled, yet careless or reckless use of LLMs to generate bad or harmful
content, but cannot similarly take credit for good or valuable content generated through a similar (low)
investment of effort or skill.
Personalized LLMs like AUTOGEN complicate this issue. Insofar as such models facilitate the
production of valuable writing, their use may diminish the effort and skill put into the writing process and
thus, arguably, creditworthiness. However, unlike more general, non-personalized models, personalized
models trained on one’s own prior writing represent the outcome of previous applications of labor, skill
and sacrifice. Perhaps, then, personalized LLM use occupies a middle ground with respect to

19
praiseworthiness: between the generation of text by non-personalized models (least credit) to entirely
human-generated writing of text (most credit). Insofar as we care about human moral creditworthiness,
that may favor training and using personalized, over non-personalized, models for text generation.
This position seems plausible to us in the case of LLMs trained by an individual using only their own past
writings. However, when the LLM is trained on a mixed dataset with co-authored texts or even other
authors' works, more complicated questions of ownership and attribution arise. One set of issues concern
the proportion of text written by a specific author of a multi-author paper. The approach we took in
training AUTOGEN was to use as training data only those publications on which SPM, BDE, or JS are
first authors, as in the bioethical context first authorship is typically a reasonable proxy for main drafting
responsibilities. This assumption may not hold for other fields, such as law, where co-author rankings
may be determined by seniority or alphabetical order. As the proportion of published text attributable to
the author seeking its use for LLM customization decreases, the moral importance of obtaining the
consent of co-authors increases.
In cases where an individual is using a model not owned by them, the output's ownership becomes even
more challenging to determine. Licensing agreements can play a critical role in clarifying rights and
responsibilities associated with using LLMs, specifying who should be credited and under what
circumstances. However, the extent to which certain licensing arrangements should be considered
legitimate for authorship ascription remains an open question that should be discussed as early as
possible. For example, roughly half of the training data used in the combined AUTOGEN model derives
from JS’s writings; the large majority of the rest derives from BDE’s writings; and only a small minority
from SPM’s writings. Should SPM be allowed to license AUTOGEN-CO? If not, what percentage of
contribution should serve as a cut-off value? If yes, what about someone who has contributed nothing to
the training data?
Further pressing questions concern whether data contributors should be listed as co-author. This might
depend on the level of their contribution to the model, and their role in the creation of the output. If
someone’s work significantly impacts the final output, co-authorship might be justified. However, listing
someone as a co-author — or even as text contributor — who has not personally written any material in
the text seems problematic. Doing so without their consent could lead to further ethical concerns.
Additionally, the risk of plagiarizing arises when the LLM generates content that closely resembles the
original works used in training. Ensuring that the output is sufficiently distinct from the source material is
crucial to maintaining ethical standards in academic writing.

20
Open technical questions
There are also numerous open questions relating to more technical matters. In presenting our findings, we
have only shown unedited AUTOGEN completions using the fixed input format described in the methods
section above. However, other methods of using AUTOGEN are possible and we are exploring these in
ongoing work. For example, completions can be made to continue manually written text by including
leading sentences in the prompt. By writing out the first or more sentences in a given section, and
including these in the prompt, AUTOGEN can be made to continue the line of thought expressed in those
sentences. Similarly, AUTOGEN outputs can be edited and fed back into the prompt. For example, if
AUTOGEN provides a five-paragraph completion but only the first paragraph is on-topic or otherwise
interesting, the last four paragraphs can be deleted and AUTOGEN can be made to generate new
completions continuing from the first paragraph.
Furthermore, and very importantly, AUTOGEN can be used as a discrete step in a wider workflow. As
mentioned, AUTOGEN is based on the GPT-3 model, which shows much inferior apparent reasoning
abilities than the successor GPT-4 model (Bubeck et al. 2023). However, AUTOGEN and GPT-4 can be
easily combined. The rough material generated by AUTOGEN could be fed into GPT-4 for improvement.
For example, we have had promising early results using prompts such as “Fix the format of the following
text and correct errors of reasoning and structure. Based on the following text and abstract, write an
introduction to a scholarly paper on [x]." GPT-4 can also be used to rewrite AUTOGEN completions
according to specific instructions, such as “Rewrite the above. Get rid of the second objection and
combine the third and fourth into one.”
The potential benefits of integrating both AUTOGEN and GPT-4 into a workflow have recently been
expanded by the release of plugins and browsing for GPT-4, and the ongoing release of the Code
Interpreter. This allows GPT-4 to access and process information from the internet (browsing), third-party
applications (‘plug-ins’), and manually uploaded documents in a variety of formats (Code Interpreter).
Other developments, such as AutoGPT, expand these capabilities even further by allowing individuals to
assign multiple goals and tasks to LLMs which then attempt to carry these out without further instruction.
Thus, the completions we present above could likely be substantially and easily improved using other
LLMs such as GPT-4.
Conclusion

21
We introduce the use of a fine-tuned, personalized LLM for academic writing, which we dubbed
AUTOGEN. Though our experiments are preliminary and informal, we believe they are sufficient to
demonstrate the potential benefits of LLM fine-tuning on scholars’ own prior publications, especially for
the generation of text and relevant ideas close to the subject matter of included writings.
The potential benefits, such as increasing academic efficiency and preserving writing styles, must be
weighed against serious concerns including intellectual property rights, privacy, and the potential for
misuse. Attempts to regulate personalized LLMs at both societal and institutional levels should address
how to mitigate these concerns whilst pursuing benefits associated with the opportunities. Further, the
ethical complexities surrounding authorship and credit attribution, particularly for multi-source trained
models, present novel challenges that remain to be worked out.
References
Bakker, Michiel A., Martin J. Chadwick, Hannah R. Sheahan, Michael Henry Tessler, Lucy
Campbell-Gillingham, Jan Balaguer, Nat McAleese, et al. 2022. ‘Fine-Tuning Language
Models to Find Agreement among Humans with Diverse Preferences’. arXiv.
https://doi.org/10.48550/arXiv.2211.15006.
Bommasani, Rishi, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von
Arx, Michael S. Bernstein, et al. 2022. ‘On the Opportunities and Risks of Foundation
Models’. arXiv. https://doi.org/10.48550/arXiv.2108.07258.
Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, et al. 2020. ‘Language Models Are Few-Shot Learners’.
In Advances in Neural Information Processing Systems, 33:1877–1901. Curran
Associates, Inc.
https://proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac
142f64a-Abstract.html.
Brynjolfsson, Erik, Danielle Li, and Lindsey R. Raymond. 2023. ‘Generative AI at Work’.
Working Paper. Working Paper Series. National Bureau of Economic Research.
https://doi.org/10.3386/w31161.
Bubeck, Sébastien, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece
Kamar, Peter Lee, et al. 2023. ‘Sparks of Artificial General Intelligence: Early
Experiments with GPT-4’. arXiv. https://doi.org/10.48550/arXiv.2303.12712.

22
Church, Kenneth Ward, Zeyu Chen, and Yanjun Ma. 2021. ‘Emerging Trends: A Gentle
Introduction to Fine-Tuning’. Natural Language Engineering 27 (6): 763–78.
https://doi.org/10.1017/S1351324921000322.
Demaree-Cotton, Joanna, Brian D. Earp, and Julian Savulescu. 2022. ‘How to Use AI Ethically
for Ethical Decision-Making’. The American Journal of Bioethics: AJOB 22 (7): 1–3.
https://doi.org/10.1080/15265161.2022.2075968.
Dwivedi, Yogesh K., Nir Kshetri, Laurie Hughes, Emma Louise Slade, Anand Jeyaraj, Arpan
Kumar Kar, Abdullah M. Baabdullah, et al. 2023. ‘“So What If ChatGPT Wrote It?”
Multidisciplinary Perspectives on Opportunities, Challenges and Implications of
Generative Conversational AI for Research, Practice and Policy’. International Journal
of Information Management 71 (August): 102642.
https://doi.org/10.1016/j.ijinfomgt.2023.102642.
Elkins, Katherine, and Jon Chun. 2020. ‘Can GPT-3 Pass a Writer’s Turing Test? | Published In’.
Journal of Cultural Analytics 5 (2).
Giubilini, Alberto, and Julian Savulescu. 2018. ‘The Artificial Moral Advisor. The ?Ideal
Observer? Meets Artificial Intelligence’. Philosophy and Technology 31 (2): 169–88.
https://doi.org/10.1007/s13347-017-0285-z.
Grimaldi, Gianluca, and Bruno Ehrler. 2023. ‘AI et al.: Machines Are About to Change
Scientific Publishing Forever’. ACS Energy Letters 8 (1): 878–80.
https://doi.org/10.1021/acsenergylett.2c02828.
Hearing, Alice. 2022. ‘Discord Users Are Grilling an A.I. Chat Bot Running for a Danish
Parliamentary Seat’. Fortune, 2022. https://fortune.com/2022/10/14/ai-chatbot-leader-
lars-the-synthetic-party-discord-russia-ukraine-crimea-policy/.
Liang, Percy, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga,
Yian Zhang, et al. 2022. ‘Holistic Evaluation of Language Models’. arXiv.
Lin, Zhicheng. 2023. ‘Why and How to Embrace AI Such as ChatGPT in Your Academic Lif’.
https://doi.org/10.31234/osf.io/sdx3j.
Moradi, Milad, Kathrin Blagec, Florian Haberl, and Matthias Samwald. 2022. ‘GPT-3 Models
Are Poor Few-Shot Learners in the Biomedical Domain’. arXiv.

23
Noy, Shakked, and Whitney Zhang. 2023. ‘Experimental Evidence on the Productivity Effects of
Generative Artificial Intelligence’. SSRN Scholarly Paper. Rochester, NY.
https://doi.org/10.2139/ssrn.4375283.
Porsdam Mann, Sebastian & Schmid, Maximillian M. 2018. Health Research Priority Setting:
State Obligations and the Human Right to Science, The American Journal of
Bioethics, 18:11, 33-35, DOI: 10.1080/15265161.2018.1523492
Porsdam, Helle, and Sebastian Porsdam Mann, eds. 2021. The Right to Science: Then and Now.
Cambridge University Press.
Porsdam Mann, S., Y. Donders, C. Mitchell, V. J. Bradley, M. F. Chou, M. Mann, G. Church,
and H. Porsdam. 2018. ‘Advocating for Science Progress as a Human Right’.
Proceedings of the National Academy of Sciences 115 (43): 10820–23.
https://doi.org/10.1073/pnas.1816320115.
Porsdam Mann, Sebastian, Yvonne Donders, and Helle Porsdam. 2021. ‘The Right to Science in
Practice: A Proposed Test in Four Stages’. In The Right to Science: Then and Now, edited
by Helle Porsdam and Sebastian Porsdam Mann, 231–45. Cambridge: Cambridge
University Press. https://doi.org/10.1017/9781108776301.005.
Porsdam Mann, Sebastian, Brian D. Earp, Sven Nyholm, John Danaher, Nikolaj Møller, Hilary
Bowman-Smart, Joshua Hatherley, et al. 2023. ‘Generative AI Entails a Credit–Blame
Asymmetry’. Nature Machine Intelligence, May, 1–4. https://doi.org/10.1038/s42256-
023-00653-1.
Porsdam Mann, Sebastian, Helle Porsdam, and Yvonne Donders. 2020. ‘“Sleeping Beauty”: The
Right to Science as a Global Ethical Discourse’. Human Rights Quarterly 42 (2): 332–56.
Porsdam Mann, Sebastian, Julian Savulescu, Philippe Ravaud, and Mehdi Benchoufi. 2021.
‘Blockchain, Consent and Prosent for Medical Research’. Journal of Medical Ethics 47
(4): 244–50. https://doi.org/10.1136/medethics-2019-105963.
Rid, Annette, and David Wendler. 2014. ‘Use of a Patient Preference Predictor to Help Make
Medical Decisions for Incapacitated Patients’. The Journal of Medicine and Philosophy:
A Forum for Bioethics and Philosophy of Medicine 39 (2): 104–29.
https://doi.org/10.1093/jmp/jhu001.
Savulescu, Julian, and Brian D. Earp. 2014. ‘Neuroreductionism about Sex and Love’. Think 13
(38): 7–12. https://doi.org/10.1017/S1477175614000128.

24
Savulescu, Julian, and Hannah Maslen. 2015. ‘Moral Enhancement and Artificial Intelligence:
Moral AI?’ In Beyond Artificial Intelligence: The Disappearing Human-Machine Divide,
edited by Jan Romportl, Eva Zackova, and Jozef Kelemen, 79–95. Topics in Intelligent
Engineering and Informatics. Cham: Springer International Publishing.
https://doi.org/10.1007/978-3-319-09668-1_6.
Schwitzgebel, Eric, David Schwitzgebel, and Anna Strasser. 2023. ‘Creating a Large Language
Model of a Philosopher’. arXiv. https://doi.org/10.48550/arXiv.2302.01339.
Shao, Qinglong. 2022. ‘Does Less Working Time Improve Life Satisfaction? Evidence from
European Social Survey’. Health Economics Review 12 (1): 50.
https://doi.org/10.1186/s13561-022-00396-6.
Zohny H, McMillan J, King M. Ethics of generative AI. J Med Ethics. 2023 Feb;49(2):79-80.

AUTOGEN A Personalized Large Language Model For Academic Enhancement Ethics And Proof Of Principle

More Related Content

What's hot (20)

Similar to AUTOGEN A Personalized Large Language Model For Academic Enhancement Ethics And Proof Of Principle (20)

More from Daniel Wachtel (20)

Recently uploaded (20)

AUTOGEN A Personalized Large Language Model For Academic Enhancement Ethics And Proof Of Principle