The Incentives Driving AI

The almost overwhelming pace of GenAI model deployments and releases makes keeping track of the permutations, developers, datasets, and related artifacts impossible to keep up with in real time. This post explores why AI entities likely feel the need to contribute to the flurry.

The GenAI Ecosystem

We will mostly focus on the primary developers of GenAI, like Meta, OpenAI, Microsoft, and Google. But there is a major caveat: these big players are part of a huge technology ecosystem and each aspect of the ecosystem is vitally important to GenAI. Here’s one chart that helps indicate the primary roles of the major participants.

In addition to a complex ecosystem, it’s also important to note that AI entities are not monolithic. Within these entities, there can be individuals or teams genuinely motivated by ethical concerns.

Incentives

With those caveats out of the way, let’s shift the focus back to the main GenAI developers and why they might be racing to create the most capable models as quickly as possible, and why a more cautious or ethical approach is not usually favored. 

1: Shareholder Primacy

It may be that companies want to prioritize ethics higher, treat contractors better, be more cautious, and so on, but the system of capitalism in the US, and especially venture capital, prevents or limits that approach because the AI entities need to make a return on investment and show continuous and significant growth. Under this framework, the goal of generating revenue takes precedence over being helpful and harmless. That’s not to say that ethics and safety aren’t also important, but those areas do receive a lower priority in all meaningful ways (computational allotment, manpower, R&D) compared to just increasing a model’s capabilities. 

A company’s obligations under VC and shareholder primacy systems are to its funders, not to society. For their part, VCs invest in multiple companies knowing only a couple will be successful enough to justify the failures of others,[1] but they don't know which ones will be successful, so they need all of them to press as fast as possible to take market share.

But this phenomenon isn’t limited to VC funding. With shareholder primacy, even established companies that have been on the stock market for decades face a similar pressure. Microsoft’s CEO said he wanted to make Google dance [2] by Microsoft frantically rushing AI into various offerings.[3] Shareholders reacted favorably, meaning Microsoft was able to internalize profits while externalizing problems.[4]

Meanwhile, as is generally the case in a shareholder-driven economic system, other parties in the field were forced to react, whether they thought it was a good idea, or safe, or ready, or not. ChatGPT led to Google’s Code Red moment, and just a couple of months later, Meta announced Llama, whose weights were leaked to the public. A month later, Google revealed Bard and Microsoft revealed Bing chat. Both announcements were accompanied by demonstrations that included hallucinations from their models. Four months after that, Meta released Llama 2 to anyone who submitted a form, including a person who claimed to be “Terrorist” from “Terrorist Incorporated.”

Companies are under pressure to show some sort of progress every quarter, whether it’s real/significant/meaningful or not. This progress is typically presented in the form of some key metrics, such as revenue growth, increased usage (i.e., daily active users), increased engagement by users (e.g., the number of posts or prompts or clicks by users), or some other proxy of what the financial markets would consider “success.” If a competitor does something that gives them an advantage, other companies are incentivized to follow suit. It’s a one-way ratchet where metrics, which inherently exclude things that can’t be easily measured (happiness, justice, liberty, fairness), reduce “success” to proxies.

2: Financial Incentives

Why might an organization want to charge ahead with AI developments? One reason could be financial.

Of course, the value of any entity is based on the presumptive ability of the GenAI companies to one day generate a profit. Notably, the many well-known AI companies appear to be significantly overvalued at the moment.

The New York Times notes that “Inflection AI, which raised $1.5 billion but made almost no money, has folded its original business. Stability AI has laid off employees and parted ways with its chief executive. And Anthropic has raced to close the roughly $1.8 billion gap between its modest sales and enormous expenses.” Anthropic has raised more than $7 billion in funding but is spending $2 billion a year while only generating “about $150 million to $200 million in revenue.” StabilityAI is also struggling, arriving at a $36 million deficit last year. Cohere is doing no better, with The Information reporting that it raised $445 million but only generated $13 million in revenue (not profit). The Times also notes that Meta “said it planned to raise its spending forecast for the year to $35 billion to $40 billion, up from a previous estimate of $30 billion to $37 billion.” So, though “OpenAI pulled in around $1.6 billion in revenue over the last year,” and it’s probably generating more revenue than any other GenAI service, it’s likely nowhere near turning a profit.

How can a profit maximizing enterprise cut expenses without reducing progress? Some observers have noted that these companies tend to rely on some unscrupulous methods:

Many have compared GenAI to social media, but one possible bright spot is that GenAI could have a better financial incentive structure than social media.[5] 

GenAI doesn’t need to promote or maximize engagement or time spent on the model or number of queries. Currently, GenAI does not generate revenue from advertisements, so it doesn’t have the perverse incentives of, say, some social media platforms, to spark outrage in order to keep people logged in. GenAI just needs people to pay a subscription fee. In fact, the more efficient and helpful the GenAI can be in providing accurate responses, the less it may be used, which saves the GenAI entity money on computation, which saves it more money out of every subscription.

This may be why some AI entities program their models to produce diverse outputs when creating images, for example. Rather than spend precious computational resources on clarifying what the user wants or having the user make several corrections to the output by presuming a particular output the user will want, the models instead produce a diverse output, casting a wide net, hoping that at least one of the outputs will be the one the user is looking for on the first generation. In short, its goal may be to be so good it doesn’t need to be used constantly. 

3: First Mover Incentives

Whereas companies like Google had been hesitant to rush AI products into the market for years because they have an established brand they must protect, smaller and newer AI entities don’t have that restraint. In fact, the incentives are quite the opposite.

The Atlantic touched on this in a profile of AI-audio generator ElevenLabs:

The core problem of ElevenLabs—and the generative-AI revolution writ large—is that there is no way for this technology to exist and not be misused. Meta and OpenAI have built synthetic voice tools, too, but have so far declined to make them broadly available. Their rationale: They aren’t yet sure how to unleash their products responsibly. As a start-up, though, ElevenLabs doesn’t have the luxury of time. “The time that we have to get ahead of the big players is short,” Staniszewski said. “If we don’t do it in the next two to three years, it’s going to be very hard to compete.” 

These same incentives likely played a role in OpenAI pushing ChatGPT into the market as quickly as it could. And when the public embraced ChatGPT, it encouraged Microsoft to adopt a move-fast mindset, even if it wasn’t able to conduct all the legal and safety checks it had suggested it would in all its AI safety and responsible AI documentation. Or, perhaps they did conduct the checks and decided the risk was worthwhile because the potential market reward was so high (a bet that has paid off so far in as much as share price is more important than societal wellbeing).

4: Data Incentives

Consider the following: If you write a great book, more people will read it. If you make great bicycles, more people will ride them. If you make a great chatbot, not only do you get more users and more usage, like good books and bikes, but you also get to harvest all the data such interactions provide. The types of data could be rich and diverse, including overall usage, duration of sessions, time of sessions, location of uses, types of questions asked, whether you thumbs up or down, whether you copy the text, how your voice intonation reacts to the bot (if using audio features), and so on. This allows AI entities to turn users into both a source of revenue and a source of raw material to harvest, which will allow them to improve the products and possibly take actions like target ads. 

The trend toward data collection is also why many entities want to rush out products, even when half-baked and lacking all the demoed features. They do so in part to get users to help them improve the products through usage and explicit feedback. Users, in a way, become unpaid employees, using a mediocre product and making it better, but sharing much less of the upside: no individual user will make billions off a 2% improvement in the model, but the entity could. This approach isn't new for AI, but it is different from other software and hardware. Social media, for instance, doesn't hype up new features and then not release them the way OpenAI has promised highly competent AI math tutors and flirty AI voice assistants while not releasing them. Hardware makers, such as Dell and Apple, which make physical products like phones and laptops, typically don't demonstrate features like better processing chips and cameras but then not include them when they make the products available to consumers. The only consistent exception is when a feature is AI-powered.

The need for data is acute. It’s also why AI entities have been more willing to roll the dice on whether training on copyrighted images is fair use, training on data known to be acquired without authorization, limiting the ability of users to have their data removed from datasets and models, and ignoring robots.txt. All these issues will be discussed in greater depth later.

Of course, better results could also just lead to greater use and dependence by users, and GenAI providers could encourage such use, charging power users more for a technology they may deem to be indispensable.

This latter approach seems to be where OpenAI is headed. Rather than try to provide a product that succinctly outputs the most useful information, it has leaned into anthropomorphizing its products, making them more engaging for prolonged interactions. We’ll explore anthropomorphism in greater depth later, but for incentive purposes, anthropomorphism helps elicit more data from users. 

In addition, anthropomorphizing can lead to humans developing strong bonds with the chatbot, which could lead to customer loyalty and a willingness to pay for premium subscriptions in order to have greater access. In a way, this is what happens with humans and pets. We often ascribe personalities and thoughts to animals, forming bonds that will lead us to spend money on birthday cakes for cats and puppuccinos for dogs. A key difference is that while the pets may genuinely care about the owner, the chatbots have no thoughts or feelings about the humans they talk to and are just as happy to never interact again as they are to interact all day long.

5: Safety Incentives

Some companies, notably OpenAI and Anthropic (founded by former top researchers at OpenAI), express a deep interest in AI safety, which means trying to ensure that the most advanced AI models will be aligned with human values and will not be harmful to society. Their thesis is that in order to ensure this type of safety, they must create the world’s most advanced models so they can experiment with them. They fear that others who may not share the same values of not harming society might otherwise outpace their development, which could have huge, adverse outcomes on society.[6] 

6: Relevance Incentives

AI entities may simply not want to be left in the dust. With the release of ChatGPT in November of 2022, it seems that OpenAI wanted to have a first-mover advantage. As of August 2023, ChatGPT has had more visitors than all other AI websites combined—by a lot. 

7: Prestige Incentives

While no major AI company lists prestige or glory among the top reasons for working on GenAI, it seems plausible, and even likely, that being the first to accomplish some impressive feat is a primary driver behind some actions. This may be especially true at the individual level, where being a lead researcher of a notable project often translates to higher salaries, more equity, speaking invitations for conferences and symposiums, and even publicity in popular media. Even if they don’t reach the national consciousness, they become famous within the smaller circles of tech wizards, venture capitalists, and fellow researchers. 

Pressing the frontier of AI development also incentivizes using more and more compute, offsetting the stated carbon emission reduction goals of large AI entities and discouraging any other AI entity from committing to any carbon goal. 

But the incentive to increase compute in order to improve model performance isn’t merely an environmental problem. It’s quickly becoming a financial one. The marginal improvement of model performance relative to the computational costs is stark. From the 2024 AI Index report out of Stanford: “[t]he training costs of state-of-the-art AI models have reached unprecedented levels. For example, OpenAI’s GPT-4 used an estimated $78 million worth of compute to train, while Google’s Gemini Ultra cost $191 million for compute.” It’s noteworthy that Gemini Ultra only barely bested GPT-4 on common benchmarks, and that despite coming out nine months later.

8: FOMO Incentives

A final reason may be FOMO, or fear of missing out. Companies and research institutions may fear that if they don’t participate in GenAI development in some way they could become irrelevant to their shareholders, the media, their benefactors, and others. Rather than take that risk, they may allocate resources to GenAI projects and hope to figure out a use case, business plan, and/or product-market fit later. They may feel pressure to partake in the hottest area of tech in years and it may mean they forgo asking fundamental questions about why they’re doing so and whether GenAI is the best tool to accomplish the goal.

9: Manifest Destiny Incentives

Convincing people AI will make them replaceable at their jobs or in society unless they learn how to leverage AI works in the AI entities’ favor. The AI entities need people to want to use AI in education and on the job even when the outputs or outcomes aren't as good as not relying on AI. The more that people feel they must adapt or die, the better. 

When “thought leaders” say that children need to be introduced to AI now and AI needs to be woven into their lives in myriad ways and that learning to prompt GenAI is an essential skill for success in everyone’s future the “thought leaders” aren’t basing their claims on any scientific research or even powerful logic. There is currently no basis to believe mastering prompt engineering as an eight-year-old will make that kid any more successful than the child who doesn’t use AI at all. In fact, the child who doesn’t use AI may end up better off because they may learn to think more deeply and critically, rather than rely on quick answers from AI (some of which will almost certainly be hallucinations). 

Another aspect of the manifest destiny approach is that convincing society that some technological development is inevitable is a way of saying society should not resist what the technologists are doing. Instead, they should welcome it and remove constraints and restraints that may impede its development and deployment. 

Finally, a manifest destiny approach could be a way to argue people are at fault if they don’t help the AI. If the AI isn’t performing as well as one hopes, the solution may be to allow the AI entity to train on more high-quality data (for free). Or, if the AI doesn’t seem safe or safe enough, the solution is for people to use it more and more often (sometimes for free, but preferably with a subscription).  

Regulatory Entrepreneurship

Some see a grander strategy in entities pressing GenAI into every corner.[7] In scholarly circles the approach is sometimes called “‘regulatory entrepreneurship’ — pursuing a line of business in which changing the law is a significant part of the business plan.” It could also be called the “everything, everywhere, all at once” strategy. Or a business blitzkrieg, if you will.

Uber was famous for this tactic. It forced itself into cities whether the local governments wanted it or not. The goal was to get as much consumer adoption as quickly as possible so that it became embedded in the way of life for many thousands of people. Then, when the government was finally able to react after going through all the necessary regulatory processes, Uber would be too difficult to eradicate. By then, people depend on it to get to work, or school, or anywhere else. Meanwhile, Uber drivers come to depend on Uber for income and may have already invested in a new car to continue working. So, even if Uber may be worse for the city over all (less investment in public transportation, more traffic, more carbon emissions), the business strategy is successful. 

In a similar vein, OpenAI and others have pressed onward over the objections of many government officials at the local, state, and national levels. By encouraging mass adoption and implementation into the daily lives of users, including commercial uses, schools, and government itself, GenAI companies may be hoping, at least in part, that they will be too difficult to remove as a technical, political, and practical matter. 

All this is despite the many unsettled legal issues around topics such as copyright and privacy. It becomes harder by the day for a government or judge to issue an order that GenAI be removed from systems or that the models and datasets should be deleted. Instead, it becomes more likely that any penalty will be in the form of a fine that is tuned to be painful–but not deadly–to the GenAI entities. In other words, in both the case for Uber and for GenAI, the businesses consider litigation the price of doing business. A fine, in a way, is another way of saying something is legal for a price. This is particularly true when fines are small relative to a company’s revenue.[8]

To be sure, the strategy may be wise. So far, the government has proven to be, at best, sluggish in the face of emergent technologies, and at worst utterly impotent. The high-risk, high-reward nature of regulatory entrepreneurship may just be high-reward. But even if there is high risk, it may be riskier still for GenAI entities to not take action and possibly miss out or fall behind others. So, the race is on.

Adding Ads

As we write this, ChatGPT is free to use and has been free for a year and a half. But the history of Silicon Valley suggests that’s likely to change. Platforms often become greedy or desperate and it leads to what Cory Doctorow calls “enshitification.” The generic cycle looks like this:

1. New platforms offer useful products and services at a loss, as a way to gain new users. 

2. Once users are locked in, the platform then offers access to the user base to suppliers at a loss.

3. Once suppliers are locked-in, the platform shifts surpluses to shareholders

Here’s how Doctorow applied it to Amazon:

1. Amazon started selling goods below cost to build up a user base. 

2. Amazon introduced the Amazon Prime subscription which encouraged users to shop exclusively at Amazon. 

3. The strong base of clients who had formed a habit of using Amazon exclusively incentivized more sellers to sell their products through Amazon, as Prime users were only searching Amazon for goods. 

4. Amazon began to focus on its shareholders by increasing profits and introducing fees. 

A similar story played out for journalistic sites and social media:

1. Users were locked in by providing a good service (Facebook had no ads, initially, for example; people saw content they wanted to see rather than content that generated the most revenue for Facebook)

2. Social media platforms offered a cheap way for the sites to reach readers, luring in traditional media

3. The platforms began charging more for less effective ad placement while feeding users access to summaries and content in a way that negated the need to visit the traditional media’s pages or subscribe while requiring traditional media to pay more for ad placement

How might this theory apply to ChatGPT and other currently free chatbots? It may look something like this:

1. The AI entity provides good, lengthy responses for free and without advertisements. This pulls in millions of daily users who develop loyalty to the brand. This also starves competitors of customers and fresh input data who can’t afford to subsidize users.

2. The AI entity adds ads at a subsidized rate to lure in advertisers, providing high-quality placements based on user prompts (e.g. “where’s the best place to get tacos in Austin?”). This pulls in top retailers and advertisers. 

3. The AI entity then reduces the quality of chatbot outputs (providing more generalized responses, shorter responses, slower responses, or some combination of the above) while increasing the cost to place ads. This allows the AI entity to spend less on computing costs while still scooping up the revenue.

4. To get a better chatbot experience and/or better advertising placement, the users and advertisers must pay for premium features. 

People may say that Meta and Google, by far the two biggest businesses based on advertising, are properly incentivized to have their models create great content and to sort the human content from the AI garbage because they need the product to be good for their customers to keep them coming back. This is the same argument they've made about search and social media for decades. 

But, and this is important to reiterate: the users are not the customer; they are the product. The customer is the entity that helps pay the bills: advertisers. Google and Meta foster user engagement so they can cultivate user manipulation. They aren't paid when you stare at your screen, so the end goal isn’t just getting user attention. They make their money when you click an ad. 

This may mean Google and Meta are disincentivized from admitting when their AI is bad or harmful because their high share prices rely on convincing people the AI is very good, and that will be a safe and effective place for advertisers to place their advertisements.

Market Failures

Finally, it’s worth giving a brief explanation of what causes markets to fail. A market failure just means that the market is inefficient. This can happen in four ways, presented below.  In general, if a market is efficient and there are no market failures, then there tend to be fewer ethical issues. But companies, including AI entities, are incentivized by the market to pursue market failures because they lend themselves to greater revenue, and revenue is a key driver of our economic system (see: Shareholder Primacy above).

1. Monopolies

2. Externalities

1. Exploitation of public goods

2. Asymmetric information

[1] This is called the Power Law. For example, a fund may invest in 10 companies knowing nine will fail, but the one that exceeds can do so spectacularly, more than offsetting the losses from the failed companies.

[2] “And I hope that, with our innovation, they will definitely want to come out and show that they can dance. And I want people to know that we made them dance, and I think that’ll be a great day.” Microsoft CEO Satya Nadella says he hopes Google is ready to compete when it comes to A.I. search: 'I want people to know that we made them dance'

[3] “Mr. Nadella said in an interview that Microsoft was working at a “frantic pace” to incorporate the technology into its products. By releasing a new search tool — what he called “the most used product on the planet” — people will see how their “everyday habit” could lead to “something magical.” A Tech Race Begins as Microsoft Adds A.I. to Its Search Engine

[4] Some harms may include misinformation, accelerating distrust in online content, generating violent images, copyright infringement, carbon emissions, and others

[5] However, some may argue social media may still be more ethical by some measures. For example, Meta and X seem to be upfront that their users are their product. GenAI companies are selling what some claim is a derivative product based on work by people who were not the GenAI entities’ users.

[6] Counteracting this idea, AI ethicists Tristan Harris and Aza Raskin propose that technology powerhouses like China in reality fear what they call GLLMMs (Generative Large Language Multi-Modal Models) because they have too little control over them and are not incentives to hurriedly create them. The A.I. Dilemma - March 9, 2023

[7] https://www.forbes.com/sites/bernardmarr/2023/03/06/microsofts-plan-to-infuse-ai-and-chatgpt-into-everything/?sh=1279813253fc. And don’t expect Microsoft to always admit when AI does something weird: Microsoft removed a set of bizarre travel articles made with 'algorithmic techniques.' But it won't blame AI.

[8] For example, in 2019 the FTC fined Facebook for $5 billion, or less than 10% of their annual revenue of $70 billion that year. In 2023 Facebook generated $134 billion in revenue.

[9] The following students from the University of Texas at Austin contributed to the editing and writing of the content of LEAI: Carter E. Moxley, Brian Villamar, Ananya Venkataramaiah, Parth Mehta, Lou Kahn, Vishal Rachpaudi, Chibudom Okereke, Isaac Lerma, Colton Clements, Catalina Mollai, Thaddeus Kvietok, Maria Carmona, Mikayla Francisco, Aaliyah Mcfarlin