The Impact of AI’s Rising Costs: What It Means for Innovation and Strategy

The trajectory of AI in the last few years has been wild. Back in 2017, training the most advanced AI models cost around $600,000, a huge sum, sure, but nothing compared to where we are today. Fast forward to 2023, and training GPT-4 set back OpenAI by a staggering $40 million. If the current trend holds, the biggest models could cost over $1 billion to train by 2027.  Research from Epoch AI's highlights just how steep this curve has become, showing how exponential growth in compute demands is translating directly into massive cost increases.

This raises some pretty big questions: how sustainable is this path, and what does it mean for the competition? If only a few companies like OpenAI, Google, and Microsoft can afford to play at this level, where does that leave everyone else, smaller players, startups, even mid-sized enterprises? Are they locked out of the game, or is there another way forward?

In a previous article, I explored how the "stagnation thesis", the idea that building ever-larger models delivers diminishing returns, has implications for enterprises adopting AI. But the financial barriers to training frontier models reveal a deeper issue: the power of constraints. Power, chips, data, and latency are not only driving up costs but also redefining the competitive landscape. These constraints favor well-funded organizations while potentially sidelining smaller companies and startups.

In this article, I look into the numbers and trends driving these skyrocketing costs and explore what they mean for innovation and competition. By analyzing the bottlenecks at the heart of AI scaling, using the insights from Epoch AI, we can try to make sense of the AI arms race and perhaps rethink our strategic perspective going forward.

Scalability Constraints: Understanding the Bottlenecks of Frontier AI Scaling

The exponential scaling of AI training has unlocked groundbreaking capabilities but has also exposed critical bottlenecks: power availability, chip manufacturing capacity, data scarcity, and the latency wall. The analysis by Epoch AI shows that these constraints define the limits of what can be achieved by 2030 and underpin the skyrocketing costs of training advanced models. Let’s explore what the research says.

Scalability Constraints: The Bottlenecks of Frontier AI

1. Power Constraints: Training frontier AI models demands vast energy resources, with AI-related uses already consuming a growing fraction of the 20 GW utilized by U.S. data centers. By 2030, single training runs may require up to 6 GW, comparable to a mid-sized power plant.While some companies may achieve 5 GW through advanced facilities, scaling further requires distributed training across multiple data centers, introducing challenges like inter-data center latency and bandwidth.Grid expansion and regulatory hurdles slow the pace of power supply growth, cementing energy as a critical bottleneck.

2. Chip Manufacturing Capacity: The availability of advanced GPUs is a cornerstone of AI scaling. Despite efforts to ramp up production, constraints in advanced packaging (e.g. CoWoS) and high-bandwidth memory (HBM) persist.Projections suggest 100 million H100-equivalent GPUs could be produced by 2030, enough for a single 9e29 FLOP training run. However, competition among labs and chip allocation for inference tasks limits resources for individual projects.These dynamics risk fostering monopolistic dynamics within the industry

3. Data Scarcity Large-scale models rely on enormous datasets, yet the estimated 500 trillion tokens of web text show the limits of natural data.Multimodal data (e.g., images, audio, and video) and synthetic data offer extensions, but these approaches face challenges. Synthetic data generation risks model collapse and carries significant computational overhead.Ensuring data quality and diversity further compounds the complexity of scaling.

4. The Latency Wall As models grow, training computations encounter unavoidable delays, creating a "speed limit" on efficiency.Current GPUs maintain low latencies, but distributed systems magnify communication delays, especially under existing protocols.Scaling batch sizes to mitigate latency hits diminishing returns beyond a critical point. Without innovations in network topologies or batch processing, the latency wall will likely cap feasible training sizes and durations by the late 2020s.

All in all, power and chip availability are the most immediate hurdles, with data scarcity and latency closely following. These constraints slow the pace of AI advancements but create opportunities for firms that prioritize efficiency and innovation.

Rising Costs: The Reflection of Constraints

The rising costs of training frontier AI models directly result from these scalability challenges. The research from Epoch AI shows some interesting data (Cottier et al., 2024). It found that since 2016, training costs for the most compute-intensive AI models have grown by approximately 2.4× per year. It also estimates that by 2027, the cost of training a single frontier model is projected to exceed $1 billion, concentrating large scale AI innovation among a few wealthy players and nations. Let’s review it in more detail.

Key Drivers of Rising Costs

1. Hardware Costs AI accelerator chips (e.g., GPUs, TPUs) account for 44% of amortized training costs, with upfront acquisition costs 10-100× higher. For instance:GPT-4 incurred $40 million in amortized costs but required $800 million in hardware acquisition.Organizations relying on third-party cloud platforms face even steeper costs compared to firms with proprietary infrastructure.

2. Energy Expenses Although energy costs represent only 2-9% of total training expenditures, the demands are escalating. Frontier models could require up to 6 GW of power by 2030, necessitating investments in grid infrastructure expansions.Companies like Google, Amazon, and Microsoft have already announced major investments in nuclear energy to address these challenges.

3. R&D Staff Costs Engineers and researchers, including equity compensation, make up 29-49% of development expenses. This highlights the financial intensity of cutting-edge AI development.

4. Data Acquisition Procuring and preparing massive datasets, whether from natural, multimodal, or synthetic sources, introduces another substantial cost.Licensing agreements, quality assurance, and engineering optimization further amplify expenses.

5. Latency Solutions Addressing latency challenges requires significant investments in network topologies and low-latency infrastructure. These efforts compound the already high costs of scaling.

In sum, the rapid escalation of AI model training costs consolidates power among the wealthiest players while raising tough questions about sustainability and equitable access. Navigating these challenges will take creative thinking and a real commitment to fostering a diverse and competitive AI landscape.

Dynamics of Rising AI Training Costs

Now we know that the rising costs of training advanced models can reshape the landscape of enterprise AI adoption. These escalating expenses influence pricing structures and, at the same time, can drive new strategies for mitigating costs and capturing value. Let’s break them down:

1. Pass-Through of Training Costs

• Direct Link: As training costs rise, AI providers are likely to pass on some of these expenses to customers to maintain margins. This is especially true for advanced or highly customized services, where development costs have a direct impact on pricing.

• Cost Structure: Pricing models for AI services typically incorporate training, operational, energy, and maintenance costs. A disproportionate increase in training costs will inevitably drive enterprise prices up unless offset by other factors.

2. Economies of Scale

• Cost Spreading: Large-scale players like OpenAI and Google can amortize high training costs over a broad customer base through subscription or usage-based pricing models. This diffusion reduces the per-customer cost burden.

• Efficiency Gains: Providers also invest heavily in innovations such as model compression, fine-tuning, and hardware/software optimizations to mitigate operational expenses, potentially keeping prices stable despite rising training costs.

3. Market Competition

• Price Sensitivity: Competitive pressures from smaller companies or open-source solutions incentivize providers to moderate prices, even as training costs climb.

• Market Differentiation: Providers may segment their offerings, passing costs onto premium customers seeking advanced solutions while targeting cost-sensitive customers with more affordable, lower-performance models.

4. Cost Divergence: Training vs Inference

• Focus on Inference: While training involves significant upfront investment, enterprise AI services are often inference-heavy. Optimized deployment models help control inference costs, insulating customers from steep price increases.

• Reuse of Pre-trained Models: Fine-tuning pretrained models for specific applications spreads training costs across multiple use cases, further reducing the financial impact on end-users.

5. Alternative Revenue Models

• Tiered Pricing: Providers already offer tiered pricing, adopting different levels of access, usage, or customization, enabling cost-sharing without uniformly raising prices. We could expect to see more tiered pricing.

• Collaborative Development: Partnerships between providers and enterprises to co-develop models can help stabilize costs while aligning capabilities with specific business needs.

6. Enterprise Strategies to Mitigate Costs

• Adoption of Efficient Solutions: Enterprises may shift to smaller, more efficient models (the most appropriate as opposed to the shiniest) or explore open-source alternatives if pricing becomes prohibitive.

• In-House AI Development: Some organizations are already investing in their own AI infrastructure and may see more following suit, tailoring models to specific requirements and reducing dependency on external providers.

In summary, while rising AI training costs may lead to higher enterprise expenses, the final impact depends on how effectively providers manage these increases through scale, competition, and efficiency. Enterprises, too, can adopt strategies to control costs, including leveraging efficient models or pursuing in-house solutions.

Bottom Line: For enterprises, the key is to stay flexible. Whether that means choosing efficient models, exploring open-source tools, or investing in in-house capabilities, there are ways to keep AI costs manageable while still benefiting from its potential.

Barriers and Opportunities in Innovating with Large AI Models

The rapid growth in AI capabilities has created a bifurcated landscape: on one side, large-scale model development dominated by tech giants, and on the other, smaller organizations leveraging niche opportunities to innovate. Here’s how this dynamic unfolds:

1. Barriers to Innovating with Large Models

• High Resource Requirements: Training state-of-the-art models like GPT-4 isn’t just expensive, it’s staggering. We’re talking millions of dollars spent on hardware, energy, and R&D. Only a few organizations, like Big Tech giants and government-backed labs, can afford to play at this level.Economic Barriers: The capital-intensive nature of large-scale model training discourages new entrants, creating a high barrier to entry.Technical Barriers: Proprietary data, massive compute clusters, and expertise in scaling further consolidate the advantage of established players.

• Growing Industry Consolidation: Rising costs are most likely accelerating the concentration of power among a few organizations, potentially leading to monopolistic or oligopolistic dynamics. This restricts competition and could stifle innovation in frontier AI development.

• Regulatory and Governance Challenges: Large-scale AI projects face increasing scrutiny over ethics, safety, and environmental sustainability. Regulatory hurdles can add further complexity, discouraging smaller or less-resourced players from entering the field.

2. Opportunities for Smaller Companies in Niche Markets

The good news? Small and midsized companies don’t need to train giant models from scratch to make a big impact. While large-scale model training may be out of reach, smaller companies can find significant opportunities by focusing on specialized applications.

• Fine-Tuning Pretrained Models:Adapting open-source models (e.g., BLOOM, LLaMA) or using APIs from larger providers allows smaller organizations to create domain-specific solutions at a fraction of the cost of training from scratch.Fine-tuning models with proprietary or niche datasets offers high-value outputs without the financial burden of full-scale training.

• Leveraging Proprietary Data:Enterprises with unique datasets can gain a competitive edge by fine-tuning models for specific needs, such as adapting a general AI for medical diagnostics or legal analysis.The cost of fine-tuning scales with data size but remains far cheaper than full-scale training.

• Domain-Specific Expertise:Smaller companies often have deep knowledge of their industry’s unique challenges. Leveraging AI to create tailored solutions, such as optimizing supply chains for a specific manufacturing sector, can have greater impact than generalized models.

• Cloud-Based AI Platforms:Services like AWS SageMaker, Azure AI, and Google Vertex AI lower infrastructure barriers, enabling smaller teams to deploy and build AI-powered solutions without heavy capital investment.

• Collaborative Ecosystems:Open-source platforms (e.g. Hugging Face, OpenMMLab) provide powerful tools and collaborative environments where smaller companies can build on existing R&D. It’s a great way to innovate without starting from scratch.

3. Comparative Market Dynamics

The interplay between smaller companies and large players creates distinct advantages and challenges for each group:

Smaller Companies:

• Advantages: Agility, specialization, and lower costs of innovation.

• Challenges: Dependence on external models and infrastructure, vulnerability to competition from larger players entering niche markets.

Large Players:

• Advantages: Deep pockets, access to top-tier talent, and control over foundational model development.

• Challenges: High costs, slower adaptation to niche markets, and regulatory pressures.

4. Economic and Technical Feasibility

• From an Economic Perspective:Rising costs create barriers to entry for training large models but also open avenues for smaller players in niche applications where costs are more manageable.Smaller companies can remain profitable by leveraging existing models and focusing on differentiation through proprietary data and expertise.

• Technical Perspective:Techniques like fine-tuning, transfer learning, and open-source frameworks democratize AI access, enabling resource-constrained organizations to innovate effectively.Advances in model efficiency, such as LoRA (low-rank adaptation), distillation, and deployment techniques like quantization and edge AI, are also making it easier to get more from less.

 Yes, training massive AI models is becoming an exclusive club, but that doesn’t mean the rest of the field is out of options. However, this challenge has led to a bifurcation in AI innovation. Large corporations dominate frontier AI development, while smaller firms thrive by leveraging niche expertise, proprietary data, and fine-tuning pretrained models.

This evolving dynamic highlights the growing importance of collaboration, efficiency, and specialization in ensuring a diverse and competitive AI landscape. By focusing on areas where they can excel, smaller organizations can continue to drive innovation and deliver high-impact solutions.

Confronting the Constraints

AI’s growth is hitting some serious bottlenecks. Power, chip manufacturing, data, and latency are no longer just technical issues, they’ve become the economic and competitive fault lines shaping the industry. These constraints drive up costs, concentrating power in the hands of a few companies that can afford to keep playing at the frontier. Smaller players, meanwhile, are left trying to carve out their own niches.

A Split in the AI Landscape

We’re witnessing a clear divide in the AI ecosystem:

• Big Tech: The Googles and OpenAIs of the world dominate the training of massive, state-of-the-art models. With their deep pockets and massive infrastructure, they’re pushing the boundaries of what AI can do.

• Smaller Players: While training models on this scale may be out of reach, smaller companies are finding ways to thrive. By fine-tuning existing models, leveraging unique datasets, and targeting specialized markets, they’re delivering impactful, tailored solutions without the need for massive upfront investments.

This divide reflects the growing importance of efficiency, focus, collaboration and the creation of strong ecosystems.

Opportunities in the Face of Challenges

The barriers to training large models are real, but they’re not insurmountable, and they even create opportunities:

• For Smaller Companies: By focusing on domain-specific problems and leveraging open-source models or cloud-based tools, smaller players can stay competitive without breaking the bank.

• For the Industry: Open-source ecosystems and new techniques for improving efficiency are making it easier for organizations of all sizes to innovate.

• For Collaboration: Partnerships between enterprises and AI providers are becoming more common, pooling resources to co-develop solutions that neither could achieve alone.

What It Means for Enterprises

With training costs skyrocketing, enterprises will see some of those costs passed along in the products and services they rely on. But it’s not all bad news. Competition, efficiency gains, and smarter pricing models can help offset the impact. Enterprises themselves can take steps too: adopting efficient models, exploring open-source tools, or investing in in-house AI capabilities.

A Path Forward

So, where does all this leave us? Here’s what needs to happen:

• For the Industry: We need to double down on efficiency and sustainability. Smarter power usage, better chip manufacturing processes, and innovations in training techniques will be critical to managing costs and environmental impact.

• For Smaller Players: Specialization and collaboration will be the keys to staying competitive. Focus on areas where you can add unique value.

• For Policymakers: Ensuring equitable access to AI resources is essential. Policies that encourage competition and diversity in AI development will benefit everyone.

Key Takeaway

AI’s trajectory is thrilling and wild, but it’s also becoming increasingly complex and challenging. Rising training costs demand a new level of creative thinking, collaboration, and adaptability. For Big Tech, success hinges on balancing innovation with efficiency to navigate these growing demands. For smaller players, the key lies in leveraging niche opportunities, use alternative innovation strategies beyond creating large models, and making the most of proprietary expertise.

The road ahead isn’t without challenges, but it’s also rich with possibilities. Whether you’re an industry giant or an agile startup, the ability to adapt and innovate will determine your impact. The future of AI is being written now, play your cards wisely to be part of the story.

Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej, and David Owen. ‘The rising costs of training frontier AI models’. ArXiv [cs.CY], 2024. arXiv. https://arxiv.org/abs/2405.21015.

Jaime Sevilla et al. (2024), "Can AI Scaling Continue Through 2030?". Published online at epoch.ai. Retrieved from: 'https://epoch.ai/blog/can-ai-scaling-continue-through-2030' [online resource]