My 2024 AI Predictions

I am writing this as 2023 is drawing to a close and we are about to welcome 2024 in the PST time zone. It's definitely been both a very busy but also exciting year for everyone in the AI space, so think it's appropriate to share some personal predictions of what 2024 might bring for AI.

I've grouped the predictions into 3 sections - the less surprising ones under the heading "Duh!" and hopefully somewhat more insightful under the heading "Huh?". Finally, Responsible AI is near and dear to my heart, so I separated all expected developments in this space into a separate section under "RAI".

So let's jump right in!

Duh!

Domain-specific models. We have already seen many models that are tuned and optimized for specific domains. It's not hard to see this trend to continue in 2024. Imagine models that are designed for accounting, economics, jurisprudence, manufacturing, construction, fisheries, and so on. Different domains are aplenty and the only question is whether there is enough training data from that specific domain to make it feasible.

Cost-efficiency. Modern AI models are computationally intensive to train and operate. Expensive! Multiple efforts are underway to make it more cost efficient and these efforts are not going to be any less relevant in 2024. How does one make it less expensive? Here are multiple possible dimensions:

• Build smaller models. If there are fewer parameters to tune, it's cheaper to train and run inference against. Duh!
• Use specialized hardware. GPUs are great, but custom AI chips can make a huge difference for both training and inference. Given that we are still in an environment of manufacturing shortages, AI chips will remain very relevant.
• Smaller curated datasets. Part of the reason training is so expensive, besides the large number of parameters to tune, is that datasets contain potentially redundant information, so being more selective could help (for other reasons too!), but the challenge is exactly how to accomplish that.
• Various tricks. Even if we are stuck using a specific pre-trained model, it may be possible to make inference less expensive, by reducing the number of tokens in the prompt or completion. Prompt templates that seem to work best, is one simple approach, but we already see other tricks emerge, such as prompt compression, or automated reduction of prompt size without substantially losing the quality of completion.

Use-case disruptions. 2023 saw a veritable explosion of new highly capable models. And while we already see how these are incorporated into various use cases like computer-aided design, help centers, pharmacology, personalization, and so on, we expect many more existing use cases to be disrupted by modern generative AI technologies. Think about your own business. What are the business processes that can be disrupted in novel ways?

Huh?

What's the date today? It's easy for us to make fun of the models today when they are not aware of some recent developments. They've been trained on datasets that are point in time, and yet the clock keeps ticking. In 2024 I expect more progress to be made in keeping the models continuously updated with more recent developments. Querying the internet for some updates is one way to do it, but keeping the training datasets up-to-date and offering faster updates to pre-trained models is going to be unavoidable. I would not be surprised if entire industry will emerge around offering curated and current training data.

Not just language or images. I am not thinking about other expected modalities of content such as audio and video. Instead, what if the transformer-like architectures were applied to much more different types of data and use cases. What if we train a model on software logs? Wouldn't this be helpful for detecting anomalies? How about models that are trained on various quantized metrics? It would not be hard to get exabytes of such data collected and patterns extracted from such data would be highly valuable. Predictive maintenance, weather forecasts, sales anomalies... Sky is the limit.

Black swan discovery. It's impossible to predict black swan events by definition, but I feel like the rate of innovation in AI has accelerated in the past few years to the point that the likelihood of a discovery that will bring about a new step change in capabilities of the so-called frontier models is fairly high. Could there be a discovery of a new architecture that makes the models significantly more capable, especially when it comes to reasoning? I put the odds of this in the neighborhood of 30% easily. It's up to us in 2024!

RAI

Hallucinations. Much has been said about this property of the generative models. And, of course, hallucinations are there practically by design - the models are probabilistic in nature. While we have some success with limiting them via prompt engineering, more explicit fact based logical reasoning capabilities are required. Turns out this is really hard! While I expect progress here in 2024, I am actually skeptical that this "problem" can be outright solved in the next year.

Deep fakes. We've seen them. They get better and better with time. Harder and harder to separate from reality. We may already be at a point where we can't rely on human faculties for this and need specialized tools. While I expect progress on this front in 2024, I also expect the fakes to, unfortunately, be ever more successful at manipulating us. I expect a few notable incidents to occur in 2024. It's probably not helping that it's going to be an election year, here in US.

Regulation. With no clear and specific regulations around AI in the past years, it's not surprising that many AI practitioners looked at Responsible AI with "best intentions" attitude. Meaning that the approach was haphazard and inconsistent. But things are gradually changing. We've just seen the publication of ISO 42001 standard on AI Management System. Recent progress on the EU AI act with more clarity to come next year. While 2024 will not be the final chapter in the creation of AI regulatory landscape, it will certainly clear up a few things!

When I talked about "progress" in 2024, I really meant, "substantial progress". There are many other areas where one expects progress, but the fundamental problem will not be solved. Take explainability of AI models for example. Many new techniques in this space have emerged and continue to be emerging, but we still struggle with explainability of simpler non-frontier models and even the concept of explainability itself, that I don't think we can expect substantial progress in 2024 alone. Another area of a similar nature, are the various intellectual property questions that have emerged with respect to generated content. While we will no doubt make progress in the public discourse in this space, I expect the legal theories to be working itself through the courts well beyond 2024.

Needless to say, the above is a highly subjective account. I am sure I have forgotten about some other important aspects. Whether you agree or disagree with any of the above, your comments would be highly appreciated.

Let's check back in a year. Happy 2024!