Challenges in Developing LLM-based Applications: LLMOps Explained.

Before these proprietary LLMs, like OpenAI's GPTs, were known to the general audience, big companies' general approach was to train in-house models from scratch or tune pre-trained models, scared by the idea of letting their data leave the company boundaries. This approach may require multiple development teams and significant money and time. However, in the wake of the exciting advancements seen with the introduction of ChatGPT 4.0, business stakeholders in many companies became impatient. They demanded quick integration of AI capabilities into their applications, sometimes even assigning a single developer directly linking their applications to OpenAI's endpoints.

However, this rushed approach often leads to shortcuts in implementation, which may result in applications being linked directly to external services without backing a robust abstraction layer. Such shortcuts might seem appealing initially, but they are bound to present challenges down the line.

In essence, these uncoordinated integrations lead to what we can aptly term as "Spaghetti Connection," which denotes a system with complex, tangled, and hard-to-manage linking structures, reminiscent of the era before the adoption of Service-Oriented Architecture (SOA) and Enterprise Service Bus (ESB). If you weren't working on IT 20 years ago, read this article from Sander Hoogendoorn that covers the IT evolution of that time: Microservices. The good, the bad, and the ugly.

The result is an unmanaged, interconnected mess of dependencies, increasing technical debt and relinquishing control of the codebase. The implications of a chaotic setup are manifold. Vendor lock-in emerges as a pressing issue, with hasty adoption of AI features leading to an over-reliance on a single vendor, inflated costs, and difficulty in transitioning to a different provider. Moreover, cost planning becomes a nightmare in such a scenario as it becomes challenging to estimate future costs due to a lack of structure and oversight. 

The implications of such a chaotic setup include the following:

• Vendor Lock-In Threat: Relying heavily on specific AI features from one vendor can make it difficult to switch providers in the future. This dependency limits flexibility and could potentially lead to operational and strategic constraints.
• Cost Planning Challenges: Without a structured and managed approach to incorporating LLMs into applications, estimating future costs becomes complex and uncertain. This lack of foresight can impact budgeting and resource allocation.
• Model Lifecycle Management: As LLMs evolve rapidly, companies face challenges in versioning, non-regression testing, and dealing with concept and data drift. This complexity can significantly hinder the effective management of the model lifecycle.
• Living in a VUCA Scenario: The unprecedented pace of advancements in LLMs creates a Volatile, Uncertain, Complex, and Ambiguous (VUCA) environment for businesses. Keeping up with these changes is daunting and increases the uncertainty in operations.
• Regulatory and Geopolitical Flux: The legislative landscape concerning AI and data privacy is constantly changing. For instance, the recent overnight ChatGPT ban from the Italian Privacy Authority asking about implementing GDPR requirements shows how quickly changes can happen. Businesses must therefore be agile and prepared to adjust their IT infrastructure promptly to comply with evolving regulations.

To navigate these challenges, it's crucial to take a proactive approach. Companies must focus on 'Design for Change.' They must build a flexible architecture that can quickly adapt to LLM or other system parts changes rather than purely seeking design elegance, or worse, "whatever works."

LLM Ops Explained

Let's start by recapping the definitions: 

1. DevOps is a set of practices and principles to improve collaboration between software development (Dev) and IT operations (Ops). It emphasizes a culture of shared responsibility, frequent communication, and integration, enabling faster and more reliable software delivery.
2. MLOps, short for Machine Learning Operations, is an extension of the DevOps philosophy applied to machine learning. It focuses on the lifecycle management of machine learning models, aiming to streamline the process from development to deployment and maintenance. MLOps involves practices like versioning models and datasets, automated testing, continuous integration and deployment of models, and monitoring model performance. These practices enable teams to deliver reliable ML solutions at scale and speed.
3. LLMOps takes the principles from both DevOps and MLOps and applies them to Prompt Engineering, which involves designing, implementing, and testing prompts that guide the output of language models. Some companies, like Weights and Biases, recently announced tools to support prompting experiments' traceability.

Prompt Engineering is fast becoming a general-purpose skill, poised to be as common knowledge in a few years (or, considering the current rate, possibly months) as browsing the internet, using a smartphone, or writing a document on a word processor is today. But Prompts are the "code for LLM," following the principles and carrying the risks of programming languages. However, their simplicity means that soon, there could be a ratio of 100, 1000, or more LLM developers for every software developer in your company. Given this scale, it's essential to apply rigorous standards to this new "code," including robust testing, peer review, and automatic documentation.

In conclusion, LLMOps principles may include:

• Prompt Engineering should be treated as software engineering, with versioning, traceability, peer review, and accountability.
• Quick prototypes and experimentation should be performed in environments with automatic experiment tracking, allowing full reproducibility.
• Fully automated deployment of new versions between environments, such as testing, staging, and production.
• QA and automated prompt testing.
• Production monitoring to quickly identify Data and Concept Drifts.
• Design for Change: Loosely coupled architecture and patterns to mitigate vendor lock-in risks and simplify switching.

By adopting these practices, businesses can effectively manage their LLM-based applications and fully leverage the potential of AI while being resilient to the fast-paced changes in this exciting field.