Why Fully Autonomous Software Engineering with AI Remains a Distant Dream

I am often surprised by how many people in technology remain unaware of why fully autonomous software engineering is impractical with Artificial Intelligence agents based on current methods (Neural Networks, including Reinforcement Learning). Multi agent frameworks (Devin, etc) based solely on language modelling, both now and for the foreseeable future (until a breakthrough occurs in Neurosymbolic AI), become caught in infinite loops due to the four major limitations of neural networks, along with an additional limitation specific to language models. I will now outline these limitations and explain how they impact our ability to fully automate software engineering.

The primary reason multi agent frameworks become stuck in loops when writing software is the Out of Distribution (OOD) problem. This occurs when the language model encounters input data, such as a novel bug, for which no information exists within its training distribution. Neural networks typically experience a significant decline in performance on out of distribution inputs because they are unable to extrapolate effectively. While Chain of Thought (CoT) can occasionally help address such bugs, the solutions generated are often irrelevant, causing the system to enter infinite loops. This remains true even when additional context is provided through prompts that include error messages from the Language Server Protocol (LSP), linting tools, compilers, or interpreter exception handling. Attempts to use Retrieval Augmented Generation (RAG) for out of distribution data would still require examples of all possible out of distribution scenarios, an unachievable task akin to modelling infinity.

This issue is closely related to the problem of inference on under represented data within the training distribution, known as the Long Tail Problem. When inference is performed on under represented cases, performance suffers because sparse training data limits the model’s ability to interpolate effectively. This leads to a higher rate of errors for edge cases. Attempting to model (through Pre-training, Fine-Tuning, and RLHF) all potential edge cases is infeasible, as it would again require modelling infinity. Slightly off topic but Adversarial attacks exploit out of distribution and under represented observations from the long tail within the training distribution.

The third issue is known as the Autoregressive Error Propagation. Machine learning involves an optimisation process where the difference between input data and labels is minimised across a dataset. Even at a global minimum, errors persist due to noise within the dataset. Each inference carries a small probability of error, and these errors propagate in autoregressive language models. As the length of the generated output increases, so does the error rate and the likelihood of producing an incorrect output. In a multi agent setup, error cascades occur when the output from one agent is passed to another, compounding the error rate further.

The fourth issue is the Symbol Grounding Problem. In language models, tokens (or sequences of tokens) represent symbols in our languages, but these symbols lack inherent meaning. A symbol serves as input that the mind interprets into a stored abstract representation. However, this abstract representation does not exist within the symbol itself. These representations are learned through cultural and experiential data (including multi sensory fusion across our senses), and multi agent collaboration that cannot be replicated by models trained solely on natural language data.

As machine learning models identify shallow statistical relationships between sequences of tokens, they lack the ability to reason meaningfully because meaning is not encoded within the symbols themselves. Without the capability to learn true abstract representations, they are unable to perform extrapolation, logical deduction, or abductive reasoning required for high level software engineering, language models only imitate reasoning through fuzzy retrieval from their training distribution.

The final issue, Hallucinations, specific to language models, is already widely recognised as such I will spend little time on the issue. Hallucinations, where the model generates incorrect or fabricated information, further increase the error rate, exacerbating the challenges discussed above.

The first four issues have been well documented in the literature for decades, yet many remain unaware of them. I'll finish up with Software engineers roles will not disappear. Code generation will become ubiquitous, with software engineers reviewing the generated output for errors and alignment with requirements and designing the architecture of the solution, tasks at which language models perform poorly because they lack the ability to understand abstract concepts.