Fallibility

From time to time, I’m going to point out important research. So much of the conversation is driven by hype. The hype is driven by people who have been driven by hype. The echo chamber is so intense that it makes LLMs look like objective deliverers of fact.    While I am astonished by the things I can do with the latest tools, the potential for error is high. The likelihood that an LLM will do something really dumb grows with each inflated claim about the future, with each unreviewed publication of LLM output. 

Take the claim that Artificial General Intelligence is right around the corner and a logical extension of the machine learning/LLM model. There are very smart voices (like Gary Marcus) who try hard to keep the hype in check. 

In the years that I built ethics boards for AI companies, my teams looked hard at the downsides – bias, discrimination, unreviewed use of the output, security, role-based permissions, data cleaning, management, and disposal. These are fundamental safety issues. Our view of the tools we use has to be grounded in something other than VC-driven hype. 

Thinking soberly about AI usage is incredibly challenging. The tools are often designed to get the user to overlook the flaws. (Even if that’s an unintentional consequence.) Coupled with a well-trained human tendency to believe the output of a machine, it’s easy to (1) low- complexity tasks where standard models surprisingly outperform LRMs, (2) medium-complexity tasks where additional thinking in LRMs demonstrates advantage, and (3) high-complexity tasks where both models experience complete collapse overestimate utility. 

In early June, Apple published a remarkable paper in its machine learning section (link in first comment)- “The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity”. The paper is a study of what happens when the latest ‘reasoning models’ try to tackle complex questions.    Large Reasoning Models (LRMs) are LLMs that show their work. They provide an ongoing description of the decisions they are making. Most of the major chat providers offer this capability. The underlying idea is that the user can watch (oversee) the LLM as it works. 

The paper is the result of a detailed study of that ‘reasoning’ process. The authors looked for both clear ‘thinking' and accuracy in the answer. (Note: That means they were dealing with extreme complexity that has a verifiably accurate answer; like solving a puzzle.) Theuy found three different scenarios    (1) Low- complexity tasks where standard models surprisingly outperform LRMs. It’s fascinating to think that the introduction of reasoning makes LLMs under-perform on simple tasks. That’s the opposite of what is claimed 

(2) Medium-complexity tasks where additional thinking in LRMs demonstrates advantage. This is the core claim vendors are making   

(3) High-complexity tasks where both models experience complete collapse. This is likely caused by failure at the edges of word predictability. What is troubling about this finding is that the LRMs can’t tell you when their model failed.    The user is left to wonder how to tell when the model has failed.   

Pretty quickly, this flaw is big enough to create massive uncertainty about answer quality. It is one of those gray area ethical questions. You might ask, ‘Should LLMs be trotted out to an unsuspecting public when they haven’t been thoroughly tested?’ You might also ask, ‘Who has the liability when the LLM fails at sophisticated problem solving.’     

As I inferred, I am very optimistic about the ultimate utility of our new AI tools. At the very least, they suffer from this sort of quality problem and are dangerous as a result.  

We are all a part of a vast beta test.    The paper is dense but worth a scan. The link points to a summary (the paper’s conclusions).