How it Works: Why AI 'reasoning' is nothing like human reasoning

Welcome to Your AI Guide — a 30-day challenge where I explore what AI can (and can't) do in everyday work. Each day, I'll introduce one AI tool or tip and break it down in simple steps to apply yourself. Subscribe to follow along, weigh in with #30DaysofAI and let's learn together.

Today’s task: Understand why human reasoning still matters

“Time to reshoe the EV,” I said “so I’m going to Canadian Tire to get a jack.” My wife’s face was pure confusion: “Why? Do EVs need special jacks or something? What’s wrong with the one you used on the old car?” 

This type of situation, where we need to explain our reasoning process with words, serves as the foundation for a language model feature AI scientists have chosen to call “reasoning.” It also explains why the term “reasoning” is so confusing when applied to AI, and why some AI enthusiasts are so eager to downplay the extraordinary reasoning abilities of humans.

Back in our house, here’s roughly how the conversation played out:

- The EV doesn’t come with a spare tire. It has a foam reinflation kit instead. - That doesn't answer the question. Why do you need to buy a new jack when we already have one? - We don’t have one. - What? I’ve seen you use it several times! - That jack belonged to the old car. The new car doesn't have a spare tire, so it doesn’t have a jack. - OK, so take it to the shop then. - That’s more expensive. - More expensive than a jack? - Yes. - How much is the jack? - Probably $150 or thereabouts? - And how much does the shop charge to swap the wheels? - $30 per tire - Right. OK. - Actually, it’s probably going to cost more, since it’s an EV. - Wait, so EVs do need a different jack? I’m confused. - It’s because of the batteries. - Why? - They are heavy. So the jack needs to take more weight. - Right. Do you have time for this though? - No, but I’ll make it work. - Sure…

One of the many weird things about reasoning is how difficult it is to put it into words.

How it works

If you’ve followed me on LinkedIn or other social media for any length of time, you’ve probably heard me say that AI systems don’t reason, and then have long discussions with people in the comments about what reason actually is that tend to go nowhere due to a fundamental disconnect about what the word “reason” actually means. 

This is a problem - not just for me but for all of us - as we figure out how to build our future with AI.

When people talk about AI systems “reasoning,” they are referring to some variant of what’s called “Chain of Thought” (CoT) prompting. This technique emerged as a successful prompt engineering technique to get ChatGPT and other chat systems based on language models to produce more accurate completions (responses) when handed complex tasks. 

Chain of Thought prompting is a literal description of the technique: Instead of giving one instruction in one prompt, for example “What should wear when I speak at George UX Conference in November?” which will return a fairly generic response, you set the AI up to do a sequence of prompts like this:

1. Get information about the George UX Conference, including location, industry, target audience, previous speakers, etc.

2. Based on the results of 1, analyze the trends for speaker costumes, focusing on middle-aged people presenting as men with an ectomorph body type

3. Based on the results of 2, provide a recommendation of appropriate costumes for speaking at the conference.

This chain of thought where each question is informed by and builds on the completion from the previous one, consistently produces higher quality results. Which should come as no surprise as it is a far more detailed inquiry, and looks a lot like how we explain our reasoning to others. 

Today, variants of this approach are built into AI models and chatbots, typically under the moniker “reasoning” or “advanced reasoning” models. The big change is instead of you having to come up with the chain prompts and either put them in one at a time or set up a system to run them in sequence, the AI generates a step-by-step “plan” (chain of prompts), then ingests it as a new instruction, and executes each step and its own generated completion to produce a long document that looks a lot like how we explain our reasoning to others.

Some models and chat apps show you this reasoning, others don’t. The giveaway is how long it takes for the system to respond. If the answer takes more than a few seconds and the system tells you it is “thinking,” there’s a Chain of Thought sequence being executed on the back-end.

“Deep Research” and “Advanced Reasoning” are the current most advanced versions of this process. Here, the system not only generates an original set of instructions for itself (the “plan”) but also “reviews” the “plan” after each step of the instructions is complete, and modifies the “plan” based on that “review.” I use quotes liberally here because the words “plan,” “review,” and “reasoning” are all human metaphors for computer processes that are nothing like what those words describe in us humans.

This is the flashpoint that ignites endless debates in the comments. Because, when we are asked to describe what reasoning is, we tend to describe exactly this step-by-step brick-by-brick chain-of-thought reasoning. 

But that’s not how humans reason. Most of the time, our reasoning is lightning quick, making giant leaps, skipping over steps and taking shortcuts based on a myriad of factors including prior experience, current context, subconscious memory and intuition, the list is endless. And when we’re asked to explain our reasoning, we don’t necessarily spell out our chain of reasoning as it was done. Instead, we work backwards to provide a rationalization for how we got to our conclusion after the fact!

When I decided to reshoe the car, my immediate thought was “I need to buy a new jack.” It was only when my wife asked why that I started formulating an actual reasoned argument for this. And the conversation I had with her was not a reflection of my actual reasoning, but rather an explanation of how I thought she would land at the same conclusion as me (a strategy that went rather poorly as you can see above).

Here’s a more accurate description of what happened in my head when I made the reasoned decision to buy a jack:

1. I just turned on the air con in the car to cool it down before we drive our kid to class.

2. That panel behind the driver seat is still a bit loose from the summer wheel hitting it when the dealership shoved it in the back seat last fall.

3. We were lucky the ferry didn’t weigh our car when we went on that trip. The higher weight class would have cost us an extra $50!

4. The guy at the dealership said people often forget to put the wheel wrench in the trunk when they trade in their old car.

5. Father’s day is coming up, so Canadian Tire probably has a big sale on.

6. The owner of the auto shop up the street hates wheel locks because they cause endless trouble and are so easy to break.

7. I should get some jack stands as well!

A bunch of random pieces of context, smooshed together, with the need for a new jack hanging over them like a ghost, understandable to me, and a chaotic mess to anyone else. 

So to justify my reasoning, I made up a chain of thoughts that made more sense.

And millions of those chains of thoughts have made their way into the corpus texts used to build language models.

So when we prompt an AI with language patterned as a request for reasoning, the language model completes the prompt by emulating those chains of thought patterns.

And when we see those patterns, we recognize them as how we ourselves describe our reasoning and apply the same name to them: 

Reasoning.

Except it’s not.

It’s synthetic language that looks like how we describe reasoning.

Which is a thing we are not equipped to distinguish from actual reasoning, so we tend to think they are the same thing.

Even though they are not.

Which is the reason I say that AIs can’t reason.

The reason human reasoning matters is because without it, we wouldn’t be human. Replacing our reasoning with a generated mimicry of how we retroactively explain our reasoning to others is to sell ourselves frighteningly short; a devaluation of one of our core traits for the sake of making something we built sound better. 

What AI provides us in the functionality we’ve chosen to call “reasoning” is a tool that in some cases can alleviate the cognitive strain of the slow, deliberate, and logical processing Daniel Kahneman calls “System 2” or “slow” thinking for repetitive or non-critical tasks to free up our capacity for more novel and critical things. The challenge we face now that we have this tool is to use it in ways that extend our abilities rather than replace them. 

How well we succeed in that task will determine our collective future, so we have every reason to think critically about AI “reasoning!”

If you’re still with me at the end of all this, I know you have reasoned thoughts of your own on this matter. Share them with me and the world in a post or video using the hashtag #30DaysofAI, or leave them in the comments below. That way we can all reason about how to build our future with AI together. 

Thanks for reading! If this helped you make sense of AI, share it with a friend who's trying to keep up, too. Hit save so you can come back to it later – and if you're not subscribed yet, now's a good time to fix that.