What Mark Twain has to teach us about AI

This morning I was listening to the excellent Mystery AI Hype Theater 3000 podcast, in which Emily Bender and Alex Hanna explored the use of LLM technology to replace humans in categorising data, among other things.

The particular episode took various research papers to task around their methodology for measuring the efficacy of LLMs in annotating research in comparison with humans. Emily and Alex come from a far more academic standpoint than me, and it was interesting to hear their reasoning and arguments for and against the use of LLMs in this work.

I have been experimenting with various LLM models over the past few months, to understand how they can help with our work here at Jumar, my own productivity and that of our customers, with interesting results. They are a fascinating set of tools. I've use them to do initial research, test assumptions and provoke questions. I can ask an LLM to reword or rethink something again and again without it getting bored of my questions or calling me questionable names.

I've also tried using LLMs to summarise documents, not to remove the need for reading them, but to see if they highlight things that I might have missed or given a lower priority. This has been useful, but it's given me some insight into their limitations.

I can't help coming back to this quote, often attributed to Mark Twain.

I didn't have time to write a short letter, so I wrote a long one instead.

The point being, condensing a lot of information into a shorter precis takes time, linguistic talent, and an understanding of the context of the information, not only in the view of the writer, but also of the reader.

When we use ChatGPT, or Claude, or any of these tools, it's easy to forget that no matter how much training they have, no matter what they are trained on, no matter how much tuning they have, they have absolutely no "understanding" of the context or meaning of the text they produce. When you know this key fact, it is possible to use them effectively. If you lose sight of this you are in trouble.

LLMs are fascinating, powerful tools. They are undoubtedly here to stay, and will form part of our solution set for some time yet, maybe until the next generation of AI surpasses them. Just don't overestimate their power.

#AI #LLM #PracticalSolutions #Jumar