The Language Problem

In the second half of the 2010s I became really obsessed with Lisp. I presume this happens to most computer-sciency people at some point. There is something delightfully nerdy about the simplicity of the language and the complex structures you can build out of it. For someone who learned other languages first (most of us, let’s be honest), programming in it starts out as an immensely rewarding challenge – it turns work into a bit of a game – “how can I solve this problem in the most emotionally rewarding way?”

Lisps are a species of programming languages where (this is a massive over-simplification, check out Wikipedia) rather than much of the structure of the program being represented by key-words (for, if, then, else etc.) in pseudo-English grammar (if X then Y, X = X + 1, call integrate(f), etc.) the language is constructed out of something called an S-expression which, (to massively over-simplify) looks like:

(verb noun noun …)

Where verb is a function call, and the nouns are an item. So, for example to do x + 1:

(+ x 1)

These can be nested so:

(+ (* y 5) x)

Will multiply y by 5 and then add it to x and return the result. Combined with a set of built-in, slightly weirdly named built in functions like car, cdr and so on for manipulating lists, and, in the case of Common Lisp an extremely powerful macro system, you can build up elegant (but completely unreadable to the uninitiated) code. Most implementations just-in-time compile their code, so it is usually quick as well. 

Lots of programmers really like Lisp, and there are several popular dialects (Scheme, Common Lisp, Clojure etc.) along with multiple implementations. Frequently you will stumble across it in an Open-Source project, and you know that you’ve run across someone’s pet (good examples being the extensive use of it in the Emacs editor, and as part of the implementation of the Julia programming language). If you look in my Github repos, you will find “Hy”, a sort of Lisp flavouring of Python, in places where I don’t expect (or want to discourage) anyone else running the code. My default for “Owain needs a quick calculator” is a Common Lisp REPL.

Non computer science nerds, by in large, do not use Lisp (there’s a weird edge case where the AutoCAD scripting language appears to be a variant of Lisp). They don’t use Haskell either, a functional programming language also beloved by computer scientists (but this love I do not understand!).

Researchers use Matlab, R, Python and spreadsheets.

A lot of people (myself included) have written a lot about the problems with Matlab as a language but if we were being practically minded (instead of fairly ideological – there are definite ideological problems with Matlab), Mathworks have done a lot to make the performance of Matlab not suck and it’s difficult to overstate the amount of the world’s research output that depends on Matlab.

R is in a strange position where it is largely mis-used. R is a functional programming language and it is generally written by a lot of researchers in an imperative style which is dramatically bad for performance. But it is extremely popular in some fields. 

And Python. Well Python is the language I use most of the time.

I have tried and know much, much better programming languages. I certainly know better packaging systems. So why Python? 

The answer is of course, that despite pip, conda and every other attempt to make Python packaging be predictable, safe and reproducible being a complete and utter disaster, all the frameworks I could want to use are available in Python, whether it’s for machine learning (PyTorch, Tensorflow, Jax, higher level abstractions like the transformers and diffusers libraries), yanking stuff out of the MariaDB databases our accounting information ends up in or quantum computing, the libraries are written to be available in Python. There are certainly other fields where this is just as true of Matlab, and, one assumes, R.

I can write interesting and pleasing code in any number of other well-structured languages.

But they don’t have the right ecosystem. So, I write it in Python, which gives me no pleasure, and so my code is horrendous.

As an experiment I recently went looking to see if I could actually do any of this generative AI stuff I’ve been doing in Python in Clojure.

Several years ago I watched a really interesting webinar at Christmas (we are that kind of family) about doing machine learning in Clojure so I had high hopes. Well, I was due to be disappointed. Cortex, the library contributed to by that talker was last committed to SEVEN YEARS AGO. And there seems to be no good replacement. Encog does not support GPUs. Clj-ml hasn’t had a commit in OVER A DECADE. Unless Bing is missing things (Bing has been a good Bing 🥺), Clojure, despite JVM enterprise mindshare and, like being a Lisp, is out.

As programming language nerds, we talk a lot about what is the “best” programming language, often about the language features that are cool and allow us to write cool code. An amazing amount of effort is put into developing new languages, with neat features which never make it to the "sharp end" of actual use. How many times at conferences have we sat through presentations about Charm++ or Chapel or Clasp where it is explained how these languages will revolutionise research software (they are all good languages!) but they just then... don't. I am definitely a programming language nerd - in the last month I've experimented with Kotlin and Zig - both as a result of off-hand comments from other people. Nothing in computing quite excites me quite like diving into something I've not tried before and making the computer do things. But it's all ... unimportant effort. I'd be better off writing Python better.

But the best languages to know remain (to much general chagrin) Matlab, R, Python and Fortran in the “research” space and C++, Java, Microsoft Java, Netscape Java and, yes, COBOL in the "enterprise" space. The end goal, sadly, is to make the computer do what you need it to do, not make code that makes you happy.