We found my first ever exploitable vulnerability
I have mentioned last week here and on 𝕏 that I had a great deal of stress because of something I was ethically bound to not talk about.
Now I can.
TL;DR: We found a way to trick Min.io into allowing PUT requests into buckets without signature verification - this is fixed in RELEASE.2025-04-03T14-56-28Z - patch now!
Unidentified Contact
On Friday, we had our regular meeting of the team who are working on building out our “Unified AI” stack and my colleague Andrew Esterson (who was standing up Min.io as an S3 server for the stack) mentioned he was having a weird bug with authentication – namely that it seemed to ignore the fact that he’d accidentally put the wrong secret key into his requests and it was still allowing him to PUT (aka write) to his buckets (but not list them or do GETs etc.).
As you may know I suffer from a great degree of OCD about security stuff (so much so that I am medicated) and immediately I started trying to replicate it on the Min.io server I use for teaching object storage as part of a module in the Computer Science Department. And I could not. Relieved, I messaged him, and he mentioned that he was using Amazon’s boto3 library, while I was using the Min.io client and the Min.io python library.
And so I took his code, installed boto3 on one of the Almalinux 9 test VMs in our private cloud and again could not replicate the problem until at his suggestion I updated it to the latest version from pipy and voila! I could write with the wrong secret to my buckets.
In a bit of a panic I hopped on the Min.io slack and asked who I should send security reports to, go the address and wrote up what we had and sent it, along with, at their request, logs.
Pause in hostilities
This was all very badly timed, because of course it was the weekend and because due to the way UCL chooses to run its IT, we were meant to expend all of Monday in our increment planning session.
I did work on narrowing down some of the scope of the problem Friday night, with the help of Brian Maher (who put together our Ingress setup) and I wanted to be super sure that that was not the cause.
We identified that the change seemed to be related to the arbitrary introduction by Amazon of a breaking change (https://github.com/boto/boto3/issues/4392) to boto3 between v1.35 and v1.36 when they added enhanced integrity checking to put requests, partly because this seemed to add a bunch of guff to the headers of requests and partly because versions <1.36 did not exhibit the ability to write with the incorrect secret key.
Hitting the books
On Tuesday I tried to solve a problem – why was it that some Min.io severs were vulnerable and others were not, even when I ran the same code against them? I had tried standing up toy model systems, a lot of toy model systems, both with an ingress server like our own, and without and none of them were exploitable, while systems attached to our main ingress server, and Andrew’s deployment with its own ingress server were. Andrew’s ingress server in particular was worrying because it is a completely plain install of ingress-nginx into the same Kubernetes cluster as the helm chart for Min.io was deployed – this must be a startlingly common configuration. But I couldn’t replicate it.
I had to do some reading. I understood conceptually at a high level when you do a write to an S3 server, but I knew none of the detail – clearly authentication was breaking but how?
When you make an S3 request, a signature is generated based on both your secret key + your access key (which Min.io has three versions of – users, access keys and role users) and the value of specific headers for the request. This is inserted into the headers as the field “Authorization”. This is how your request is authenticated – if the signature in the request matches one generated by the server with the same data and algorithm (hence why you must also specify in the Authorization field which header fields make up your signature) the request is accepted.
Looking at the logs, I noticed that the headers for requests to a server with ingress through our standard ingress server were different than they were to otherwise identical test servers I had set up (i.e. with the same ingress setup).
I started to suspect that boto3 was treating different servers differently.
And then it hit me – the only difference between the servers that boto3 sent the headers that bypassed verifying the signature and the ones that were not was that the former were using TLS. I had skipped setting up certificates on the test systems because it was a pain, unknowingly violating a cardinal rule of doing experiments – changing two things at once. I enabled TLS on my test systems and boto3 immediately started sending them the problematic headers. It’s important to point out that the problem here is not the use of TLS – that version of the boto3 library merely sends somewhat different headers to https endpoints.
I pinged Andrew on Slack and told him I’d worked out how to get boto3 to send the headers to a Min.io server reliably, I was going to try and create a minimal exploit and send it to the Min.io people the next day and that “the gold standard for that would be a curl command with the right headers”, and went home for the (now late) evening.
I work with amazing people
I got into work on Wednesday with a plan – I’d stand up a VM with Docker on it, run the Docker image for Min.io with TLS configured, and write a minimum reproducible python script using boto3 to trigger the flaw, and send it to the Min.io people. Of course, my early morning was dominated by meetings, so I left that VM provisioning and when I returned, Andrew had sent me a 1-line curl script that did it.
At this point I should maybe explain that I come across a lot as being full of myself. I’m extremely good at a narrow field of things but those things do not include either “reverse engineering web requests”, “auditing code”, or, until this week “any knowledge in detail about S3”. If I had created that line of curl it would have taken me quite some time, learning how to extract the relevant details from the ingress server, and then I’d have had to read the curl man page (because I use wget) etc. This was entirely side-stepped by ARC having a vast pool of brilliant people of which Andrew is one.
I generalised this into a short shell script, giving it a definitely wrong signature of “deadbeefdeadbeefdeadbeeddeadbeeddeadbeefdeadbeefdeadbeefdeadbeef”, proved that it worked against the Docker deployment of Min.io, wrote everything up, sent it to the Min.io people and had lunch.
After lunch I spent some time minimising the headers sent and narrowed it down to three or four options which seemed not only to disable signature validation, but also much of the other validation of the request headers (e.g. time)
Time zone lag → Fix
The Earth being a globe, the Min.io developers are on a different time schedule from us and by the time I got in on Thursday morning, I had emails confirming that they had found the exact header and problem code and that a fix was on the way. We identified the PR as it went through and this morning, fixed binaries are available to everyone. You can see the release here:
Security Notice: https://github.com/minio/minio/security/advisories/GHSA-wg47-6jq2-q2hh
(private for at least 1 month) Github repo of exploit code: https://github.com/owainkenwayucl/minio-kerfuffle
I have deployed the Docker containers and the RPMs for RELEASE.2025-04-03T14-56-28Z and can confirmed they are fixed.
Thanks
None of this would have been possible without Andrew Esterson and Brian Maher’s hard work, nor without advice from many people at UCL – our fantastic Information Security Group in particular. Thanks also to the Min.io developers for being quick to respond and identify the exact bug and release fixes.