Nobody Codes a Bad System On Purpose

I have been writing up a little one pager for a JasperFx Software client for their new CTO on why and how their flagship system could use some technical transformation and modernization. I ran my write up past one of their senior developers that I've been collaborating on for tactical performance improvements, and he more or less agreed with everything but felt bad that I was maybe throwing the original development team (all since departed for other opportunities) under the bus a bit -- my words, not his.

My response was that their planned approach might have been working just fine upfront when the system was simpler, but maybe they would have happily and competently adapted over time as the system overgrew the original patterns and reference architecture, but just weren't around to get that feedback.

And let's be honest, I know I've created some clever architectures that got dropped on unsuspecting other people in my day too. Including the (actually kind of successful) workflow system I did in Classic ASP + Oracle that had ~70 metadata tables and the system that was written in 6 different programming languages.

That brings me finally to my main point here, and that's even though I see plenty of systems where the codebase is very challenging to work with and puts the system at risk, I don't think that any of the teams were necessarily incompetent or didn't care about doing good work or didn't have an organized theory about how the code should be structured or even what the architecture should be. Moreover, I can't say that I've even seen a true, classic ball of mud in a couple decades.

Instead, I would say that the systems that I've seen in the past decade that were widely known as having code that was hard to work on and suffered from poor performance all had a pretty cohesive coding approach and architecture. The real problem was that at some point the system or the database had grown enough to expose the flaws in the approach or simply grown too complex to be confined within the system's prescriptive approach, but the teams who owned those systems did not, or were not able to, adapt over time.

To try to make this post not ramble on too long, here's a couple follow up points:

• I think that if you have technical ownership over any kind of large system, or are tasked with creating what's likely going to grow to become a large system, you should adopt an attitude of constantly challenging the basic approach and at a minimum, being aware of when intended changes to the system are difficult because of the current architectural approach
• Moderate on the idea of consistency throughout your codebase or at least between features. On my recent appearance on DotNetRocks, I veered into a sports metaphor about "raising the floor" vs "raising the ceiling" of the technical quality of a codebase. Technical leads who are worried about consistency and prescriptive project templates are trying to "raise the floor" on code quality -- and that works to a point. On the other hand, I think that if you empower a development team to adapt or change their technical approach over time or even just for new subsystems, and if the team has the skillset to do so, you can "raise the ceiling" on technical quality because I have found that one of the main contributors to bad system code is rigid adherence to some kind of prescriptive approach that just doesn't scale up to the more complicated use cases in a big system.
• If you follow me or have ever stumbled into many discussions about the Critter Stack, you'll know that I very strongly believe that reducing code ceremony. For me this means forsaking too many abstractions over persistence, reducing layering, favoring a vertical slice architecture, and honestly, letting in some "magic" through conventional approaches (that's a debate all by itself of course). I think there's a huge advantage in being able to easily reason about a codebase throughout a use case from system inputs all the way down to the database. On the other side of that, I think that complex layering strategies will often put too many layers of code to the point where teams cannot easily understand the cause and effect between system inputs and what the outcomes actually are. That is, I think, the number one cause of poor system performance by teams comes from not being able to easily see how chatty a system becomes between its front end, server layer, and database. As an aside, I've seen OpenTelemetry tracing be a godsend for identifying performance bottlenecks in unnecessarily complicated code by showing you exactly how many queries a single web request is really making.
• Just to hammer on the code ceremony angle yet again, I think the only truly reliable way to arrive at a good system that meets your company's needs over time and is easy to change is iteration and adaptation. High ceremony coding approaches retard your ability to quickly iterate and adapt, and but more of an onus on teams to get things right upfront -- which just isn't consistently possible no matter how hard your try.

Summary

Anyway, to close out, I think that the mass majority of us really do care about doing a good job in our software development work, but we're all quite capable of having ideas about how a system should be coded, structured, and architected that simply will not work out over time. The only real solution is empowered teams that constantly adapt as necessary instead of letting a codebase get out of control in the first place.

Wait, what's that you ask? How do you work with your product owners to give you the space to do that? And that's my cue to start my week long vacation! Good luck folks, and try to be a little easier on your feelings toward the "previous folks". And that goes double for me.