Overengineering and Overadoption

As coders, we usually begin our careers by throwing together commands sprinkled with single-letter variable names and wild contortions of logic. The code usually does the job inefficiently and fails in catastrophic ways in unexpected situations. As we progress, our code becomes more robust and correct, but for a while, we continue to put together code that isn’t very well-designed. Code like this is “underengineered.” Underengineering is bad, unless it’s the result of a conscious decision to get the job done faster by skipping most of the design phase and taking on some “technical debt” (AKA “design debt”).

After a couple years of unintentionally writing underengineered code and dealing with the fallout, we compensate by planning ahead and designing better systems which anticipate future needs. Our code becomes easier to read, understand, and work on. That might be the sweet spot, because the next step for most of us is overengineering.

Once we get good at designing systems in abstract ways, we often proceed to develop them in the most abstract ways possible. Overengineered projects often take months or years to complete. It’s difficult to get consensus when every last detail has to be perfect. When we overengineer, it’s likely that by the time we’re done, the business or technology environment will have changed in ways that render much of that excessively well-planned work irrelevant. And, we can’t see the future, no matter how hard we try. When we overengineer, despite all of our planning and abstractions, we often fail to anticipate the abstractions we’ll actually need. Voltaire is credited with the saying, “The perfect is the enemy of the good.”

Why do we overengineer? It seems smart at first. If some abstractions are good, then lots of abstractions are better! Programmers also get bored. We like to solve new problems and develop new skills. A coder friend both wise and tired beyond his years once told me, “There are only seven problems in software engineering, and we just solve them over and over again.” It took me years to understand what he was saying, because I’ve always said, “Software engineering is the best job in the world, because it’s never the same job twice. If it is, you’re doing it wrong!” In fact, we’re both right. His point was that although we may be solving new problems with new languages, abstractions, techniques, and technologies, at the bottom of it all will always lie just a few fundamental solutions. Loops. Conditionals. Functions. Reading from and writing to files and databases. Because of this, programmers can get bored even while engaged in one of the most interesting jobs in the world.

Overadoption is a similar problem that doesn’t get nearly as much attention as it deserves. Programmers love new technology. But the newest, coolest technologies are also the most buggy. Last year I found and fixed a bug in our payment system that could have caused us to approve all payments of a certain type without actually checking to see if those payments were authorized. The bug was not introduced directly through the fault of any of our coders. It had been made possible long ago by the inclusion of an open-source software package that we were using before it was quite ready for enterprise use. After we installed it, it went through so many changes so quickly that we couldn’t keep up. Updating the package would have broken our code, so we were stuck living with a bug that had been fixed years ago. We were early adopters. But it’s possible to adopt too early. We’ve also quickly adopted and trashed various wiki systems, file systems, project management software systems, and version control schemes. We’ve gotten to a pretty good place today, but several of the steps in between could have been skipped entirely if we’d just been a little more patient and reflective before jumping on those rickety bandwagons.

Engineers sometimes, usually unintentionally, make themselves indispensable by hoarding knowledge about arcane legacy systems. It’s quite possible to become indispensable for the opposite reason as well. If the latest, greatest technology is installed the moment it’s released, it will be difficult for everyone in the organization to keep up. We all enjoy learning new things, and building our skills helps the organization as a whole. But sometimes the best thing we can do for the company is to go to work and solve those seven boring problems all over again for a few hours. Sometimes we just have to sit down and bang out some boilerplate.

William F. Buckley, Jr. said, “A Conservative is a fellow who is standing athwart history yelling ‘Stop!'” I’m not a conservative. I’m just standing athwart technology calmly advising, “Slow down a little.”