Breaking free of the limitations of modern-day software development

Part 1 – Background

About 8 years ago, before our company was born, the co-founders and I had come to notice that there were fundamental, but often overlooked, issues in the software development industry. Issues that seemed to exist universally in almost every substantial software project we came across. And while most of the developers recognized these problems once highlighted, many felt it was simply the unavoidable nature of software development - that all projects were ultimately doomed to become rigid, unmaintainable, legacy systems.

Some developers saw it as the best reason for rewriting a system every few years. Others believed the problem could be solved if business leaders were more understanding and allowed them the time to fix these issues retrospectively. The more experienced developers took refuge in architectural principles, strict governance, and managing businesses expectations, as a way to delay the effects.

As a founding team of software architects, we experienced these issues first-hand. We were being asked to build software systems for clients - often large and complex - under fixed-price contracts. We were given teams and budgets, with very little under our control. Even our estimates were often cut short by months, because “the client didn’t have the budget”, and we were left to somehow “figure it out”. It was a fiery crucible that exposed all our weaknesses.

Naturally, we turned to architecture and agile processes. But when deadlines were being missed, despite immaculately following best practices, our frustrations began to mount. At first, we blamed the business for setting us up for failure. But then as time went on, we began to notice recurring patterns.

At first, we noticed that most projects started well but ended badly and that the bigger the project, the higher the risk of this happening. In time we came to realize that there were fundamental forces in software development that were turning our projects from beautiful greenfield nirvanas into nightmares that would keep us up at night. And as we dug deeper, sharing our experiences with others in the industry, it became clear that these weren’t just affecting us, but almost everyone. As we set out in search of answers, we found that there were only ways to mitigate the causes and delay the effects. We found that despite advances in technologies and methodologies, ultimately all projects that change over time are destined to share the same fate: progressive deterioration - of the codebase, the team’s velocity, happiness, retention, ability to hire, etc.

Codebase Deterioration

Software codebases deteriorate as they change, and the graph below shows how this affects the team’s velocity over time. The point of complete deterioration is when even moderate changes to the system are considered too risky or expensive by the developers and business. The time it takes to get to this point varies between projects but can occur within as little as 6 months - the fate of many Minimum Viable Products (MVPs) which valued speed and time-to-market over investment in good architecture and sustainability.

‍

‍

From the development team’s perspective, some of the tell-tale signs that a project is experiencing this deterioration are:

• Features that used to take days to implement, now take weeks, or even months.
• The technologies have become outdated, making it harder and more expensive to hire good talent.
• It’s too complex to have a good mental model of the project.
• Inconsistency – many parts of the system are written differently.
• Technical debt is rife.
• A growing consensus among the team that the system needs a rewrite.
• Turning down new requirements from business stakeholders because “the system wasn’t designed to do that” or “the change would be too big”.
• Strong key-man dependencies on the veteran developers who know the system best.
• New joiners take months to become productive.
• Nobody wants to work on it anymore.

Unfortunately, most developers are blind to this because most never spend long enough on a project they built from scratch to see the end result. To them, legacy systems are painful, cumbersome, and inflexible simply because the team before them didn’t know what they were doing. Ironically, these are probably the same thoughts of the team currently maintaining the last system that they built. The narrative is pervasive, and it keeps most developers blind to the bigger problem.