Overengineering: Why We Do It and 10 Ways to Tackle It
Overengineering impacts more than you think
More often than not, people act on the side of caution rather than taking a risk. It’s human nature to be risk-averse.
When developing software products, this sometimes creates more complexity than necessary. And other times, the necessary details are skipped which results in an under-engineered product. The balance is non-trivial.
Max Kanat-Alexander explains that overengineering is “when your design or code actually makes things more complex instead of simplifying things.”
“When your design or code actually makes things more complex instead of simplifying things, you’re over-engineering.” — Max Kanat-Alexander
Overengineering software products is the norm and often happens without realizing immediately.
One reason for this is that we’re focused on continuously delivering value whilst managing the pressures of deadlines, competition, uncertainty, and ambiguity that comes with developing software.
As a result, the overengineering has a negative impact on a team’s productivity because the burden of additional complexity must now be managed.
It’s more time-consuming, takes our focus away from working on the “cool stuff”, and further adds to the pressure of already tight deadlines. In the end, reaching the defined business objectives and goals becomes more complicated to achieve.
As a product manager, it’s crucial to be able to detect when your product is being overengineered since it can affect your ability to move fast. One of the common symptoms of overengineering is confusion about the product’s direction. I’ll elaborate on more symptoms shortly.
The reality is that we’re not going to be able to prevent or eliminate overengineering because of the innate traits of being human. We’re naturally biased. Quick, automatic thinking is in our nature. But that’s no excuse for ignorance.
Here, I’m going to share my thoughts on why we tend to overengineer software products, what it looks like, why it’s a problem worth addressing, and the steps we can take to tackle it.
Why Overengineering Is the Norm
Imagine that you’ve been tasked to build a mobile app that makes it easy and convenient for people in densely populated areas to exchange homegrown produce and homemade goods (e.g. organic sourdough, kombucha, etc.).
Immediately, several questions come to mind. They might be some of the following:
- How much will it cost to build?
- How long will it take to build?
- What is the most important problem we want to address?
- What features should we build?
- What skills do we need to build this mobile app?
- How might the target users gain access to this app?
There are countless other questions that could be asked. Some questions may even seem irrelevant but they are not because each question has a cause or effect on another. It’s all connected.
Donella Meadows, an influential author of the book Thinking In Systems, explains this connection in great length with the concept of feedback loops.
Going back to the mobile app example, is the problem that locally produced goods are going to waste or that we’re consuming foods of mass production that lack quality nutrition, or something else?
If a specific problem has not been identified as worthwhile solving, then how do we know which features to prioritize and invest in building first?
A problem not clearly defined can promote overengineering simply because the team will end up designing and building features either just in case or because it might solve one of the problems. This makes the measurement of success more difficult as well.
Broadly speaking, there are two main reasons why we overengineer software products.
Reason 1: Software development is a problem-solving exercise
Firstly, developing software is an uncertain journey of experimentation that’s guided by a product vision. We use “tools”, methodologies (e.g. Lean Startup and The Design Sprint), and a variety of skills to bring concepts to life.
There’s no manual or template for developing software products and implementing a product or feature can be done in hundreds of ways. It is unknown specifically when a product will attract mass adoption or achieve product-market fit.
Our product requirements often change or evolve as we gather insights from research, collect feedback from customers, test new ideas, and prototype concepts.
That’s all because developing a software product is a problem-solving exercise. The formula for creating commercially viable software products is complex. Getting it right the first time would be like winning the lottery.
Reason 2: Humans are often irrational
Secondly, humans develop software products in dynamic environments. We crave certainty but nothing about software development is certain. We also have complex minds but the way we make decisions and deal with bias is anything but simple.
It’s estimated that an average adult makes 35,000 remotely conscious decisions each day. Surprisingly, it’s interesting how little control we have over our day-to-day thought process.
I can only imagine what’s going through the mind of the decision-makers who decide how much to invest in new business initiatives.
Smartphones and other smart devices have also changed the way we absorb and interact with information because they act as an invisible cable to deliver digital information to and from our minds.
This further encourages our instinctive, automatic, and emotional (system 1) thinking. However, this is the opposite kind of thinking that is predominantly required for developing successful software products. We need to think critically to make good decisions.
In addition to the way that we are programmed to work as humans, there is pressure to demonstrate value to customers, capture market share, and attract talent. All business initiatives ultimately need to generate an expected return on their investment.
More importantly, the software product needs to be commercially viable. This, combined with the two main reasons above, is why overengineering is the norm.
And because it’s not obvious what exactly should be designed and built to achieve a specific return and capture a specific market size, we need to keep the design and development of software products as agile as possible.
To identify overengineered software, we first need to understand what overengineered software looks like.
Symptoms of Overengineering
Software products are often over-engineered from either a design or code perspective. They manifest as particular behaviors or actions. Many of those involved in developing software products would likely be familiar with the symptoms of overengineering.
Here’s how some people in the software development community describe the symptoms.
Jeff Sternal describes overengineering as:
“Code that solves problems you don’t have.”
Over-engineering can appear in software architecture, infrastructure, and security. Marko Vuksanovic shares some examples in these areas:
“Creating a micro-services architecture when you don’t have a single customer.”
“Designing a system for 50ms latency when your users care about 10 seconds.”
“When it comes to security, we design for threats that are not relevant to your business.”
Whether it’s writing an article, creating a presentation, designing a UI, or designing the architecture of a platform, pen to paper is still a great technique for getting your thoughts out so that they can be better organized.
The same goes for writing code. Dsimcha explains:
“…when everything goes through so much indirection that it’s hard to find the piece of code that actually implements some concrete, domain-level piece of functionality.”
Once we’ve had a bad experience, from that point forward we tend to “do a little extra” to prepare for an unforeseen unknown. In product development, this is dangerous because we:
“build more than what is really necessary based on speculation.”
John Kim explains that the two golden rules of programming: 1. “It’s easy to understand,” and 2. “It’s easy to change”. He states that they are often in conflict with each other:
“Code that is easy to read and understand often means the code is hard to change when program requirements change. And the inverse is strangely true as well.”
Mark Canlas, an Apple software engineer, shared on Quora that:
“Overengineering is the act of solving a problem while optimizing for other, tertiary conditions that are unlikely to be true, thereby driving up cost.”
There also are times where we have no legitimate reason at all. Features are designed and code is implemented based on an assumption that we likely won’t need it but we do it anyway, “just in case” or because of the classic “we have a grand vision.”
Earlier, I mentioned that overengineering is also a concept that applies to the team designing the product. As product designers, we conduct user research to help us determine which pain points should be prioritized for resolving and which gains should be created for the end user.
The short story is that the data will neither tell us exactly what to design nor will it tell us how to design the solution. Hence, getting the feature or product design right the first time is a rare occurrence.
We make some assumptions and sometimes too many. Essentially, we end up imagining too far into the future. This results in a combined effort of the product, design, and engineering teams sometimes building more than what the users need.
I’m confident that many, if not all, the symptoms above you were familiar with if you’ve been involved in building a software product. But why should we care?
Let’s take a look at some of the consequences and impacts of overengineering.
Consequences of Overengineering
There are many consequences from overengineering a product but ultimately it limits an organization’s ability to react to change. Let’s unfold that.
Higher business costs
Over-engineering leads to higher business costs, many of which are “invisible”. The biggest challenge here is accurately tracking all the impacts and specifics about where the costs are incurred.
Furthermore, being able to proactively forecast these unexpected costs appears to be a one-off exercise rather than daily exercise. Costs can be incurred in the form of technical debt, decreased operating efficiency, increased expenses, and/or failure to deliver value in a timely manner.
The latter has many flow-on effects across the initiatives in different teams.
For example, sales and marketing often have efforts running in parallel with product development in preparation for the product launch so changes to this schedule mean interruptions for other planned work that sales and marketing have in the pipeline.
The more money that’s spent on unexpected costs and interruptions, the less money there is for new opportunities such as exploring innovation and hiring unique talent.
Technical debt accumulation
Products are frequently developed in fast-paced, dynamic environments. Deadlines are often missed and unknown unknowns are the norm. Product managers make trade-offs by the second and engineers are continuously pushing code changes to production by the minute.
Frequent changes and high-pressure conditions make it difficult to make good decisions which often result in technical debt. Psychologically, it’s easier to put a card for the technical debt in the backlog column. In reality, we don’t look back because we’re too focused on rapidly moving forward.
This technical debt, if left unattended for too long, decreases the productivity and operating efficiency across the entire product development team.
For example, the increased complexity that exists in the code makes it harder for engineers to understand and evolve in the future. Another example is that more time is spent having discussions about how to manage or fix the burden caused by the shortcuts taken.
This takes away time from discussing ways to grow the platform, e.g. what experiments we can run to increase retention.
Likewise, in the case of increased complexity in the code, engineers spend more time understanding implementing future solutions and managing the debt instead of how to improve the product’s performance, streamlining the release of new features, etc.
Decreased team productivity and efficiency
Failure to deliver value in a timely manner is also related to the main consequence. When working with teams who were unclear about their business’ objectives and goals, they become inefficient at successfully delivering the product as one team.
For example, I was working with a business that prioritized launching a new feature. It needed to be a combined effort from design, product, engineering marketing, customer support, and sales to deliver it successfully.
However, the business objectives and goals for the feature were not clearly understood across the different teams. The result was a poor quality feature delivered due to the mismatched understandings. As you can imagine, this resulted in a low-quality user experience for the customers.
The consequences above eventually create lower team morale and motivation declines. Also, the product development costs end up being higher than originally estimated.
All these consequences boil down to costs. Because of this, businesses run the risk of not getting their expected return on investments. To mitigate this risk, let’s next take a look at how we detect when software is being overengineered.
How to Tackle It
Products that are overengineered often have exponential flow-on effects that are dangerous because they don’t appear immediately.
For example, I was working on a product that defined business workflows for a new product. When it came to developing the workflows in the product, instead of starting with the key workflows, all the workflows were implemented at once.
A lot of time was spent researching and trying to perfect the workflows prior to implementation. To our disappointment, the result was an overly-complex set of workflows that were difficult to use in real life.
From early user testing, we learned that some workflows were too rigid in “special” and rare situations. We also learned that the workflows were more guidelines rather than precise instructions that were followed strictly.
Not catching this soon enough led to unnecessary development work and poor user experience for the users. Hence, it’s important that we make informed decisions on how to prioritize work and mitigate risk.
So, how do we catch ourselves in the act?
Marko Vuksanovic, lead consultant at ThoughtWorks, has a quick and easy way to detect if you’re overengineering. Simply do this:
“Ask: “What would happen if we omit X?” and if the answer is: “Nothing,” or “Nothing critical to business objectives” then you’re likely overengineering.”
But this is neither a silver bullet nor does one exist. Perfect information does not exist when it comes to developing software products. Therefore, critical thinking plays an important role.
Asking questions like: “What’s too much code?” and “What’s too much design?” are not useful because they result in a subjective answer. Plus, there is no reliable way of measuring what the right amount of code or design is.
The amount of code or time spent on a design is not the same for any feature, enhancement, or product. It all depends on the context, requirements, and current problem that you’re looking to solve.
On the other hand, coaching and enabling good decision-making skills within and across teams can help to detect overengineering. Specifically, we can catch ourselves in the act by asking some questions.
These questions can assist in identifying the right balance between how much to design and code written during the implementation of a product. Thus, we are enabled to make better decisions.
Six questions to detect it
- What is the problem we’re solving and how does this code/design directly solve it?
- What are the scenarios or user stories that your code/design will solve? What is the most important user story that we need to deliver?
- Why did you choose this design/implementation approach?
- What are the alternative ways in which we could have approached the implementation (e.g. framework, language, etc)?
- Is this the simplest way to design/implement the requirement? If not, what are some other ways?
- Can we make this simpler and still achieve the requirement?
If there is a lack of confidence in the answers to one or more questions above, there’s a high chance that some overengineering has occurred. At this point, it would be wise to stop and clarify what you and your team aim to achieve.
Then, thoroughly review the designs and/or code to minimize any overengineering. This isn’t the complete list of questions to ask, but rather ones that I’ve found useful in my experience when working with different product and engineering teams.
“The best way to avoid overengineering is just don’t design too far into the future.” — Max Kanat-Alexander, author of “Code Simplicity” and “Understanding Software”
Whilst detecting and minimizing overengineering is an improvement, the best approach is to avoid overengineering in the first place by not designing too far into the future.
Four ways to avoid overengineering
- Define and communicate objectives with clear metrics of success.
- Make requirements transparent.
- Be sure to communicate dependencies and collaborate as necessary to come up with the best solutions.
- Evaluate your product early and often with simulations of how it will be used so that you can identify if your product is delivering the expected value.
There may be rare times where overengineering is the result of a certain business decision or a strategy to get a product to market sooner.
However, this is done with the knowledge that additional time and costs will be incurred with a plan on how to address the impact of overengineering in the near future.
“Perfect is the enemy of good enough, done is better than perfect” — Helen Macqueen
If we can develop mindfully, think critically, and collaborate efficiently, teams will be enabled to get things done, specifically effectively solve problems and adapt swiftly to change.
If you ask: “What would happen if we omit X?” and the answer is “Nothing” or “Nothing critical to business objectives,” then you’re likely overengineering.
Conclusion
Do you have experiences like this to share? I’d love to hear any other questions or techniques that you have found useful in identifying if your software product is being overengineered. Feel free to share in the comments below.
