Planning Engineering Projects Effectively
The database schema is not what you should worry about
Solving a customer problem with software is a complex activity. It goes from understanding the problem to identifying possible solutions to improve the situation and then delivering it: plan and execute the initiative to create the software.
Within planning and execution, an activity that usually falls within the Engineering Managers’ (EM) responsibility is breaking down the work. In this situation, breaking down means taking a product vision as input and dividing it into tasks that the team (primarily the engineers) can execute.
Unfortunately, teams usually don’t put enough attention into this activity. What is seen as a simple task of splitting the work into small parts influences how the team works, how engineers collaborate, how they perform, and, most importantly, the initiative’s outcome. As expected, lack of attention does lead to suboptimal results.
Breaking Down Collaboration
The first challenge with the idea of engineers being responsible for breaking down work is that thinking of it as an engineering-only activity is already a problem.
Cross-functional teams exist in software delivery so that product, design, engineering (and all other disciplines that might be involved) work together, not in a mini-waterfall process. In this case, planning how the team will execute the project should involve every discipline.
That’s because these perspectives should influence each other. For example, what a team needs to build will impact how engineering structures the work. And the engineering constraints in any project should affect what will be built. Because of this, this conversation cannot be a pipeline of work that goes from research to planning to delivery. Instead, it has to be a series of iterations that go from an abstract concept to the concrete feature that will solve it.
Solving It Together
A better scoping conversation considers the whole, not the product, design, and engineering aspects, one at a time. From a high-level view, the team wants to solve a customer problem as quickly as possible and do it in a high-quality way from everyone’s perspective, including engineering.
From there, engineering teams should guide how the work is planned and divided. But they should approach it from the customer problem down and not from the planned architecture up. The main question should be How can we solve this problem in the simplest way?, and not How do we build part one of this complex solution we have in our minds? Iterative and incremental.
In practice, that means focusing the team on thinking about iterations as layers of quality, where each milestone makes the solution better, but every milestone is also a complete solution. And that can only happen if engineers work with the rest of the team to define the path forward.
“When to use iterative development? You should use iterative development only on projects that you want to succeed.” — Martin Fowler
For example, in a team I have managed, we used to have a multi-month design, research, and product planning process. An engineer would be involved in it, but as a passive player, giving feedback on presented ideas. At the end of that, the product and design shared the project with the whole engineering team to be divided into epics and tickets to execute. The time from the start until we got real user feedback from using the feature, was often in 6+ months.
To reduce the feedback loop, we focused on a process in which all the disciplines planned together, with the engineering team involved as a full participant from the beginning. As a result, we reduced the cycle time from idea to production to 1 month, having the team deliver the project in multiple milestones. And that was only possible because the work was not broken down by engineering but instead shaped by the whole team.
These are familiar concepts, and excellent material is available about approaching them from a product perspective. However, EMs should also prioritize more collaborative planning and scoping since:
- It lowers the delivery risk. Focusing on shorter milestones reduces the delivery risk, as the software will be released and used sooner, and any unforeseen challenges will be visible earlier in the project.
- It provides better delivery trade-offs. With the team delivering multiple short and complete milestones, there is a much lower risk of a delivery crunch, where everything needs to come together at the last minute. Any pressure from delays is also diminished since customers will always have a solution to work around the problem.
- It delivers simpler technology. The team will naturally focus on simpler architecture by focusing on quality iterations since smaller solutions require less complexity. Because of that, technology complexity can evolve with the product, as the customer needs, and not be inserted into it from the initiative’s beginning.
- It fosters better collaboration. With the whole team collaborating on planning, engineers will better understand the problem and solution, making it easier to discuss it across disciplines and within engineering.
The Usual Roadblocks
However, iterative planning and delivery are easier said than done. There are many practical objections an EM might find when trying to move their team towards it. Here are some examples of common ones I have seen in the past.
“It is not efficient” — the most common friction point iterating on planning and execution (both at the project and story level) is that it is inefficient from an engineering perspective. Engineers often feel like delivering in iterations will lead to rework, and they are right. However, EMs should highlight that the tradeoff will be a lower-risk project from an engineering and product perspective.
“The plan might change” — Another concern is that if plans are iterative, they will be executed before fully thought through. For example, engineers might have a problem with a product decision that changes further along in the project, which is also a valid concern. I’ve seen PMs (or designers) being told off by engineering teams for that. However, iterative execution does mean that change becomes more manageable, so focusing on it will improve how the team perceives it.
“It reduces individual impact” — if a company focuses on evaluating individual impact, engineers might think they will lose points unless they plan the project’s execution alone. However, EMs should ensure the team understands that individual impact can still be observed in a highly collaborative environment.
“It doesn’t allow for good architecture” — how does a team create solid architecture if they build one piece at a time? While there are nuances to that, having an evolutionary architecture doesn’t mean not having a vision of how the architecture should be. Instead, technical leads should strive to guide the team through an architectural idea while still building it iteratively.
“It will lead to idle time” — having all disciplines working together will raise concerns that the engineering team might end up idle since there is a lead time from planning to having work ready to be executed. How will we feed the beast? The answer here is that it is okay if the beast has a little rest. Engineering slack time will be compensated by the efficiency of execution after it.
Approaching it Right
Ultimately, moving towards more iterative execution planning in software requires a holistic perspective and some conviction to be done successfully. A common challenge is to have teams implement one specific step (i.e., we should have milestones or smaller tickets) without changing how they approach the problem. Unfortunately, that usually leads to unsuccessful results, frustration, and even more ingrained habits against it.
To succeed, EMs (and their product/design counterparts) should keep in mind the overall goal of solving a customer problem in the simplest way possible, driving their planning and execution from that.
More concretely, here are a few principles that teams need to keep in mind:
The engineering focus (together with the rest of the team) should be planning a solution, not breaking down work.
Planning software execution should be part of a single conversation that goes from the problem to the solution implementation and allows quick feedback cycles. A typical example of that cycle is the team agreeing on a milestone (As a user, I want to be able to pay my bill online) that is then scoped out by engineering from a customer-first perspective (I want to be able to pay my bill via credit card and I want to be able to pay my bill with bitcoin), which then leads to a redefinition of the milestone (we should release credit card payment by itself).
That conversation is only possible if all disciplines work together on the problem and how it will be solved in terms of product, design, and engineering.
The customer problem is the main focus for everyone.
While any organization will have constraints, like the tech stack for an engineering team, planning activities should zero in on the customer problem and avoid preconceived ideas on how things should be done. It is a common challenge that people naturally bring past experience into any project (i.e., I always wanted to refactor this part of the codebase) and retrofit the project plan to their perceived needs.
While teams should consider those ideas based on their merits, the alignment point for the team needs to be the customer value they are providing.
Engineering thinking should come after the problem is defined, not before.
Engineers should allow time first to understand the proposed problem and solution and then think about how to build it. While architectural thinking is critical, it should facilitate the creation of a customer solution rather than the other way around. If I had a penny for every project I’ve seen having a great technical solution (built over a long time) and failed at first customer contact, I would have at least a handful of pennies.
This challenge is not engineering-exclusive and also applies to design and product thinking. While planning should allow people to bring their experience and research into it, the team needs to feel comfortable questioning that previous knowledge when aligning on the new solution.
Be comfortable iterating. On the work and in the process
Lastly, iteration means that there will be some rework. Iterating applies to code, as it will churn as new tickets, epics, and milestones are delivered. It also applies to the process, as the team will learn what worked well or not with every iteration. One of the main challenges of moving away from big planning upfront is that long-term planning gives people certainty.
Moving to a more iterative planning and execution mindset means accepting lessons will be learned more frequently, as mistakes and improvement opportunities are perceived more often. While that is sometimes uncomfortable, it is a positive trade-off, and it will lead to a better result in the end.
Ultimately, breaking down work is planning and should be done collaboratively. The more that silo doesn’t exist between product, design, and engineering, the better will the results of an engineering team be.
If you have found this content interesting, this post is part of a series on Leading Software Teams with Systems Thinking. Follow me if you are interested in effective engineering leadership.
