All Software Estimates are Lies
You might expect me to follow that up with some pithy reversal, like George Box’s famous quote about models, but are estimates useful? That’s the question we’ll be digging into here.
Software Development is Research
Do pharmaceutical companies ask researchers to provide an estimate on when they’ll have a cure for cancer finished? Researchers can provide estimates on how long it will take to complete a particular study — really good estimates, usually, since research plans generally have schedules — but outcomes like “a cure for cancer” depend on what those experiments uncover. To estimate an outcome like that, we’d need to know in advance what results our experiments will yield, but if we knew that, we wouldn’t need to conduct the experiments. You can’t really see past the result of the next experiment, because what that experiment teaches you is going to inform what the next step should be.
In software development, we don’t really spend any time on problems we already know the solutions to. If solutions already exist, we include a package or library with that solution as a dependency, or download a script, or copy the existing code, or something else that takes seconds to implement so we can move on to the next problem. Nearly all development time is spent on novel problems, problems that we don’t know the answers to. They’re frequently novel in astoundingly boring ways, like “How do I save a data model with these specific fields to this specific database?” But it’s precisely the ways in which this situation is unlike any other (or at least, any other that we could find) that takes up all of our time.
Software development is almost entirely a rapid cycle of experiments. We have an idea about how we might solve the problem, we take a few seconds to write it up, and then we run it. If it works, we move on to the next problem; if it doesn’t, we try a different idea. When we run out of ideas we turn to Stack Overflow and try something we find there. Where a clinical trial may last months, most of the experiments we conduct in software development take less than a minute — a quick change to a line of code, and then seeing if our unit tests still pass or what the result is when we rerun the program or reload the browser. Our cycles are much shorter, but that doesn’t make it any more possible to see past the result of the next experiment. If it works, we’ll be done; if it doesn’t, we’ll have to try something else. Every software developer has faced problems that they poured hours into solving, but even when you’re deep into it, at every single point it’s possible that the next run will be the one that works. At no point do you know if you’re thirty seconds or three hours away from solving the problem, until it’s solved. All we can honestly tell you is the thing we’re trying right now — but stopping to tell you about it would probably take even longer than it would take to just do it and tell you if it worked or not.
The more experience you have laying bricks, the better you can get at estimating how long it will take to build a wall, but that’s because these are all knowable variables. It’s not easy work, but in terms of complexity, it is a simple problem. There are few (if any) emergent properties. Each brick is generally pretty similar to every other brick, at least in the ways most relevant to building a wall. The amount of time it takes to lay all the bricks will generally be pretty close to the time it takes to lay one brick multiplied by the number of bricks to lay.
None of this is true of software development. Software developers might fool themselves into thinking that they can get better at estimation with experience (especially when they’re encouraged to think so), but since we spend all of our time on precisely the problems that no one has solved before, any past experience we have is, by definition, irrelevant. If we have relevant past experience, then we’ll fill in that part in minutes, or even seconds. Where we’ll spend nearly all of our time is with the problems we haven’t solved before — where we start a cycle of tiny experiments.
The cone of uncertainty is, of course, a real thing. As we find the solution to more problems, fewer unsolved problems remain, so there must necessarily be less uncertainty about what it will take to solve ten problems than there would be for those same ten problems plus ninety others. On the other hand, with software development problems, it can be hard to explain the difference between the easy and the virtually impossible, and it’s not always intuitive, even to veteran developers. Sometimes it’s not until you’re deep into engaging with a problem that its true complexity comes to light. Yes, we likely have less uncertainty when we’re down to just ten problems remaining, but any one of those ten could turn out to be a monster that takes twice as long as the previous ninety-nine combined. The cone of uncertainty tells us that’s unlikely, but human brains have a terrible habit of conflating “unlikely” with “impossible.” When that very human innumeracy gets baked into our project plans, companies and livelihoods get crushed in the gap between them.
The Deadly Costs of Our Deceits
For most software developers, estimates are lies that we tell our bosses — generally at that boss’s insistence. Good developers push back against it. They try to explain why their estimations are useless and should not be used for, well, really, anything. But their bosses insist that they need estimates, so they give them something — generally significantly padded to protect themselves, with lots of qualifications and reservations. In a less healthy organization, the boss pushes back, pressuring the developers to provide a smaller estimate. You might hear the exasperated “I don’t understand why it would take so long,” from someone who clearly isn’t interested in the weeks of lectures it would take to explain it. Even in organizations that trust their developers, though, those estimates become Gantt charts and project schedules. They quickly transform from a best-guess into an iron-clad contract, with people’s jobs on the line if the deadlines derived from those guesses aren’t met.
The Project Management Institute’s 2018 “Pulse of the Profession” survey reported that about 48% of projects miss their deadlines. This probably isn’t a very surprising fact to most of us, and yet we remain steadfast in our faith that this is the only way we could possibly run a business — with an approach that we all know is as likely as a coin toss to end in success or disaster. I’m sure most of us have faced long nights and weekends to meet a deadline, even though those longer hours result in less productivity, and can even make the project take longer as you introduce more bugs, which ultimately require more time to fix than the extra time you put in during the “crunch.” We have whole shelves of studies showing us that this isn’t just an ineffective strategy, but an actively counter-productive one, requiring even more of the time we already don’t have. This would be bad enough if it were simply a bad business practice — sadly, businesses make terrible decisions every day — but it also kills three-quarters of a million people every single year.
But what other outcome would we expect? All of our estimates are lies, so the project plans and deadlines that we create out of them are fantasies. Why would we expect their relationship with reality to be any better than a coin toss? And what would we expect to happen to human beings when we pin their lives to those fantasies?
Above, I mentioned that we transform best guesses into iron-clad contracts; did that remind you of the third value from the Manifesto for Agile Software Development?
Customer collaboration over contract negotiation
We can see that the current approach of contract negotiation isn’t meeting anyone’s needs — not the developers, not the business’s, not the customers’. Remember in our scenario above, when the boss came to a developer to ask for estimates, the developer pushed back, and the boss insisted that the business needs estimates so it can plan? Let’s back up for a moment and dig into why the business needs estimates in the first place. There are a few really good reasons here, including:
- The business needs to plan resources, to make sure we’re being efficient with the tasks we assign to people.
- The business needs to make sure that everything is coordinated for the big product launch.
- The business needs to update its executives and shareholders on what the state of the business is.
In each of these cases, though, the fact that our estimates are all lies creates a tangled web of new problems that ultimately defeat our original purpose. These are important goals, and failing to meet them is a serious problem. The only way to really accomplish these goals is to be truthful and transparent — and that starts by moving away from contract negotiation and towards customer collaboration.
Project Planning and the Agile Enterprise
One of the most critical reasons why businesses need good estimates is so that they can plan resources. If it’s going to take the UX team two months to create the designs, we need to make sure that the development team is going to be ready in two months to accept the hand-off and get to work. If the development team is going to take six months to implement those designs, then we need to make sure that the QA team is ready to test them in eight months. If the UX team misses its deadline, it causes problems all the way down the line, as all the teams that depend on them are impacted.
This is Big Design Up-Front (BDUF). We’re placing a big bet — in this example, nine months of work, which could be the difference between success and going out of business for a lot of companies — that we’ve completely understood the problem, come up with exactly the right solution, and nothing important is going to change in nine months while we do our work. Each one of those assumptions is individually a bet that few gamblers would want to roll the dice on, but all together, and with such high stakes, it’s a minor miracle if it ever works at all.
Stake everything on one enormous, risky bet is not a great idea; instead, we should make a small bet, and then another, and then another. Don’t commit to nine months of work; just commit to two weeks, and see where you are then. To do this, you will very likely have to do reorganize how you work; instead of the UX team doing all of the research and design and then throwing it over the wall to the developers, and then the developers doing all of the primary developments before throwing it over the wall to the QA team, put UX designers, software developers, and QA specialists into a single cross-functional team, and then carve the work up into vertical slices — whatever piece of full, user-facing functionality the team can complete in the allotted amount of time — and then see where you are at the end of that.
A cross-functional team doesn’t need a long-term plan to fill their capacity. With each new cycle, they’re ready to take on whatever is the most important work that needs to be done next. They don’t need estimates; rather, they just need to know what the organization needs them to take on next. They need to communicate with the rest of the organization and provide transparency into what they’re doing, so it’s clear exactly where things stand. The organization needs to use that transparency to decide on the next most important thing to be done, so that the team knows what to start working on when the next cycle of work begins. When the market changes, you can reassess what the next most important thing is in light of that change. Each vertical slice that’s delivered brings with it new discoveries and new understanding, which might change the way you understand the problem. When that happens, you can reassess what the next most important thing is in light of those new discoveries. That ability to respond is why we call this agility.
This may all be well and good for small teams, but clearly it doesn’t apply to large organizations, right? If you have 50 designers, 100 developers, and 30 QA professionals, are you going to combine them into one cross-functional team of 180 people? That would never work — which is why you break them into 30 teams, each with 1 QA professional, 1–2 designers, and 1–4 developers. If you combine a vertical slice approach to the work you take on with a vertical slice architecture and/or microservices (if you take care that what you’re making really are microservices), each of those 30 teams can take a different slice of the same product and you still won’t have to worry much about collisions (better known to developers by one specific type of collision, the merge conflict).
Ready for Launch
Even if we can work more effectively in small, cross-functional teams that regularly deliver working vertical slices of our ultimate product, that doesn’t change the fact that we need solid estimates to plan our product’s launch, does it? Launches are difficult to pull off, and they rely on proper coordination. Marketing efforts can be large, complicated affairs, and making sure that everything is lined up to promote a launch is a big job. We can’t just slap something together at the last minute. We need a target date so that we can plan our marketing efforts around that.
This is all true — at least, for what’s called a hard launch. That name exists to differentiate it from a soft launch, which has become increasingly popular over the past decade. In a soft launch, you put something out into the world to fairly limited fanfare. This approach works really well for start-ups and minimum viable products (MVPs) — and MVPs work really well with cross-functional teams that regularly deliver working vertical slices. Since you’re prioritizing the next most valuable slice with each cycle, you simply identify when there are enough slices to provide some value to some real people in the world and release that. The team’s work barely skips a beat; now they’re adding value to a live product, but that’s purely a business decision.
With a soft launch, you run the risk of releasing a product that some users might consider underwhelming; that’s why you don’t put a huge marketing effort behind it. Early adopters give you feedback, which you use to prioritize the next most valuable slices to deliver. When you pursue this strategy, waiting too long to release is riskier than releasing too early, since you risk spending time on slices that your users don’t actually care about. The sooner you can put something out, the sooner you have users who will let you know what they want. As Matt Mullenweg put it:
Usage is like oxygen for ideas. You can never fully anticipate how an audience is going to react to something you’ve created until it’s out there. That means every moment you’re working on something without it being in the public it’s actually dying, deprived of the oxygen of the real world.
With this approach, you don’t have to try to predict the future. As the real product that’s in the world, being used by real people, gets better, you put more marketing behind it, gradually. There is no big launch, no coordinated marketing blitz. You start to develop and run campaigns only once the product exists and is ready to be marketed to a wider audience.
That said, it’s undeniable that some products really do call for a hard launch, but how often are release dates given for highly anticipated products, only for those releases to be pushed back again and again? We can’t get away from the underlying fact that all of our estimates are lies, meaning that the release dates around which we’re planning all of our efforts are almost guaranteed to slip, no matter what draconian levels of “crunch” we might coerce out of workers (since, again, this strategy just pushes the completion date further out of reach).
If we want to be honest with ourselves and make plans that will actually work, then we may need to take some notes from the success of soft launches on how better to plan and execute hard ones. If we develop software iteratively, then when we reach the point where a soft launch strategy would launch an MVP, a hard launch effort can start a closed beta with a select group of participants. Only when you have feedback from beta testers that you have a reasonably good (if still unfinished) product do you begin planning your marketing campaign in earnest. At this point, you’ll have a much better idea of what it will take to make a great finished product, and how much more work there is to be done. Those details will likely change between the time you start planning and your launch date arrives, but since you’ve already developed a good, core product, you know that you’ll have something that you can launch. You can start making announcements that will build up excitement and anticipation, while you book appearances at trade shows, appearances on podcasts or television, ad-buys, and all of your other marketing efforts for a coordinated campaign around the launch date. Instead of rolling the dice on whether or not you’ll actually have a product that’s ready to go when that date arrives, you’ll know that you already have a great product. The only question left will be how much better it will be by the time it’s ready for launch.
Being Honest with Bosses
At this point, you might be able to guess what I’m about to say about the use of estimates to report to executives and shareholders. The people making the decisions do need to know what’s going on and how things are going so that they can make important strategic decisions, but estimates don’t provide that, because all of our estimates are lies. They obscure what’s actually going on. Complicated Gantt charts, projections, plans, and estimates can make a project that’s been disastrously off-track for months look like it’s still going well, right down to the last moment when the deadline comes and goes. Herein, though, we find the real value of estimates: not in the information they provide, but in the information that they hide.
Agile teams can provide far greater transparency on what they’re doing, what they’ve accomplished, and what’s yet to be done. This gives decision-makers as clear an understanding of the situation as can be provided, and allows them to make decisions which the team can begin implementing as soon as the start of the next work cycle. The downside is that the decision-makers will know exactly what the teams are doing, what they’ve accomplished, and what’s yet to be done — which can be terrifying. Software development is risky. There’s no way to get around that. We could find out that a digital product isn’t technically feasible or commercially viable. Good product management puts the biggest risks up-front (this is where prioritizing the most valuable next slice comes in; revealing your biggest risks is immensely valuable), so if that’s the case we learn it sooner rather than later. But even when we do everything right, software development is ultimately research. Even at the very end, there’s a chance we may discover something that means all of our efforts were an enormous waste of time and money.
Transparency doesn’t leave anywhere to hide from that terrifying truth. On a human level, we can all appreciate why a decision-maker would gravitate towards a system that hides all of that anxiety behind a big, impenetrable wall of project planning. The risk is still there, you just don’t have to see it. If it all works out, you never have to see it at all. We have a big, successful launch and we all get bonuses for the great job we did. If it doesn’t work out — as happens about 48% of the time, according to that aforementioned survey — those risks explode catastrophically from behind that curtain. Shares plummet. People are fired. We demand to know what happened and who’s responsible for all this, but if we’re really being honest with ourselves, the answer is obvious: this isn’t a failure of the employees nearly as much as the inevitable result of hiding from all that risk instead of facing it.
Agility allows us to stop lying to our bosses, and it gives them the transparency to see what’s really going on, but this is the challenge that separates bosses from leaders. Uncertainty is a scary thing to confront. You’ll need to face the fact that there is risk in this business. You can move it around, you can hide it — we even have some pretty effective ways to minimize it, if you’re willing to confront it — but you can never eliminate it. It’s an inextricable part of the work. If you can accept that and have some faith in your people, then they can be honest with you, and you can find ways together to confront, minimize, and manage those risks. If you can’t, then you’ll have to tell your people to keep lying to you, and then wonder why so many projects keep inexplicably ending in failure no matter how perfectly you update your Gantt charts.
The Primary Measure of Progress
When you pick a proxy as a “good enough” measure for the thing you’re really interested in, you embark upon an exacting education in every possible way in which that proxy is not the thing you’re actually interested in. Long ago, social media platforms reasoned that time on site was a good proxy for how much users enjoyed what they had to offer, and began to optimize for that. They ended up mimicking addictive painkillers and slot machines. Likewise, I’m sure many of us can recount stories of organizations that complete incredible sums of story points or Scrum teams who consistently increase velocity with every sprint, and yet produce frustratingly little working software.
The seventh principle of the Manifesto for Agile Software Development is:
Working software is the primary measure of progress.
Not story points, not velocity, not estimates: working software. If you make those your measure of progress, Goodhart’s Law tells you what you can expect: you’ll produce a lot of story points and a lot of velocities, but maybe not much-working software. If we want different results, then we need to focus instead on the things that we actually want.
The #NoEstimates tag on Twitter began with people who’d recently completed large, complex projects without ever making any estimates sharing their stories. It is possible. People are doing it, even with huge, complex projects like yours, right now. Instead of writing science fiction about what the future could be like, we can organize cross-functional teams who deliver vertical slices of working software so we can measure our progress not in terms of story points, velocity, or any other lies, but on the actual working software that exists and runs.
If, after all of this, you’re still certain that there’s some use for which you need estimates, I’ll offer this. The use of story points and velocity is well-known in Scrum teams. The team gets together at least once per sprint to groom the backlog, during which time they go over the user stories in their backlog and assign each one some number of story points. A common choice is Fibonacci sequence numbers. This allows the scrum master or product owner to calculate a velocity (an average number of story points that the team has completed over the past few sprints). Using this average, they can predict that if the team maintains that velocity, they’ll be able to finish so many user stories in the next sprint, which can be turned into a prediction of how long it will take to get through a backlog.
In his book #NoEstimates, Vasco Duarte points out that if we drop the Fibonacci numbers (e.g., replace 1, 2, 3, 5, and 8 with 1, 2, 3, 4, and 5), we get almost identical predictions. Furthermore, if we drop those estimates entirely (effectively, every user story is worth 1 story point), we still get almost identical predictions. This means that we can either do regular backlog grooming for an hour a week, go through each story and use planning poker to properly size them, or we can just count the number of stories in the backlog. Either way, we’ll get equally useful predictions (though one of these requires far less time and effort).
The Scrum Guide highlights certain values that the process calls for. One of these is “Courage,” a value that it copied from its predecessor, Extreme Programming. Some question its inclusion. Why would Scrum require “courage”? Ultimately, though, I’ve seen more Scrum teams fail for lack of courage than almost anything else. It’s usually a lack of courage that leads organizations to build cargo cults instead of agility, and it’s often a lack of courage that pushes us to hide behind estimates rather than confront the inescapable fact that software development is a risk. When we’re too frightened to face that, those risks accumulate and grow in the places where we hide them. They wreck our projects, our companies, our careers, and even our lives. If we confront them, we can minimize them. We can discover them before we’ve wagered everything on the outcome. We can mitigate them. We can work around them. We can deal with them. We can manage them. But only if we first have the courage to confront them and be honest about them — with each other and with ourselves.
