Agile in Large Complex Organizations
Is there room for innovation in agile?
The title may sound more ominous than intended. I would like to unpack some of my learnings and thoughts. In complex systems, large scaling frameworks create rigidity and will over-constrain an organization.
It is quite easy to throw rocks at scaling frameworks and blame them for all kinds of dysfunctions, the reality is there is a shared accountability of the people adopting them either on their own or through the support of a consultancy.
Paying your way to enterprise agility is quite tempting for large complex organizations looking for a packaged solution.
It is a little bit like buying a house fully furnished thinking you’ll get everything you need. Then you realize it was built for a family of a smaller size and you are now forced to share the same bedroom, surely the two pool tables the house has are more important than your family, having beds.
For the few of you not familiar with Geoffrey Moore’s book ‘Crossing the Chasm’ and the famous diffusion of the innovation curve, you have seen it before even if you don’t know the name/origin. It nicely plots the spread of innovation overtime on a curve and how each market has differing needs to be convinced.
It holds true for any innovation and one of the overlooked parts, is the inertia required to cross from the early adopter stage to the early majority — this is often where customers and markets have different needs and take distinct types of proof to get their business.
Quite often emerging and novel practices quite make it over the chasm and remain limited to fewer people.
Chris Matts talks about this in his talk ‘Agile — the broken learning machine’ e.g. quoting Feature Injection as one such example.
Agile is not new
Where would you say agile really is on this curve? I have heard many practitioners claim that agile is now at the point where it is somewhere between the late majority or the laggards (depending on your perspective).
If you took a similar graph and plotted agile on a Wardley Map, you would see similar progress, although Simon differentiates Scrum and XP. The point is that they have reached their novelty and are no longer differentiating, instead, they are taken for granted.
I don’t fully agree that on Simon’s depiction of where agile works best, in terms of the X-axis, I will save that for another time. It is fair to say that where you have less certainty in your work and you want to reduce your risks, you should use shorter feedback loops to enable faster learning.
Is Agile dead?
There are numerous videos and talks with the same name, I don't think agile is dead, but I think agile has become something far different than was originally positioned some 20 years ago in the Agile Manifesto. It was clearly intended for software development and building products.
Now it is used to explain a set of more common practices that any organization/team can consider using in their ways of working. I don’t have any concerns with adapting methods to your context and getting value out of them. What does tend to create concern, is when agile gets turned it a commercial / packaged offering, which is focused on giving everyone a role with a top-down command and control structure that looks nothing like agile. Unless the system is there to enable and act in an incremental way, then everyone at the top will still be expecting big bang delivery.
I’m a fan of the approach set out from Klaus Leopold around Flight Levels. It’s an interesting way of focusing on strategy, coordination, and delivery of the work such that everyone is thinking about limiting the work-in-progress and not just the teams delivering. I recommend his talk on how agile teams don’t create business agility on their own.
Agile isn’t a company-wide operating model
When agile is implemented in large organizations these days, it tends to be done so in a push manner vs a pull manner. Dan Mezick has talked about this extensively and has an amazing series of books and an approach he offers called Openspace Agility which I highly recommend looking into.
The idea is to have people turn up on their own volition and volunteer to be involved in the change rather than top-down leader selection. This has adverse effects if people have agile thrust upon them. People are far more likely to adopt something new when they are participating and wanting to contribute.
There are countless examples, books, and talks about the reversion of behaviors after an Agile Transformation where people went through a sheep dipping exercise, they get assigned roles and a couple of weeks training and then pushed into the deep end. Then 3 months later they are back to their previous ways of working.
Look out for where leadership holds the ‘agile coach’ responsible for other people’s commitment and behavior. They may be talented, but it’s a whole team sport and cannot individually be responsible for the success of the team.
Is there room for innovation in agile?
Yes, there is still room for new patterns to develop, especially when it isn’t always clear how to handle complex dependency chains that span an organization. This is likely to be a common enough set of problems that other organizations can share their practices and approach too so that the wider community still has something to learn. Not every company is a tech company, in fact, most of them are not. Most companies start with a messy current state which includes many dependencies and significant amounts of technical and organizational debt.
There’s a lot of room to reduce the need for large complex programs/initiatives over time. Where instead organizations and delivery structures are set up to be able to deliver incrementally and more independently over time so that things like feature flags can be used, as well as creating more trust / confidence in testing such that a complete front to backtesting effort isn’t required either as often or at all.
That means prioritizing efforts to address technical debt, not only for obsolescent technology but for architectural patterns that enable better isolation and modularity when it comes to testing. Many of these concepts and practices are not new, the challenge is that those who know the practices are not prevalent in the Financial Services industry.
Aside from the technical and architectural challenges, there is more to explore without resorting to large scaling frameworks when complex coordination is required. I believe that we don’t need to resort to ‘the stick’ to get things done. You don’t have to deliver ‘on time’ if you deliver early and often. Creating those sorts of capabilities is far more resilient than making sure people hit the delivery date, especially if that means paying down technical debt as you go.
An interesting pattern to explore is hexagonal architecture (popularised by Alistair Cockburn) which lets you layer your code/architecture in a way that abstracts ports and adaptors so that things can be done independently between the inside and the outside of your system but with loose coupling!
So rather than communicating that Agile will ‘solve all problems’, I prefer to phrase it as agile helps highlight the problems and allows for the experimentation of practices to explore! Experiment, learn, and share!
