The UK Government is running an unprecedented agile transformation involving hundreds of development teams across many of its departments and agencies. This paper reveals an insider’s independent accounts of the inspiring successes and heart-breaking failures witnessed during a recent two-year period of the transformation.
1. INTRODUCTION
There are countless tales of failed agile transformation, often citing agile’s inability to scale. However, I have witnessed agile transformation at government scale. Success is possible, but only if there is true agile leadership at all levels, driving business agility – enabling not just a single team, but the entire organisation to minimise the time it takes to respond to customer needs. If just a single phase of a value stream has high cycle times, the team will become a constraint, limiting the agility of the organisation.
Leaders at all levels need to understand, embrace, and promote the core values of agile rather than just the rituals. Core values like iteration, collaboration, and customer focus enable business agility. But the rituals alone – stand ups, user stories, scrum masters – provide only the illusion of agility. Teams that only adopt the rituals will maintain high cycle times, thus they will be bottlenecks. If leaders do not understand the core values, they cannot discern the bottlenecks, and they cannot remove them.
Based on my dramatic experiences of leading development teams during the unprecedented digital transformation in the UK government, but relevant to almost any large organisation, this paper will demonstrate the need for agile leadership at all levels, and the consequences of its absence. This paper will focus on three key types of leadership: organisation-level, business-level, and individual-level.
Note: The definitions of organisation and business are used here loosely. An organisation in this paper aligns with the entire UK government, comprised of many businesses – government departments comprised of between one and one hundred delivery teams. However, the general concepts presented are transferrable to similar organisational structures regardless of precise naming.
2. DIGITAL TRANSFORMATION OF THE ENTIRE UK GOVERNMENT GUIDED BY GDS
The UK government took an unprecedented decision in 2011 to begin a government-wide digital transformation. Government Digital Service (GDS), a newly-formed centre of excellence, was created to lead the revolution. Their mission, to transform the relationship between citizen and state. GDS aimed to build the mindset of user needs first combined with a high-speed agile engineering capability across government, on par with the best organisations in the private sector.
A trigger for digital transformation, and subsequently the creation of GDS, was the shocking amount of taxpayer money being wasted on government IT. Large consultancies would hold government departments to ransom, locking them into strict waterfall contracts. The cost of making even the smallest change could run into tens of thousands of pounds, taking months – or even years – to deliver. And when finally delivered, quality was usually low and usability was poor. With the strict waterfall contracts in place, government departments had no ability to regularly iterate and improve their services for the benefit of UK citizens.
IT in the UK government reached a devastating all time low in 2011 – £12 billion was declared wasted on a failed National Health Service (NHS) IT project. The project, now referred to as “the biggest IT failure ever seen” (The Guardian), was symptomatic of a fundamentally broken IT capability in the UK government. It was one of many failed and over-budget attempts at delivering IT projects.
As a transformation specialist, it was the sensational challenge GDS were undertaking that inspired to me to work in government – from the biggest IT failure ever seen, hoping to become the most digitally advanced government in the world. I had chosen to join one of the biggest and most important government departments – Her Majesty’s Revenue and Customs (HMRC). HMRC are responsible for taxation. I specifically went to join an agency, part of HMRC, responsible for property taxes.
Joining the government agency as a consultant, I had two clear responsibilities. Firstly, I was to be the hands-on technical leader in digital development teams, collaboratively architecting systems and coding. Secondly, I would be supporting civil servants with the transition away from waterfall and outsourcing.
I had been part of transformations at three organisations previously, and lucky to have worked alongside many talented agile practitioners applying Extreme Programming (XP), Continuous Delivery, and Theory of Constraints. Highly aligned to the GDS philosophy, with the skills and experience to deliver their vision, I went into government full of motivation to be part of something historic. My personal goal was to help apply the GDS philosophy at one government agency and demonstrate a positive example for the rest of government, showing the brilliance that can be achieved by following GDS.
3. ORGANISATION LEVEL AGILE LEADERSHIP
For many, GDS and their grand ambitions were typical government – making bold claims to win support, but unlikely to actually deliver on their promise. Yet from the outset, GDS had been clear not just about their goals, but their strategy. GDS insisted user needs would always come first, and new services would be delivered using agile techniques. The big mystery was how GDS were going to apply those concepts at government scale. The magnitude of the challenge should not be underestimated – there are hundreds, maybe thousands, of development teams spread across many government departments where big IT outsourcing contracts and the waterfall mentality are deeply ingrained.
3.1 Standards and Assessments
GDS needed to take strong measures, and they did. They created a service manual and service standards to support every new government IT service. The service manual taught departments how to be agile, and how to put user needs first. The service standards provided hard criteria for all government departments and agencies to follow. For example, points 2 and 4 from the 18 point digital service standard:
1. Do ongoing user research
Put a plan in place for ongoing user research and usability testing to continuously seek feedback from users to improve the service.
2. Use agile methods
Build your service using the agile, iterative and user-centred methods set out in the manual.
For GDS, guidance is not sufficient to lead a successful agile transformation. GDS enforce service standards by requiring standard assessments at alpha, beta, and live stages of the project lifecycle. GDS can prevent government departments launching a new service or spending more money until the new service satisfies all of the assessment criteria.
Many would argue GDS themselves are in-fact not agile by dictating in a command-and-control fashion. However, my experiences of GDS assessments are largely positive. Government are so deeply locked into their current ways of working with little motivation to change. There is no pressure to turn a profit or out iterate competitors by developing software faster. In fact, I saw a government agency completely crippled by fear. Fear of changing anything to avoid negative publicity, resulting in little effort to change. If GDS did not strictly enforce standards, I am certain there would be no meaningful progress in government IT.
Service standard assessments are certainly not a case of friends in government doing each other a favour. I was the technical representative for my team at an assessment and we failed. We hadn’t done enough user research. GDS asked many probing questions and my team couldn’t provide the evidence of having spoken to enough different users about how the service would solve their problems. I was upset for my team, but I was happy we failed because we really hadn’t done sufficient user research. I was actually excited – I could see GDS were the real deal. I started to believe they were going to achieve something special in the history of government, and I was going to be a part of it.
I believe that all organisations attempting transformation at scale should consider adopting service standards and assessments, especially in the early stages. Although, I don’t believe they’re essential for all organisations. When teams know they have to demonstrably put users first and prove they have engineering capability to iterate frequently, they cannot take shortcuts or hide away from change. However, it is important to note one crucial characteristic of GDS standards and assessments – clarity of their goals. GDS made the service manual and standards publicly accessible online, taking great care to explain why their standards benefit all UK citizens. By being transparent and working in the open, GDS have made themselves accountable to not only the rest of government but every citizen of the UK. GDS are not typical dictators.
4. Business Level Agile Leadership
It is clear from the GDS service manual and assessment criteria, the future vision is for entire value streams to be optimised for customer responsiveness. Accordingly, senior executives in government departments should be designing their organisations so that each service is optimised for continuously receiving feedback from UK citizens and continuously being improved based on the feedback. It shouldn’t matter if the feedback requires UI, business logic or data changes, all types of change should be possible in short time frames.
Unfortunately, like many in government, senior executives who are expected to lead digital transformations and drive agile adoption don’t actually understand business agility. They aren’t focused on the flow of work and limiting batch sizes, and they don’t understand the agile software development practices that provide the foundation for business agility. As a consequence, there is a strong tendency to create separate digital teams alongside existing enterprise IT teams. This is where things start to go horribly, horribly wrong in government IT and GDS seem powerless to prevent it.
4.1 Shared Service Ownership Leas to Costly Handovers
Instead of improving customer responsiveness by creating cross-functional teams with end-to-end service ownership, at the government agency I worked for, digital and enterprise IT became competing silos. Each silo had partial ownership of services. The leaders of each silo were competing for power. Digital teams were responsible for conducting user research and demonstrating agile practices to satisfy the GDS criteria. However, digital teams only owned the websites. They could rapidly iterate on the look and feel of web pages, but the business rules and data were still owned by enterprise IT teams with the waterfall mentality.
The result was significantly longer lead times for delivering value to UK citizens due to the significant costs of continual collaboration and bureaucracy between the siloed teams. In some cases the cost of a simple change was so high that the change wasn’t even feasible.
Despite attempting an agile transformation, the government agency had maintained its existing waterfall organisation structure. Teams retained their horizontal, activity-oriented structure: the frontend, middle tier, and data teams. Aligning by activity like this necessitates handovers between teams, leading to bottlenecks. Each major piece of work must flow through all of the teams. When one of those teams has cycle times in weeks or months, they act as the weakest link in the chain, causing the entire system to have lead times in weeks or months. I have not seen a single agile success story that involved aligning by activity – the coordination costs are too high and the speed is too slow.
Figure 1: Activity-oriented teams necessitate handovers – work items require processing from each layer
4.2 Outcome-oriented Teams Minimise Handovers
Modern agile organisations strive for autonomous teams who own full business outcomes. Such vertical structure minimises inter-team dependencies. Each team is capable of gathering feedback from customers and acting on it promptly, by implementing the frontend, backend, and data changes without incurring the high costs of collaborating with other teams; other teams likely to be locked into conflicting priorities or the waterfall mentality.
Note: I am not suggesting 100% autonomy is feasible or even desirable. I am arguing that the rate of change within a team should be greater than the rate of change across teams, to maximise autonomy and minimise the costs of collaboration so that lead times are as short as possible.
Figure 2: Outcome-oriented teams have end-to-end ownership, increasing autonomy
On a sunny summer afternoon, during an enterprise IT show-and-tell session, all my ambitions of affecting change in government instantly crashed and burned. My mission was inexorably going to fail. It began when an enterprise IT project manager announced she should now be referred to as the scrum master. She then kicked off the meeting by explaining the large number of interdependent teams involved in delivering the new service. She told us how immensely difficult it was to coordinate all of the teams, yet how unavoidable the task was. She didn’t have the experience to understand the glaringly obvious – the problems were totally avoidable. The problems were the result of a flawed, activity-oriented organisation design.
I do not hold the project manager responsible in any way, nor was there any malice on her part. She was part of an enterprise IT department that would only commit to agile rituals. With guidance and support, the project manager’s tireless effort and devotion to her team could have been used to powerful effect in driving the agile transformation.
Later in the session there were sales-pitch demonstrations of the new enterprise service bus, the generic rules engine and the business process management tool. Even more dishearteningly, different pieces had been outsourced to different suppliers. The enterprise IT silo were building a huge swiss army knife platform so it would be easy to add any new requirement in the future with little need for programmers – one of the biggest fallacies in IT. During the meeting, I questioned the user need for those tools, and I questioned how the tools would improve customer responsiveness. Angrily, the head of enterprise IT asserted “we’ll get to that later”. I was extremely disappointed. We were stuck in the old ways of technology first, user needs last. If the enterprise IT teams were slow, digital teams I was working with would be helpless to achieve anything. Worse, enterprise IT were locking the waterfall organisational silos into the technical architecture, leaving absolutely no hope for future improvement.
Later in the meeting the enterprise IT chief ruthlessly asserted “IT has been a liability to the business, too slow. The industry is moving towards these tools [that don’t need programmers]”. When I attempted to question his misinformed logic, he quickly pointed to the consultant who was selling the BPM system and defensively told me “if you don’t believe me, ask him”. I was beyond words. They were continuing to practice so many of the anti-patterns that caused government IT to be in a huge mess. How could this complete disregard for user needs and lack of agility pass through a GDS assessment?
Speaking to people in government, there were rumours GDS didn’t have the authority over enterprise IT departments. This theory is supported by one of Mike Bracken’s final remarks before leaving his position as head of GDS. He bemoaned, “you can’t be half agile” (GDS Blog). This is exactly the problem I was seeing – a half agile organisation. Similarly, Paul Shetler left his leadership post in the Australian Digital Transformation Office with similar grievances: “Since great digital services also require great back-office processes, policies, operations, people and systems, IT and operations can’t be separated from digital.” (Paul Shetler).
Take notice of these systemic problems. If you’re applying agile only to pretty frontend websites, but you still have the enterprise mentality for your core systems, you’re just putting lipstick on your pig. If you want to be responsive to customer and business needs, you need to have the ability to continuously change software across your entire IT estate. It is therefore, essential, to have in place senior management who have the power and understanding to realign organisational and technical boundaries to optimise the entire value stream. Otherwise, your transformation may leave your organisation in a worse position than before you started.
5. Individual Level Agile Leadership
At its core, agile is about moving quickly. Moving quickly necessitates faster decision making, implementation and delivery, all of which require higher collaboration and trust. Consequently, it is hard to achieve agility when many people are not comfortable with the changes. Inside and outside of enterprise IT teams, I encountered many such people who were not passionate about the change in UK government IT.
For many civil servants I spoke to, GDS were not the heroic saviours they were in my eyes. GDS were a bunch of egos in London who had started putting hurdles in the way of getting their job done. I noticed this apathy present at all levels, from individual contributors, to middle management, to senior executives. For a number of different reasons, the values GDS espoused – the focus on user needs and the agile mindset, weren’t radiating to all in government. This intrigued me a lot. I could see how bad government IT was. I could see the wasted money and horrific IT systems pushed on both citizens and civil servants. Why wasn’t everybody passionate about improvement?
One digital team I worked with disliked GDS assessments. The team would rehearse assessments, carefully coordinating themselves for the event like a theatrical performance; “remember not to mention X”, “when GDS ask Y tell them Z”. People were doing the minimum amount of work possible to satisfy GDS, and they saw it as a chore, an impediment to their jobs. They had no passion for the monumental goal of being part of a government scale digital transformation.
No matter our job title, this is a problem we all have the power to solve by being purposeful and patient leaders. The government agency I worked for had fragmented leadership who were locked in a power struggle. The result was a fragmented, chaotic organisation, where delivery teams lacked purpose, autonomy, and motivation. Individuals were always under constant pressure to hit the next big deadline. They couldn’t see past the next problem, they weren’t aligned with the long-term vision, so they mistook GDS for their enemy. To combat this kind of short-term thinking, reassure people about the benefits of change by continually explaining the long-term goals and the benefits agile approaches will bring.
5,1 Patience Enables Change
One of the lessons I learned is that patience leads to positive outcomes. In one digital team, a long-standing civil servant had been brought in to work alongside contractors. For many years, she had been part of a waterfall QA team. Any bugs she didn’t catch found in production would be her fault. According to the old philosophy, a tester’s job was to do lots of manual testing and catch all the bugs. After so many years, these beliefs had become ingrained into her mindset. When I arrived talking about deploying to production every day without running a full regression suite, she was horrified.
I didn’t want to combat resistance by beating agile into her or having a quiet word with her boss. Instead, I shared my vision with her. I let her know her role in the team was vital. I showed how she can play a key role in our ability to achieve continuous delivery by providing more value upstream, working with business analysts to understand user needs and define acceptance criteria. At first she mostly ignored me, probably thinking I was yet another person coming and making lots of grand promises without delivering anything as she had become used to. As I continued to encourage her she slowly changed, though. She changed her story, now telling me there was no point investing all this time in her because she was too old. She kept saying she was five years away from retirement; it was a waste of time to teach her about BDD, HTTP, and pairing on code. She “wouldn’t be able to learn them anyway”. And that’s when I could see clearly that she did want to change; she was just worried she couldn’t.
It was a remarkable yet poignant experience to watch her grow into the agile tester role. But it did take time to let go of her old behaviours. For example, she once panicked when there was a bug in production which she hadn’t caught during testing. The whole team was always there to constantly reassure her that quality is the responsibility of the whole team. She was a special person who taught me many valuable lessons. We shouldn’t write anyone off, especially not because of age, not even if they are 5 years from retirement or have never heard of agile before. The key is not to confuse fear of change with resistance to change. Support those who initially show signs of anxiety and resistance. Make them feel valued without patronising them and they will become your allies and friends.
6. Lessons Learned
My mission to help drive digital transformation in the UK government ended in failure. But the experiences have taught me vital lessons – notably, the need for agile leadership at all levels. Despite my failure, I’ve still witnessed plenty of inspiring achievements in government, convincing me that agile transformation is possible even at huge government scale. Experienced agile leadership really is key, though.
6.1 True Business Agility Requires Executive Buy-in
At the very top of an organisation, leadership needs to provide support and reward behaviours that encourage true business agility. With service manuals and assessments, GDS provide rich support for government departments, guiding their transformation. And with assessments, GDS reward positive behaviours. If teams demonstrate they are putting user needs first and developing software effectively, they can continue to spend taxpayer money.
6.2 Design Organisation Boundaries to Improve Flow
Agile leadership is also needed at the department or business level. To move faster and achieve the agile benefit of improving customer responsiveness, organisations need an organisation design guided by Theory of Constraints. Each service should ideally be owned end-to-end by a single team. In the government agency I worked for, strategic agile leadership was lacking. The entire organisation was held back by enterprise IT who adopted rituals rather than values, resulting in no improvements to lead times because dependencies between teams resulted in high collaborative overheads. Senior management were incapable of spotting the constraints which anyone who understands true agility would have.
6.3 We all Need to Lead with Patience
The final insight I want you to take is the need for everyone to lead no matter their level. We can all help our peers to acclimatise to the fast pace of agile, regardless of their age or experience. A high-performance culture needs high performance from everyone. As you try to introduce agile, there will be resistance. For some people, it will be driven by anxiety – scared they can’t change. Through patience, perseverance and continued reassurance, you can transform their worry into passion. Your opponents will become allies of your agile transformation.
7. ACKNOWLEDGEMENTS
My shepherd, Irina Tysganok, was instrumental in shaping all aspects of this report. From the narrative, to blend of facts and emotion, to the fine details and formatting. Through our journey together she continually asked meaningful questions, proposed suggestions, and highlighted areas of the report that had potential for improvement. The first draft of this report was an 18-page braindump. Yet she read all of it and provided detailed feedback and we’ve continually iterated countless times since then. She worked so hard on this report, I hope my efforts were worthy of hers.
I also want to say a special thank you to all of the inspiring people I worked with in the UK government. Even though I shall forever hold the disappointment of failure, I shall forever hold great memories of the heroic team efforts we put in to lead transformation in a system so mightily resistant to change.
REFERENCES
The Guardian, Abandoned NHS IT system has cost £10bn so far, https://www.theguardian.com/society/2013/sep/18/nhs-records-system-10bn
GDS Blog, You can’t be half agile, https://gds.blog.gov.uk/2015/07/10/you-cant-be-half-agile/
Paul Shetler, Why’s it so hard trying to make things easy?, https://www.linkedin.com/pulse/whys-so-hard-trying-make-things-easy-paul-shetler
