In recent years, we have seen substantial pushback on many fronts against Agile as a viable and important project management methodology. In my 2016 book, “Growing Business Intelligence”[i] (a book about Agile BI), I quoted from a 2014 article by Dave Thomas, one of the signers of the “Agile Manifesto,” in which he recommended retiring the noun “agile” in favor of the verb “agility.”[ii] Similarly, in 2018, Ron Jeffries, another author of the Manifesto, wrote in a blog post that “developers should abandon Agile.”[iii] In 2019, tech writer Kurt Cagle published the article “The End of Agile” in Forbes.[iv] This article spawned a torrent of anti-Agile comments from developers on the Slashdot website.[v] In 2020, Cliff Berg and 14 others launched the Agile2.net website to promote their re-imagining of Agile principles. Their book, “Agile 2 – The Next Iteration of Agile” was published in 2021.[vi] And in 2022, professor Miriam Posner of UCLA published Agile’s epitaph in an article titled “Agile and the Long Crisis of Software.”[vii]
In other words, a scant 15 years after authors like Scott Ambler told us we all needed to get on the Agile bus — or else — we are now being told that Agile was a failed endeavor that should be abandoned. So, what does this mean for all of us who work in IT? Where do we go from here? Questions need to be asked, and answered: Was Agile always nothing more than the latest fly-by-night IT development fad? Or just an excuse for hucksters to make a fast buck? Did/does Agile have any lasting value? Is there a way to “fix” Agile, and remake it into whatever it was supposed to be? And, if not, what can we find to replace it with?
This and subsequent articles will delve into these questions and attempt to find some answers. First, let’s try to understand how Agile came about in the first place.
A Brief History of Agile
When I first started writing about Agile, in 2005, one of the first questions I addressed was whether Agile needed to be taken seriously at all. Wasn’t Agile merely the latest application development methodology du jour, here today and gone tomorrow? And should anybody other than application developers care?
My answer to those questions was that Agile did need to be taken seriously, and by people other than developers. Why? Because (in my view) Agile was a reasoned attempt to come to grips with some of the root causes of software project failure. It’s important to remember that the 1990s played host to a large number of highly publicized, extremely expensive software project failures.[viii] When this many software development projects go south, corporate CEOs start using words like “outsource.” So, software developers were in a very real fear for their livelihoods. And the rest of us in IT need to pay attention because when software development jobs leave, our jobs won’t be far behind.
Some of the root causes of software project failure responsible for the birth of the “Agile Manifesto”[ix] are as follows[x]:
- Poorly understood or poorly communicated requirements
- Insufficient alignment with critical business needs
- Too broad a project scope (trying to do too much in one project)
- Insufficient customer involvement during development
- Lack of attention to product quality
- Unavailability of critical resources when they’re needed
- Poor planning, estimation, and risk management
- Rigid and inflexible processes and methodologies
- Lack of collaboration and communication between the various groups involved in the project
- Long delays in delivering the final product, by which time it no longer meets the user needs or business requirements
In response to these issues, a group of developers met in February of 2001 and formed the Agile Software Development Alliance (usually referred to simply as the Agile Alliance). They published a document called the “Manifesto for Agile Software Development” (usually referred to as the “Agile Manifesto”), which consisted of four values and 12 principles. The four values are as follows:
- Individuals and Interactions over Processes and Tools. This is based on the belief that success on projects is more often the product of talented, dedicated individuals working in close collaboration than of rigid, formally defined processes and toolsets.
- Working Software over Comprehensive Documentation. In the words of the Agile Alliance, “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” Working applications, even incomplete ones, add value to the business and deliver competitive advantage. Reams of product and process specifications, however accurate and well-done, do not.
- Customer Collaboration over Contract Negotiation. Since product requirements can change rapidly over the course of a project, getting continual feedback from customers is essential. If a project relies on a single product requirements document, prepared at the beginning of the project, there is a serious risk that: a) what the developer thought the requirements were was not what the customer was requesting, and b) what the developer delivers is no longer what the business actually needs.
- Responding to Change over Following a Plan. While there is certainly nothing wrong with planning, it’s important to realize that circumstances change over the course of a project. Business requirements change. Technology changes. Priorities change. People come and go. Projects need to be able to adapt rapidly to inevitable changes and unforeseen contingencies.
From these four values, the Agile Alliance derived a set of 12 working principles, illustrated by the following figure:
Principles of Agile Software Development include the following:
- Solutions are best developed by self-organizing teams of talented, highly motivated individuals in continuous collaboration with customers and with each other. Each project team member should have a general understanding of a wide range of technologies and should avoid being pigeon-holed into a single specialty. Project teams should be allowed to work as free as possible from what Scott Ambler refers to as “ossified management.”
- Application development processes should be iterative in nature, with a new software product being delivered every 2-4 weeks. Working software is defined as the primary measure of progress. Business stakeholders provide continuous input into the development process. Testing is done continually, so that problems are discovered as early as possible. When problems are discovered, the team works together to solve them, without finger-pointing or blaming. When changes are required, there is a process in place to manage them. Requirements are not “set in stone.”
- Agile processes should emphasize simplicity, “the art of maximizing the amount of work not done.” One of causes of software project failure is trying to write software that does everything for everybody, instead of defining those business needs that can be met quickly, effectively, and (relatively) inexpensively with a software solution, and empowering people to take care of the rest. One tenet of Agile that is almost never talked about is empowering people to handle problems, rather than trying to automate everything.
- Agile processes should support sustainable development. Project team members should be able to maintain a constant pace indefinitely.
- There should be continuous attention to technical excellence, good design, and supportable processes. At the end of each iteration, teams should examine what they’ve done and strive to improve in the next iteration.
One very important point is made by Cliff Berg et al: The four values of Agile are not “either/or.” They do not say, “Ignore Processes and Tools in favor of Individuals and Interactions.” They say, “Assess each situation and use your best judgment as to how much to favor one over the other.” Each situation is different, and so judgment must be used to find the best solution for each situation.[xi]
I would go a step further: Agile talks a lot about the importance of “empowered teams,” but empowerment stems from two things Agile never talks about — knowledge and responsibility. You can only be empowered to the extent that you understand the possible consequences of your actions and are willing to take responsibility for them. This means that Agile teams can be empowered only to the extent that they know enough about the business context of the project to understand the possible consequences of their decisions and have been authorized by the business to exercise responsible judgment in this area. Cliff Berg et al emphasize that empowerment is based on trust and that trust has to be earned through the delivery of value.[xii]
I would also add that the Agile need to have knowledgeable, responsible, and trustworthy workers is continually undercut by management’s insistence on outsourcing work to the lowest-cost (external) provider instead of developing in-house knowledge and expertise.
Agile sets some very laudable goals:
- Replace formalized, monolithic development processes with a more informal, interactive, and iterative approach.
- Replace reams of out-of-date documentation with actual working products.
- Emphasize customer collaboration and continual delivery of business value.
- Allow changes to requirements and priorities over the course of the project.
- Allow development processes to change and improve over the course of the project.
- Encourage experimentation and prototyping of solutions. Treat failures as learning experiences and move on from them.
- Bring resources together to collaborate on solving problems.
So, what happened to make these very laudable goals seem like nothing but a pipe dream? Why are even Agile developers recommending that we abandon Agile?
Stay tuned.
[i] Burns, Larry. Growing Business Intelligence (Technics Publications, 2016).
[ii] Thomas, Dave. “Agile is Dead (Long Live Agility)”. Blog post, published March 4, 2014. pragdave.me/thoughts/active/2014-03-04-time-to-kill-agile.html
[iii] Jeffries, Ron. “Developers Should Abandon Agile”. Blog post, published May 10, 2018. ronjeffries.com/articles/018-01ff/abandon-1/
[iv] Cagle, Kurt. “The End of Agile”. Forbes.com, August 23, 2019. forbes.com/sites/cognitiveworld/2019/08/23/the-end-of-agile/?sh=563ad95a2071
[v] “Is Agile Becoming Less and Less Relevant?” Slashdot.org, August 25, 2019. developers.slashdot.org/story/19/08/24/1748246/is-agile-becoming-less-and-less-relevant
[vi] Berg, Cliff et al. Agile 2 – The Next Iteration of Agile. John Wiley and Sons, 2021.
[vii] Posner, Miriam. “Agile and the Long Crisis of Software”. Logic Magazine, March 27, 2022. logicmag.io/clouds/agile-and-the-long-crisis-of-software/
[viii] For a partial list of major software failures, see the list in the article “Why Software Fails” by Robert N. Charette, IEEE Spectrum, September 1, 2005. spectrum.ieee.org/why-software-fails
[ix] For the complete text of the Agile Manifesto, go to agilealliance.org/agile101/the-agile-manifesto/.
[x] This list is taken from my book Building the Agile Database (Technics Publications, 2011), pp. 43-44.
[xi] Agile 2, op. cit., pp. 3-4.
[xii] Agile 2, op. cit., pp. 74-75.
