In the first article, I laid out the basic premise for this series: an examination of how Agile has gone from the darling of the application development community to a virtual pariah that nobody wants to be associated with, and an exploration of the very important question of what we should replace it with.
We started with a brief history of Agile, in an attempt to understand how it came about and what problems it was trying to solve. For years, I’ve been promulgating what I refer to as The First Law of Systems Analysis: In order to change anything, you first must understand how it got that way.
In the second article, I enumerated a number of the criticisms that have been leveled against Agile, including many from my own writings.
In this article, we’ll see if we can distill Agile down to some fundamental set of principles that can be applied to all projects, regardless of the type of project or the methodology being used.
What Is Agile Really?
Given the critiques described in the previous article, is it possible to even say what Agile is (or should be) or isn’t? As Cliff Berg et al note, a good part of the problem with Agile is that there isn’t any real academic or philosophical underpinning to it; it’s just a wish list of “nice to haves.”
When I first started writing about Agile, I noted that “Agile” was a term that you almost never heard outside of IT. You never, for example, heard anyone describe themselves as an “Agile Carpenter” or an “Agile Plumber” or an “Agile CPA” or an “Agile Sales Executive.” Why is this? I believe it’s because almost everybody, outside of IT, knows how work is supposed to get done!
This is why I’ve always used analogies like home remodeling and landscape gardening to describe how Agile project work should be done; because this is how almost everybody on Earth approaches any complex undertaking:
- We start with a vision of what the end result of the project will look like.
- We take the first few steps toward the realization of the vision.
- We then reassess our end goal in light of what we’ve accomplished so far.
- We assess our progress to date and figure out ways of improving our efforts.
- We take the next few steps toward our goal.
- We repeat steps 3-5 until we get to a point that may not look exactly like our original vision but is where we are satisfied enough to call the project complete.
The Agile community itself seems to be coming around to this same perception of Agile as a natural human process (as opposed to yet another IT-centric application development methodology). Dave Thomas, one of the principal authors of the “Agile Manifesto,” now advocates abandoning the concept of Agile (as a noun or an adjective) in favor of the concept of Agility (as a verb).[i] Thomas’ approach to Agility reads very much like the “do it yourself” process I described above:
- Find out where you are.
- Take a small step toward your goal.
- Adjust your understanding based on what you learned.
- Repeat.
In other words, as I’ve said before, Agile is not so much a defined methodology as much as it is a way of working (or of thinking about working). As Dave Thomas puts it, we don’t do “Agile Development,” we do “Development Agilely”! He makes an additional important point: Ultimately, what we do trumps what we call it.
Nevertheless, let’s take another, deeper look at Agile and how it works[ii]:
The important thing to remember about Agile as a methodology is that it manages work. Specifically:
- No work is done that is not agreed to by the team.
- No work is done unless it moves the project forward.
- No work is done unless its value can be demonstrated.
- No work is done in excess of the value it creates (i.e., no “nice to have” work!).
- No work is done that is perfect (i.e., only “good enough”).
- The right work must be done at the right time by the right people in the right way.
- Processes of work must be continually improved.
When you work on, say, a SCRUM project, you spend a lot of time in stand-up meetings, talking about what work you did yesterday, what work you expect to do today, and what obstacles (if any) you are encountering. You spend time in meetings deciding what work will be done in a sprint. You participate in post-sprint review sessions where you talk about what went well, what went poorly, and what could be improved. All of these activities are work management activities.
So, why is it important to manage work? Work is managed so as to manage risk. Specifically:
- The risk of doing work that isn’t needed.
- The risk of not doing work that is needed.
- The risk of delivering too little too late.
- The risk of doing work too soon.
- The risk of doing work that must be done over.
- The risk of doing the wrong kind of work.
It is the risk (and expense) of doing the wrong work that was the original impetus for the Agile Manifesto. Recall that Agile was the developer community’s response to a litany of highly publicized and very costly software development project failures during the 1990s. Most of these software project failures were the result of spending time and money doing work that wasn’t needed, wasn’t the right work, or wasn’t delivered soon enough to do any good.
So, why is it important to manage risk? Risk is managed so as to manage opportunity. Specifically:
- Deliver as much value to the customer as soon as possible.
- Take advantage of windows of business opportunity.
- Quickly leverage breakthroughs in technology.
- Continually improve processes and business delivery.
- Continually improve quality.
- Incrementally grow knowledge, expertise, and capability.
In other words, Agile manages work so as to maximize opportunity (or value) and minimize risk. Or, to put it another way, Agile manages work so that opportunity isn’t swallowed up by risk. The intent of Agile is to always achieve an acceptable balance of risk (what we currently don’t know and can’t do) and opportunity (what we must know and be able to do in order to be successful).
The reason it’s important to understand this is that no one prescriptive approach (e.g., SCRUM, SAFe, or Kanban) is going to work equally well for all projects. Each project needs to undertake its own assessment of both the risks involved and the opportunity concerns. Is the market for the product under discussion volatile, or stable? Is the market window narrow and fleeting, or relatively open? Is the risk of marketing a product with bugs substantial, or minor? Can these risks be easily mitigated, or not? Is the product mostly customer-facing, or not? Are requirements likely to change frequently, or not? Is the associated business process well-defined, or undefined? Is the underlying technology well-understood and stable, or cutting-edge and rapidly changing? What management reporting and approval process(es) need to be accommodated by the project? The answers to these and similar questions will determine what sort of approach needs to be taken with the project.
So, what might we consider as characteristics of a genuinely Agile approach to work, regardless of the type of project or the particular methodology being used? Here are a few suggestions:
- Deliver value to customers as quickly and incrementally as possible.
- Ensure continuous interaction with the customer (even if the customer is not continuously involved with the project).
- Be transparent in communicating with all stakeholders, even if they’re not embedded with the project team.
- Accept continually changing requirements to the greatest extent feasible.
- Try to eliminate unnecessary and extraneous work.
- Support experimentation (within defined limits).
- Ensure freedom to fail (quickly and inexpensively).
- Create cycles of continuous improvement.
- Enable sustainable development, with time to rest, reflect, and refactor.
- Work positively and collaboratively.
- Automate processes to the greatest extent possible.
- Integrate testing with development.
- Empower people to organize themselves and solve problems.
- Provide supportive leadership throughout the project.
In my book “Data Model Storytelling,”[iii] I also address the question of what Agile is not. Here is what I think should not be part of an Agile approach:
- Refusing to consider architectural issues. It needn’t be “Big Design Up Front (BDUF),” but enough front-end work should be done to ensure that the final product is fit for purpose and is robust, scalable, performant, secure, and easy to maintain.
- Ignoring design considerations. Again, it needn’t be BDUF, but enough design should be done at each stage of the project to ensure that the team is all on the same page about what work needs to be done, and how. An effective design helps remove uncertainty about how things should be done, which in turn reduces errors and rework and increases the overall velocity of the team.[iv]
- Ignoring leadership. Even with Agile approaches (or, perhaps, especially with Agile approaches), firm and wise leadership is needed to ensure the success of the project. As we’ve repeatedly said, there is no one defined methodology that will work for every Agile project, and self-organizing teams are not, in and of themselves, sufficient to handle all the problems and issues that are likely to come up. Strong and knowledgeable management is needed.
- Refusing to reuse anything. Although many in the developer community oppose the idea of reuse, it should be an accepted principle that anything you can reuse to deliver value more quickly should be reused. This includes models (such as data models), data structures, data, code modules, services, and APIs.
- Ignoring non-functional requirements. Agile teams have a tendency to focus exclusively on functional requirements, as documented in the user stories and use cases. They tend to ignore non-functional considerations, such as quality, scalability, security, and maintainability. But these considerations aren’t unimportant, and they shouldn’t be ignored. They may need to be time-boxed, but they shouldn’t be ignored. Eventually, the failure to address these requirements will surface as very expensive problems.
- Insisting on a rigid methodology or approach regardless of what type of project it is, or what sort of product is being developed. As Dave Thomas remarked, Agile is not a formal methodology; it’s a way of thinking about how a given set of work should be done (“We don’t do Agile Development; we do Development Agilely!”). Cliff Berg et al put it this way: “One cannot ‘follow’ Agile ideas; one can only learn them and consider them, because Agile ideas are philosophical. They do not tell you what to do.”[v]
- A single-minded focus on the requirements of a single project, to the exclusion of larger considerations of organizational needs and values. This includes ethical considerations (such as creating a product that causes harm to customers or the general public), organizational considerations (such as violating your company’s stated values or standards, or bypassing inconvenient layers of management), and business considerations (such as generating data that can’t be used later for organizational reporting and analytics, or ignoring enterprise architecture standards that end up costing the company money).
It’s important to understand that there is not — and never has been — any prescriptive approach to software development that works for all software projects. Even software engineering allows (and always has allowed) for iterative development. Roger Pressman, in his seminal 1992 textbook on software engineering, notes that multiple approaches to software development are necessary.[vi] Applications whose requirements can be more-or-less fully specified at the beginning can use some variant of the traditional waterfall approach, while those whose requirements are continually evolving will take the iterative path. In some cases, prototyping will be used as a communication tool to help define the requirements; development may then proceed according to the standard life cycle, but this process may repeat itself over various iterations of the software. As Pressman puts it, “There is no need to be dogmatic about the choice of paradigms for software engineering. The nature of the application should dictate the approach to be taken. By combining approaches, the whole can be greater than the sum of the parts.”
It’s also important to understand that, as Ian Sommerville notes, much of the burdensome process and documentation associated with the so-called Waterfall Method exists for the purpose of reassuring managers that progress is being made, and that constraints of time and budget are being met.[vii] As mentioned earlier, it’s not the developers’ money that’s being spent. The approach taken to software development at any given company is going to depend substantially on what sort of project management and approval processes need to be satisfied.
As long as the relevant questions regarding opportunity and risk have been asked and answered, and the relevant stakeholders (especially the customer!) are on board, it should be fairly easy to find a work management approach (ideally including the Agile characteristics listed above) that will work for any particular project.
In
the next article, we will examine some lessons we should learn from Agile,
before moving to the question of Agile’s future.
[i] Thomas, Dave. “Agile is Dead (Long Live Agility).” Blog post published March 4, 2014. pragdave.me/thoughts/active/2014-03-04-time-to-kill-agile.html
[ii] Much of the following material is taken from Chapter 9 of my book Data Model Storytelling, which in turn is taken from my half-day Agile Data workshops.
[iii] Burns, Larry. Data Model Storytelling (Technics Publications, 2021).
[iv] It should be mentioned that the amount of up-front design required is dependent, to some extent, on the skill and knowledge level of the developer. A skilled landscaper, for example, will probably not need to do as much up-front design work as an unskilled one, since a skilled landscaper will have the knowledge and experience to know intuitively what risks need to be avoided, and will not make as many mistakes.
[v] Berg, Cliff et al. Agile 2 – The Next Iteration of Agile. John Wiley and Sons, 2021, p. 248.
[vi] Pressman, Roger S. Software Engineering: A Practitioners Approach, 3rd Edition. McGraw-Hill, 1992, pp. 33-34.
[vii] Sommerville, Ian. Software Engineering, 4th Edition. Addison-Wesley, 1992, p. 12.
