In my first article, I laid out the basic premise for this series: an examination of how Agile has gone from the darling of the application development community to a virtual pariah that nobody wants to be associated with, and an exploration of the very important question of what we should replace it with.
We started with a brief history of Agile, in an attempt to understand how it came about and what problems it was trying to solve. For years, I’ve been promulgating what I refer to as The First Law of Systems Analysis: In order to change anything, you first have to understand how it got that way.
In the second article, I enumerated a number of the criticisms that have been leveled against Agile, including many from my own writings.
In the third article, I tried to distill Agile down to a fundamental set of principles that can be applied to all projects, regardless of the type of project or the methodology being used.
In this article, we’ll examine what lessons we should learn from the Agile experiment, as we prepare to discuss what comes next.
Lessons From Agile
In spite of the numerous critiques of Agile that have been mentioned previously, I believe that Agile (and its failures) has taught us a number of valuable lessons that developers and other project stakeholders should keep in mind, regardless of what the future may hold for project management and application development. Here are some of the lessons I’ve learned over the years from my involvement with Agile project teams:
1. You Need to Be “Right Here”
When I started working in IT, I worked in a cubicle, working on stuff by myself. Periodically, somebody would throw a request over the cubicle wall. Whenever I could get to it (or felt like it), I would do the requested work and give it back to the requestor. I was an example of what I’ve come to call an “Over There” worker, someone to whom tasks are given asynchronously and fulfilled asynchronously. What I’ve come to realize is that “Over There” workers have limited value to organizations and are often the first notch when the organization needs to tighten its belt. Working on Agile teams has taught me the importance of being what I call a “Right Here” worker, on the front lines of wherever value is being created in the organization, available at a moment’s notice whenever an opportunity to help ensure the success of value creators in the company arises.
Another way of putting this, as Cliff Berg et al do in their “Agile 2” book, is that Agile emphasizes collaboration rather than “handoffs.”[i]
2. Projects Need a Holistic Approach
The traditional software engineering approach to projects that I was taught in school was what I might describe as a requirements-centered approach. Software project methodology consisted of specifying the requirements, getting the requirements approved by the customer, breaking the requirements down into executable tasks, and so on. When you think about it, most approaches to Agile development (such as SCRUM) follow exactly the same approach, defining requirements, turning them into user stories and creating the backlog, estimating the stories and moving them into Sprints, decomposing each user story into executable tasks, etc. This all changed for me when I discovered human-centered design (HCD), an approach to problem-solving that emphasizes stakeholder involvement, brainstorming, collaboration, iteration, and prototyping (more about HCD later!). Most importantly, human-centered design emphasizes a holistic (or systems) approach, one that takes into consideration the context in which the problem being solved exists. A software project needs to be seen as more than just the creation of a product or process; it needs to be seen as the opportunity to create value for an organization, in ways that may lie outside a set of product requirements. For example, what’s needed may be a reworking of an existing business process, or the creation of a business initiative, or a rethinking of how a company interacts with customers, suppliers, or other business stakeholders.[ii]
3. Non-Functional Requirements Matter
Cliff Berg and many others have criticized Agile practices for not regarding Data issues (such as data quality) as important. Data and Information are increasingly regarded as strategic assets in most organizations. But Agile’s disregard of data issues is only one symptom of a much larger problem: Agile, like almost every other software development methodology (including Waterfall) focuses almost exclusively on functional requirements! In the case of Agile, only the work defined in functional use cases or user stories is considered important, and the team is completely focused on achieving the successful completion of those stories. But non-functional requirements — such as robustness, scalability, quality, security, and maintainability — are important too. This goes back to the previous point about “Time to Money.” An application that merely meets the functional requirements, but is slow or difficult for customers to use, or difficult or expensive to maintain, or impossible to easily adapt to changing requirements or technologies, or that violates user privacy regulations, or that generates data that is unusable for other organizational needs, may actually end up costing a company more money than it saves.[iii]
4. Leadership Is Important
Cliff Berg et al have emphasized the importance of leadership in their Agile 2 initiative, and I whole-heartedly agree. I have never worked on a project of any size or significance where management leadership was not instrumental in the success of the project. This is mostly because a company’s management is able to see the Big Picture and has perspective into how a given project (or product) fits into an organization’s overall value proposition. They also understand the organizational politics that have to be navigated in order to successfully approve and complete a project. Here’s an example: In one project I worked on, the developers made a very ill-advised decision to use Vehicle Identification Number (VIN) as the primary key in a database table (over my strenuous objections, of course!). The project came to a screeching halt when the customers demanded the ability to change the VIN number in the application. It took numerous meetings at very high levels of company management to hammer out a workable solution to the problem the developers had created!
5. Different Projects Need Different Approaches
There is no one-size-fits-all approach to software development that will work for every project. As explained above, the approach taken should be based on a number of criteria, including the type of product being created, whether or not the product is customer-facing, the estimated cost and length of the project, the cost and difficulty of making changes after construction starts, the cost and difficulty of maintaining the product over time, the stability and familiarity of the technology being used, the cost and risk of project failure, and so on.[iv] Each project should begin with an assessment of the type of project (and product) involved, and the development approach should be chosen accordingly.
6. Behavioral Issues Can’t Be Ignored
One consequence of Agile’s insistence on bringing people together to work in project teams for long periods of time is that personnel issues become a much more critical factor in the success (or failure) of projects. In the olden days, people who worked mostly in offices or cubicles could have difficult personality issues and still be reasonably productive. Now, with everybody working together cheek-by-jowl, personality issues can be extremely disruptive to projects. On one project I worked on, the lead network architect quit in the middle of the project; he was a minority who felt disrespected by our (mostly white) management. Another co-worker (also a minority) would explode in fits of rage whenever he felt his input was being ignored. I myself have had to work very hard over the years to overcome my natural introvert tendencies so that I could work effectively with team members and other stakeholders. Agile is not a natural fit, personality-wise, for a number of people, especially people in IT. This is one reason to be skeptical of the Agile Alliance’s insistence on self-organizing teams. A number of Agile critics have made the point that Agile team configurations (especially XP’s Pair Programming) only work if the people involved are of equal social status, are largely extroverted, and are comfortable with one another. Agile teams work less well when people of differing backgrounds, social status, and personality types are thrown together. This is why effective leadership is even more important on Agile projects than on Waterfall projects.
7. An “Agile Attitude” Is Important
Because of the nature of Agile and the possibility for personality conflicts between team members and other project stakeholders, it’s important for every team member to have tools at their disposal to keep the development process running smoothly. Agile team members need to have a positive, proactive, “can-do” customer service mentality, and they must be committed to the success of both the team and the project. This is where the Agile analogy of the Pigs and the Chickens comes from: Chickens only contribute their eggs to a breakfast; the Pigs give their all. My book “Building the Agile Database” devotes an entire chapter to the importance of having an Agile Attitude, including practicing what I refer to as the “3 C’s”: Commitment, Cooperation, and Communication. One Agile concept that is almost never talked about is the importance of always assuming that people are well-meaning, that they are always trying to do their best in spite of obstacles and failures.
8. Automation Is a Necessity
Cliff Berg notes that when people like Jez Humble and Gene Kim first promulgated the principles of continuous integration and continuous delivery (CI/CD), their work was mostly ignored by Agile practitioners. Later, Agilists tried to claim that DevOps was nothing more than XP in a different form (not true). But the importance of automation (and reuse) in Agile projects cannot be understated. It’s impossible to be truly “Agile” if you are reinventing the wheel on every single project! Throughout my career, I’ve worked continuously to automate my Data and Database development processes, creating custom-built tools and scripts for exporting data models into various DBMSs, generating customized database schemas, and implementing database processes such as archiving, FSP generation,[v] change data capture, database auditing, service event monitoring, KPI generation, and multi-language text support. These tools have saved countless thousands of hours of development time.
9. Value Creation Should Be Emphasized
Two principles of Agile are that business stakeholders should be continually involved with a project, and that team members should be “generalists” rather than specialists. It’s not possible for everybody to know everything, but team members should have at least a general understanding and interest in the work being done by other team members. To my mind, what this means is that the entire team needs to have an understanding of what sort of value is being created by the project, and what each team member’s role in this value creation process is. Everybody needs to understand what the success criteria for the project is, what they themselves need to do to ensure that success, what all the other team members need to do to ensure that success, and how each team member can contribute to the success of each other team member. Cliff Berg et al make the point that Agile’s insistence on breaking down requirements into executable tasks means that team members are focused only on their individual tasks, and not on end-to-end value creation.[vi] But if business value isn’t being created, it doesn’t matter what product is being produced, or how much work is being done, or what the team velocity is. Contrary to Agile, the measure of project success is not working software, it’s delivery of business value!
10. Expertise Is Important
One of the unfortunate consequences of Agile’s insistence on self-governing teams of generalists is that the entire team feels the need to participate in every meeting and be involved in every decision. In every Agile project I’ve worked on, this has resulted in an enormous waste of time and money. It should be possible to identify the people most capable of making a particular decision or solving a particular problem, and then say, “You three go talk this over and then get back to the team with your recommendation.” While having general domain knowledge is important, it’s also important to know what strengths and competencies exist on any given team and leverage them for maximum value.
11. Don’t Fill Up All the Time
Continual delivery of business value is important. But every project, and every worker on every project, needs some time to learn, experiment, revise, improve, automate, regenerate, and — on occasion — fail and recover from failure. No project, especially not an Agile project, should resemble the Bataan Death March!
In the next article(s), we will evaluate some of the suggested replacements for Agile, keeping in mind the principles we’ve developed over the last two articles.
[i] Berg, Cliff et al. Agile 2 – The Next Iteration of Agile. John Wiley and Sons, 2021, pp. 70-73.
[ii] My readers will remember that I often use the example of American Hospital Supply (AHS). This company went from a traditional wholesale-supplier business model (using a warehouse and a fleet of trucks to ship medical supplies to hospitals and doctor’s offices) to a model in which AHS supports a web-based portal that allows its customers to order directly from its suppliers! AHS collects a monthly fee from its customers for access to the portal, along with a percentage of each customer order. The suppliers get more business, the customers get more choices and faster service, and AHS saves the cost of maintaining a warehouse and a fleet of trucks.
[iii] See, for example, my explanation of the economics of software development in Building the Agile Database, pp. 32-35.
[iv] Burns, Larry. Building the Agile Database (Technics Publications, 2011), p. 58.
[v] For readers unfamiliar with the term, FSPs (Fundamental Stored Procedures, also called CRUD procedures) are stored procedures that handle the work of inserting, updating and deleting records from database tables in a manner that ensures data integrity, including entity integrity, referential integrity and transactional integrity. The FSPs generated by my program also handle a number of other important features, including support for hierarchies and multi-language text support.
[vi] Berg, Cliff et al. op. cit., pp. 70-73.
