In the first article, I laid out the basic premise for this series: an examination of how Agile has gone from the darling of the application development community to a virtual pariah that nobody wants to be associated with, and an exploration of the very important question of what we should replace it with.
We started with a brief history of Agile, in an attempt to understand how it came about and what problems it was trying to solve. For years, I’ve been promulgating what I refer to as The First Law of Systems Analysis: In order to change anything, you first have to understand how it got that way.
In this article, we’ll look at the current state of Agile, and examine some of the critiques that have surfaced against it.
Critiques of Agile
By 2005, when I started doing data and database development work on Agile projects at my global, Fortune 200 manufacturing company, I had already begun to recognize that there were some issues with the Agile approach. I talked about many of these issues in my book “Building the Agile Database.”[i] Here are some of the problems I identified:
- Ignoring Stakeholders – The Agile approach was almost entirely geared towards software developers, especially Extreme Programmers. There was almost no attention given to any other project stakeholders (such as data managers, enterprise architects, and company management). Agile addresses the concerns of software development projects; it doesn’t address organizational needs outside the scope of projects or provide any help to people who have to work in more enterprise-facing roles. But it’s a mistake to focus solely on project stakeholders to the exclusion of everybody else. As I said in my book, it’s not the developers’ money that’s being spent on projects! This is why non-Agile processes and deliverables, such as project tollgates, estimates, burndown charts, and documentation, are often layered on top of Agile methodologies[ii] — to satisfy the needs of the non-project stakeholders who are funding or supporting the project.
- Ignoring Architecture and Design – Considerations of architecture and design are often given short shrift in Agile projects, emphasizing coding, testing, and delivery to the exclusion of almost everything else. Agilist Craig Larman famously said, “Architecture is a bad metaphor. We don’t construct our software like a building, we grow it like a garden.” But, as I pointed out in my book “Growing Business Intelligence,” even gardeners and landscapers have to do architecture and design![iii] You don’t just run out to the backyard with a shovel and start digging. You don’t site an ornamental pond at the lowest part of your property. You don’t place a patio or deck where it gets hot sun and prevailing winds in the summer. You don’t put hardscape (paths, walls, raised beds, gazebos, etc.) in before laying plumbing and electrical wiring. And you don’t just stick plants, shrubs, and trees wherever you like, with no consideration for whether they add to or detract from the landscape, whether they are “fit for” the intended purpose, or whether they will even survive in their given location. Agile should be about minimizing waste and rework, and ignoring considerations of architecture and design is a good way of maximizing both.
- Ignoring Waste and Rework – Agile development fails to emphasize several things that can reduce waste and rework on projects. Aside from ignoring the importance of architecture and design, Agile also gives short shrift to the idea of reuse (including the reuse of data, data structures, application components, and services). And yet, an effective reuse strategy can be a key component in achieving organizational agility. Agile also (in my opinion) fails to emphasize the importance of designing loosely coupled systems and rule-based (i.e., parameter-driven) components. There used to be a weekly magazine for software developers that featured two Extreme Programmers who, each week, went through numerous iterations of changing one component only to have another component break. Then, they would fix that component, and another would break. And so on and so forth. And this was being held up to developers as the ideal way to do software development!
- Ignoring Data Management Issues – There is almost nothing in Agile literature that addresses data issues or problems. There is no acknowledgment of the fact that organizations have data and information needs and requirements that lie outside the context of individual projects. Agile projects tend to just shove data, willy-nilly, into “persistence containers,” without regard for data design, business data rules, or data quality, and without regard for how (or whether) this application data can be used by downstream business processes.
- Ignoring Resource Constraints – Agile processes tend to work best when resources are 100% dedicated to a single project. This is easier to do for application developers and project managers, who generally only work on one or two projects at a time, than it is for people in the rest of the organization. Data managers, for example, have to support multiple projects concurrently, while helping the rest of the organization with its data needs and problems. Similarly, business people have to spend their time running the business; their ability to commit time to an application development project is severely constrained. I know one business manager who was fired from his job after supporting an Agile project; it was felt that he spent too much time on the project and not enough time doing his “real” job!
- Ignoring Specialization – Agile insists that people on Agile projects be “generalists” rather than specialists, able to do whatever sort of work needs to be done on a project. But, in my experience, it’s almost never the case that anyone on a project has any real expertise outside of their particular skill set. Most developers, for example, know very little about data management, data modeling, or even database design and programming. On almost every Agile project I’ve worked on, I was the principal (and often the only) resource doing data and database work. Similarly, how many devs really understand the complexities of QA testing? Or enterprise architecture? Or project management? Or application support and maintenance?[iv]
- Ignoring Behavioral Issues – Agile also requires a significant behavioral mind-shift in team members, which isn’t very well understood or discussed. Continual iterations of development with constantly changing requirements, constant face-to-face discussions, negotiations, and collaborations with team members and business stakeholders, the need for continuous learning, and lots of long hours and stress! People used to working in more controlled and self-contained environments may have a difficult time adjusting to these new demands. In “Building the Agile Database,” I talk about the importance of having what I call an “Agile Attitude.”[v] This can best be described as a positive, proactive, “can do” customer-service mindset, characterized by what I call the “3 C’s”: Commitment, Cooperation, and Communication. It’s important for Agile team members to recognize the value contributed by each other member, and to make sure the team understands their value as well. Agile requires excellent interpersonal and communication skills, which a lot of IT folk don’t have! Agile also requires a submersion of individual egos to the needs of the group, which can be hard for certain types of people.
- Ignoring “Time to Money” – Agile emphasizes the importance of reducing “time to market’; that is, the amount of time required to get new application functionality to the customers. However, this ignores what Susan Kunz, in the “Software Development Times,” calls “time to money”, the time it takes for an application to achieve “sustainable positive economic returns” to the company.[vi] This means that Agile developers need to keep in mind considerations of quality, scalability, reliability, maintainability, security, and reusability, which are not usually emphasized in Agile literature.[vii] Agile, for example, promotes a “prototyping” approach to software development over formal requirements definition. However, Roger Pressman has noted two problems with this approach: First, there is a tendency for systems “held together with chewing gum and baling wire” to remain permanently in production, because business users balk at the cost of reworking the application to make it reliable and maintainable. Second, the prototype may embody a choice of architecture or technology that is either inappropriate or not supported by the organization and, over time, the reasons for those choices (and the reasons they were inappropriate) may be forgotten.[viii] Agile has fallen into many of the same errors of thinking associated with Waterfall-style methodologies: Agile teams have become external to the creation of business value. They wait to get direction from the business, they gather and analyze business requirements, they write functional code, and they send the results back to the business. What’s missing here is the whole idea of front-line value creation. The business isn’t interested in applications; the business is concerned with value!
Let me emphasize that these are issues with Agile that I identified and wrote about between 2005 and 2010. These are not new issues, but they were issues that had gone pretty much unnoticed by the rest of the Agile community at that time. Since then, as I’ve noted above, there have been numerous other critiques of Agile, many of them by longtime Agile practitioners and advocates. Here are some of their criticisms:
- Agile Has Become Big Business – As Dave Thomas observed, once Agile became popular, it became a magnet for anybody wanting to make a fast buck. Books were written, consultancies popped up like mushrooms, conferences were held all over the world, and formal methodologies like SCRUM, SAFe, and Kanban became mandated in organizations and developed their own practices, processes, and certifications. According to Thomas, “Having conferences about agility is not too far from having conferences about ballet dancing, and forming industry groups around the four values always struck me as creating a trade union for people that breathe.” Thomas continues: “The word Agile has been subverted to the point where it is effectively meaningless, and what passes for an Agile community seems to be largely an arena for consultants and vendors to hawk services and products.” Ron Jeffries says, “Agile has become big business,” and notes that Agile principles have been subverted by formal methodologies such as SCRUM and SAFe. Kurt Cagle noted that methodologies like SCRUM had become “religions,” demanding unquestioning obedience from their adherents and making little if any sense to outsiders.[ix] In a talk at Agile Australia in 2018, Martin Fowler observed that Agile had been taken over by people who, by and large, weren’t even software developers (marketers, project leaders, product managers, team coaches, etc. etc.). As Cliff Berg et al put it: “The Agile conversation was essentially taken away from software developers — the people for whom Agile was created.”[x]
- Agile Has Become Insular and Self-Referencing – Cliff Berg et al warn that the Agile community has become extremely insular and self-constrained, only tolerating certain ideas and practices and largely ignoring ideas from other disciplines such as behavioral theory or organizational psychology, “The Agile community suffered from groupthink: the community mostly spoke to itself.”[xi] The Agile 2 movement emphasizes ideas and practices around leadership theory, systems theory, behavioral theory, and product management (as we will see, my own approach to Agile draws on ideas from human-centered design (design theory), cognitive behavioral therapy, and stakeholder economics). In other words, Agile has stunted itself by not being open to new ideas from other disciplines.
- Agile Isn’t Suited to All Projects – Kurt Cagle and others have mentioned that when Agile started, software development projects had characteristics that were more suited to an Agile approach: they were shorter (between 4-6 months), were mostly web-based or mobile, emphasized user interfaces, were client-facing, and were worked on by small teams (fewer than a dozen people). They involved products that would be useful and functional even if they weren’t complete. And these projects didn’t require a ton of leadership or management. However, the software product world has changed. Much software is now enormously complex and includes a morass of third-party products and licensed services and platforms. It is harder to construct — and demonstrate — a “minimally viable product,” and much harder to deconstruct such a product into executable tasks that can be reliably estimated. Much of the development work is not client-facing. Cagle notes that most of these types of projects would actually benefit from a Waterfall approach. Agile methods are simply not suited to all types of projects. My last project was a four-year long rewrite of a number of mainframe-based legacy systems, migrated to a heavily customized vendor product. It involved three development teams (two offshore), five database management systems, and heavy management involvement. And yet, it was treated like an Agile project, with daily standups, embedded business representatives, sprints, retrospectives, etc. etc. I’ll leave you to guess how “Agile” that project actually was!
- Agile Ignores Leadership Issues – Miriam Posner notes that the original authors of the “Agile Manifesto” were “organizational anarchists,” pushing back on what they perceived as attempts by Management to impose excessive controls on the software development process (including Software Engineering and the Waterfall methodology). The attitude of Agile developers towards Management was something straight out of a “Dilbert” cartoon strip. But as I’ve already noted, the purpose of software projects is to create value for an organization, which means that the organization’s management are legitimate Stakeholders in this endeavor. As I said, it’s not the developers’ money that’s being spent. The “Agile 2” Alliance (Cliff Berg et al) decries the tendency of Agile to try to cut managers out of the loop in favor of “self-organizing teams” and “servant leadership”. Instead, they stress the importance of Leadership in organizations, calling it “The Core Issue.”[xii] While Scott Adams, the Dilbert cartoonist, mocked the very idea of Leadership, Berg asserts that nothing can really be accomplished in any organization without wise and firm leadership. Berg raises several questions about self-organizing teams, including their tendency to become dominated by the opinions of young, mostly white males and their tendency to select as leaders people who look and act like leaders (self-confident, charismatic, extroverted), but who don’t have real leadership abilities.[xiii] Posner notes that aspects of Agile, such as Pair Programming, only work when the people involved are equal in status and comfortable with each other, and that the dynamic changes considerate when issues of race or gender are introduced. Says Posner, “We’ve long known that eliminating bureaucracy, hierarchy, and documentation feels great, until you’re the person who needs rules for protection.”
- Agile Can Mean Almost Anything – Both Posner and Berg emphasize the “wishy-washy” nature of the “Agile Manifesto.” Since it is nothing more than a set of values (or “mindsets”), people can impose any practices on top of them, call these practices “Agile,” and insist that these practices (such as SCRUM, SAFe, and Kanban) be followed to the letter. As Posner puts it, “The ubiquity of Agile means that it is just as likely to be imposed from above as demanded from below.” One of “Agile 2”’s most important tenets, according to Berg, is that it “provides the foundations of thought from which it is derived,” rather than simply mouthing platitudes that can be interpreted any number of ways by any number of people.
- Agile Ignores Ethical Issues – Another issue raised by Miriam Posner is the fact that the essence of Agile is “developers working single-mindedly toward a goal”, and this, in turn, lessens the likelihood that developers will be willing –– or able –– to question either the goals of the project or the means being used to achieve them. Posner notes several ethical issues around Agile, including the frequent use of the “Move Fast and Break Things” ethos, where possible adverse consequences of technology to the general public are ignored or laughed off.[xiv] She also notes that Agile’s approach of breaking a product down into user stories and tasks enables teams to avoid taking responsibility for the ethical or social impacts of the product (what Yvonne Lam calls “a chain of deniability”). Agile teams, in their single-minded push to fulfill requirements given them by the business, seldom question whether these requirements support sensible or ethical business goals (e.g., Wells Fargo’s practice of opening new accounts for their customers without their knowledge and charging them fees for these accounts). Agile teams may be democratic and ethical; organizations almost never are.
- Agile Isn’t a Discipline – This leads to probably the most important question raised by Miriam Posner in her article: What sort of discipline, exactly, is Agile development? The digital theorist Ian Bogost argues that this move-fast-and-break-things approach is precisely why software developers should stop calling themselves “engineers”: Engineering, he points out, is a set of disciplines with codes of ethics and recognized commitments to civil society. Posner remarks, “Agile promises no such loyalty, except to the product under construction.” A freelance developer quoted in her article even argues that “As developers, IT professionals, we like to think of ourselves as knowledge workers, whose work can’t be rationalized or commodified. But I think Agile tries to accomplish the exact opposite approach.” But if Agile development can’t be “rationalized,” then how can it be considered an engineering discipline, or any discipline at all? Developers may choose to insist on their right to simply have fun playing with cool toys, but you can’t call this a “profession.”
This brings us to an important question: If Agile now is not what it is supposed to be, then what is Agile, really? What is the true essence of Agile, and how can it be rescued from the formalists, the religionists, and the hucksters? We’ll address this question in the next article in this series.
[i] Burns, Larry. Building the Agile Database (Technics Publications, 2011), pp. 49-60.
[ii] Often referred to as “Scrummerfall”: “The practice of combining Scrum and Waterfall so as to ensure failure at a much faster rate than you had with Waterfall alone.”
[iii] Burns, Larry. Growing Business Intelligence (Technics Publications, 2016), pp. 7-9.
[iv] One of my biggest pet peeves is that, in most organizations, applications are handed off to support teams after deployment, and those teams are almost never involved in (or consulted by) the development project.
[v] Ibid, Chapter 11, pp. 239-245.
[vi] Kunz, Susan. “Forget Time-to-Market; It’s All About Time-to-Money”. Software Development Times, August 15, 2006.
[vii] Cliff Berg notes in his Agile 2 book (p. 200) that a survey of open-source software developers revealed that most feel that “securing their code is a soul-withering waste of time.” This again raises the question of whether software development, especially Agile development, can be considered any sort of engineering practice or discipline.
[viii] Pressman, Roger S. Software Engineering: A Practitioner’s Approach, 3rd Edition. McGraw-Hill, 1992. Page 28.
[ix] The British Computer Society, in a 10-year retrospective on Agile, coined the term Scrumdamentalism to describe the religious fervor around some Agile practices. slideshare.net/jcasal1/20120419-agile-businessconferencepptx, slide 11.
[x] Agile 2, op. cit., page 8.
[xi] Ibid, page 9.
[xii] Agile 2, op. cit., Chapter 3.
[xiii] Cliff Berg notes that all of the authors of the Agile Manifesto were White, middle-aged, middle-class English-speaking professionals. Diversity and inclusion have never been hallmarks of Agile.
[xiv] As I’ve frequently said, the “Move Fast and Break Things” tech bro ethos only works if two conditions are met: First, that the cost of “things breaking” is minimal, and second, that these costs are not borne by critical stakeholders such as customers, employees, or the general public.
