A Disciplined Agile Mindset for Data Management

There are very few organizations left these days that haven’t adopted agile and lean strategies within some – and very often the majority – of their software development teams. Furthermore, organizations are now applying agile and lean strategies not only across IT but across the business itself. This trend began in 2001 and has steadily grown since then. Yet, in many cases, the data groups within these organizations still cling to outmoded strategies that result in them becoming significant bottlenecks. We need to do better.

Publisher’s Note: This article is the first of a pair co-written by Scott Ambler and Mark Lines.

Data Management is very important to the success of your organization. Yet, data management isn’t addressed by agile methods such as Scrum, Extreme Programming (XP), LeSS, or Enterprise Scrum. Nor is data architecture, data governance, or even database development. Granted, if you squint a bit, then SAFe’s minimal discussion of enterprise architecture implies the need to address data architecture, so there’s that. There’s a myriad of reasons why these agile methods don’t address data issues to any great extent, including the cultural impedance mismatch [1], the data community’s glacially slow adoption of modern techniques, and a dearth of people working to bridge the agile and data communities.

Hope is not lost. Data Management is a key process component, what we call a process blade, in the Disciplined Agile (DA) framework [2, 3]. DA’s data management process blade promotes a pragmatic, streamlined approach to data management that fits into the rest of your IT processes – you need to optimize your Data Management strategy within the overall workflow of your organizational processes. You need to support the overall needs of your organization, producing real value for your stakeholders. Disciplined Agile data management does this in an evolutionary and collaborative manner, via concrete data management strategies that provide the right data at the right time to the right people.

According to the Data Management Body of Knowledge (DMBoK) [4], data management is “the development, execution and supervision of plans, policies, programs and practices that control, protect, deliver and enhance the value of data and information assets.”  Although this is good definition, unfortunately the implementation of Data Management strategies tends to be challenged in practice due to the traditional, documentation-heavy mindset. This mindset tends to result in onerous, bureaucratic strategies that, more often than not, fail to support the goals of your organization. In this article we explore the mindset required for data management professionals to be successful in an agile world.

Disciplined Agile Values for Data Management

There are several values that capture the mindset required for a leaner, more agile approach to data management. Taking a cue from the Agile Manifesto [5, 6], we’ve captured these values in the form of X over Y. While both X and Y are important, X proves to be far more important than Y in practice. These values are:

1. Evolution over definition. The ability to safely and quickly evolve an existing data source, either to extend it to support new functionality or to fix quality issues with it, is absolutely paramount in today’s hyper-competitive environment. Yes, having defined data models and metadata describing them is also important, but nowhere near as important as being able to react to new business opportunities. Luckily, agile database techniques that are long proven in practice exist, enabling the safe evolution of production data stores [7].
2. Holistic organization over data management. Earlier we said that data is the lifeblood of your organization. Yes, blood is important, but so is your skeleton, your muscles, your organs, and many other body parts. We need to optimize the whole organizational body, not just the “data blood.” Traditional data management approaches often run aground because they locally optimize for data concerns, whereas a DA approach to data management recognizes that we must optimize the overall whole. This implies that sometimes we may need to sub-optimize our strategy from a data point of view for the sake of organizational level optimization.
3. Sufficiency over perfection. Data sources, like many other IT assets, need to be good enough for the task at hand. The old saw “perfect is the enemy of good” clearly applies in the data realm – too much time has been lost, and opportunities squandered, while development teams were forced to wait on data management teams to create (near) perfect models before being allowed to move forward. Traditional data professionals mistakenly assume that production databases are difficult to evolve, and as a result strive to get their designs right the first time so as to avoid very painful database changes in the future. The Agile Data method has of course shown this assumption to be wrong – that it is very straightforward and desirable to evolve production databases. A side effect of this revelation is that we no longer need to strive for perfect, detailed models up front. Instead we can do just enough up-front thinking to get going in the right direction and then evolve our implementation (including data sources) over time as our understanding of our stakeholder needs evolve.
4. Collaboration over documentation. We’ve known for decades that the most effective way to communicate information is face-to-face around a shared sketching environment, and that the least effective is to provide detailed documentation to people. The implication is that we need to refocus our efforts to be more collaborative in nature. As data professionals we need to get actively involved with solution delivery teams: to share our knowledge and skills with those teams, and to enable them to become more effective in working with data. Yes, we will still need to develop and sustain data-related artifacts, but those artifacts should be lightweight and better yet executable in nature (see value #6 below).
5. Cross-functional people over specialized staff. Agilists have come to realize that people are more effective when they are cross-functional (also known as T-skilled or generalizing specialists) [8]. Although specialists are very skilled in a narrow aspect of the overall process, the problem is that you need a lot of specialists to perform anything of value and as a result the overall workflow tends to be error prone, slow, and expensive. The other extreme would be to be a generalist, someone who knows a little bit about all aspects of the overall process. But, the challenge with these people is that although they’re good at collaborating with others they don’t actually have the skills to produce concrete value. We need the best of both worlds – a generalizing specialist with one or more specialties so that they can add value AND a general knowledge so that they can collaborate effectively with others and streamline the overall effort.
6. Automation over manual processes. The only way that we can respond quickly to marketplace opportunities is to automate as much of the bureaucracy as we possibly can. Instead of creating detailed models and documents and then reviewing potential changes against them, we capture our detailed specifications in the form of executable tests. This is quickly becoming the norm for specifying both the requirements and designs of code, and the same test-driven techniques are now being applied to data sources [9]. Continuous integration (CI) and continuous deployment (CD) are also being applied to data sources, contributing to improving overall data quality and decreasing the time to safely deploy database updates into production.

Data Management Can Be Agile

As you can see, we’re not talking about your grandfather’s approach to data management. As Figure 1 summarizes, organizations are now shifting from the slow and documentation-heavy bureaucratic strategies of traditional data management towards the collaborative, streamlined, and quality-driven agile/lean strategies that focus on enabling others rather than controlling them. This shift requires data professionals to adopt a more disciplined – a more agile – mindset, captured in the values above.

Figure 1. Shifting from bureaucracy to enablement.

Of course, it isn’t sufficient to just “be agile,” to just have an agile mindset. You also need to know how to “do agile” as well. In our next article, we explore the workflow of Disciplined Agile Data Management. Stay tuned!

This article written by Scott Ambler and Mark Lines has been excerpted from the forthcoming book tentatively titled The Disciplined Agile Handbook.

References

1. The Cultural Impedance Mismatch Between Data Professionals and Application Developers. Agiledata.org/essays/culturalImpedanceMismatch.html
2. An Executive’s Guide to Disciplined Agile: Winning the Race to Business Agility. Scott W. Ambler and Mark Lines, Disciplined Agile Consortium, 2017. http://DisciplinedAgileDelivery.com/exec-guide-to-da/
3. Disciplined Agile Home Page. DisciplinedAgileDelivery.com
4. DAMA Guide to the Data Management Body of Knowledge. Technicspub.com/dmbok/
5. The Agile Manifesto. AgileManifesto.org
6. Disciplined Agile Manifesto. DisciplinedAgileDelivery.com/disciplinedagilemanifesto/
7. Agile Data Home Page. AgileData.org
8. Generalizing Specialists: Improving Your IT Career Skills. AgileModeling.com/essays/generalizingSpecialists.htm
9. Test-Driven Database Development: Unlocking Agility. Max Guernsey III, Addison-Wesley Professional, 2013.

About Mark Lines

Mark Lines is Managing Partner at Scott Ambler + Associates. He is an Enterprise Agile Coach and co-creator of the Disciplined Agile Delivery framework. Mark is co-author with Scott Ambler of The Executive Guide to Disciplined Agile: Winning the Race to Business Agility, Disciplined Agile Delivery:  A Practitioner’s Guide to Agile Software Delivery in the Enterprise, and an Introduction to Disciplined Agile Delivery. Mark helps clients around the world transform from traditional to lean and agile enterprises. He is a frequent speaker at industry conferences and blogs about Disciplined Agile at DisciplinedAgileDelivery.com. He is also a founding member and President of the Disciplined Agile Consortium (DAC), the certification body for Disciplined Agile.