Disciplined Agile Data Management

The Disciplined Agile (DA) process decision framework provides light-weight guidance to help organizations streamline their information technology (IT) and business processes in a context-sensitive manner. It does this by showing how the various activities such as solution delivery, operations, enterprise architecture, portfolio management, finance, security, procurement, data management, and many others work together. The framework also describes what these activities should address, provides a range of options for doing so, and describes the trade-offs associated with each option. In effect, DA provides the process foundation for business agility.

In A Disciplined Agile Mindset for Data Management you learned what “being agile” meant for data management.  In this article we overview what “doing agile” means when taking a DA approach to data management.  We start by examining how data management fits in with other aspects of your IT efforts and then overview common data management activities.  The article ends with a summary of the issues that you’ll face with transitioning to an agile way of working.

How Data Management Fits In

Key tenets of agile and lean are to work collaboratively and to streamline your workflow respectively. In Figure 1, we show that data management is a collaborative effort that has interdependencies with several other DA process blades (sometimes called process areas) and the solution delivery teams that data management is meant to support.

Figure 1.

This can be very different than the current traditional strategies.  For example, with a DA approach, the data management team works collaboratively with the delivery teams, operations, and release management to evolve data sources. The delivery teams do the majority of the work to develop and evolve the data sources, with support and guidance coming from data management. The delivery teams follow guidance from release management to add the database changes into their automated deployment scripts, getting help from operations if needed to resolve any operational challenges. Evolution of data sources is a key aspect of Disciplined DevOps [1].  This is very different than the typical traditional strategy that requires delivery teams to first document potential database updates, have the updates reviewed by data management, then do the work to implement the updates, then have this work reviewed and accepted, then work through your organizations release management process to deploy into production.

A Disciplined Agile Approach to Data Management

Now let’s drill down and see what the workflow for a DA approach to data management looks like. First, notice how all of the activities depicted in Figure 2 are collaborative in nature.  This is shown via the additional roles beside the activities or interacting with them. Second, how you address these activities will vary depending on the situation that you face. Our goal here, is to explore a baseline from which you can potentially start, but you’ll need to tailor it to address your actual situation.

Figure 2. Internal Workflow for Data Management (Click to expand in a new tab).

Let’s work through each activity one at a time:

1. Evolve organizational data artifacts. Organizational data management artifacts may include data models, including but not limited to a high-level conceptual model for your enterprise (typically a view within your enterprise architecture); metadata describing common concepts, entity types, and data elements within your organization; master data for critical entity types; and master test data to support database testing across multiple delivery teams. Data managers will work closely with Product Managers to understand their overall vision for their products and the organization as a whole to ensure that their data management strategy aligns with your business roadmap. Data managers will also work closely with enterprise architects to ensure that data concerns are addressed appropriately in your organization’s architecture and that your data management strategy aligns with your technology roadmap. These collaborations are often accomplished through regular working sessions that are often called in an as-needed, impromptu manner.
2. Enable delivery teams. Data managers work closely with delivery teams to train, educate, and coach them in data skills. The overall strategy is to enable delivery teams to be as self-sustaining as possible when it comes to data-related activities, to offload as much of the grunt work as possible to enable the teams to become more reactive and to allow data managers to focus on value-added activities such as evolving organizational data artifacts and guidance. The implication is that data managers will need to develop and maintain a training program around fundamental data skills (computer-based training often proves sufficient for this) such as data modeling, database design, and data security. They will also need coaching skills so that they can work side by side with delivery teams to help them to learn these critical skills.
3. Support deliver teams. Delivery teams will need help from time to time to address hard database design problems, to gain access to and to understand legacy data sources, and to obtain and/or generate test data. The DA strategy is for data managers to work collaboratively with the delivery teams to do so, to get directly involved with the teams to do the actual work (and to transfer skills while doing so). In a pragmatic take on the sage advice around teaching a man to fish, the goal should be to teach the delivery team how to fish, but while doing so provide enough fish to sustain them until they become self sufficient.
4. Evolve and support data guidance. Delivery teams should follow your organizational conventions around data (and around security, and user experience, and so on). The data management team is the source of this data guidance, which should address fundamental issues such as data naming conventions, data security conventions, and your data architecture and design patterns. This guidance should be developed and evolved collaboratively with the delivery teams themselves to ensure that the guidance is understandable, pragmatic, and accepted by the teams.
5. Support and monitor operations.  Data managers will work closely with operations managers to monitor your existing production data sources (operations managers monitor far more than just data sources of course). Ideally this monitoring is fully automated with dashboard technology used to render critical operational intelligence in real time. Note that operational database administration activities are addressed by the operations process blade [2].
6. Improve data quality. Data managers will guide and collaborate in the data quality improvement efforts of your database administrators (DBAs) and operations engineers as well as your delivery teams. They will monitor your automated database regression testing efforts (ideally a continuous effort) and your ongoing data source evolution efforts (implemented as database refactorings) that occur on a daily basis. Your data managers will oversee the long-term aspects of database refactoring, in particular the retirement of deprecated database schema and the scaffolding required during the deprecation periods for the appropriate refactorings [3].

How to Succeed

We’ve found that the following strategies are critical to your success when adopting a DA approach to data management:

1. Surface your challenges. You need to have an honest conversation about the effectiveness of your current approach to data management. This conversation must be driven from an organizational viewpoint so as to take into account stakeholder needs.  We’ve found that a very effective way to do this is value stream mapping (VSM) sessions to reveal the true efficiency and quality levels of your critical data management processes [4]. This will not be pleasant for anyone who still believes in traditional data management practices, but it is vital to your success that everyone recognizes that you need to improve.
2. Expect better.  One of the reasons why the data management field has languished for as long as it has is because the rest of the IT community allowed them to. For the most part we accepted the claim that data managers needed to work in a slow and onerous manner, that eventually they’d address our organization’s data quality challenges through some form of bureaucratic magic that only they understood.  Enough is enough.
3. Invest in your staff. Traditional data professionals tend to be overly specialized, often focusing on one aspect of data management such as logical data modeling, meta data management, data traceability, and so on. Not only does this result in bureaucratic, drawn-out processes but many of these specialties are no longer required when you’ve adopted pragmatic, quality-focused agile strategies.  To be effective you need T-skilled generalizing specialists [5], and that requires you to invest in training and long-term coaching to help people to modernize their skillset.
4. Hire agile coaches with deep experience in both Agile and data management. Someone with agile coaching experience alone will struggle to gain the trust of experienced data management people, and a data management coach without deep agile experience will struggle to help people to overcome their deep-rooted traditional belief system.  The bad news is that agile data management coaches are very hard to find right now due to high demand and low supply.

Improving your data management processes and organization structure to support a Disciplined Agile way of working is a daunting task. Along with evolving your governance strategy, it is likely the hardest part of any organizational transformation, but it is one that you cannot ignore. Effective data management is critical to your success as an organization.

This article has been excerpted from the forthcoming book tentatively titled The Disciplined Agile Handbook.

References

1. Disciplined DevOps.  DisciplinedAgileDelivery.com/disciplineddevops/
2. IT Operations. DisciplinedAgileDelivery.com/agility-at-scale/it-operations/
3. Refactoring Databases: Evolutionary Database Design. Scott W. Ambler and Pramod J. Sadalage, 2006, Addison Wesley.
4. Value Stream Mapping: How to Visualize Work and Align Leadership for Organizational Transformation. Karen Martin and Mike Osterling, 2014, McGraw-Hill Education.
5. Generalizing Specialists: Improving Your IT Career Skills.  AgileModeling.com/essays/generalizingSpecialists.htm

About Mark Lines

Mark Lines is Managing Partner at Scott Ambler + Associates. He is an Enterprise Agile Coach and co-creator of the Disciplined Agile Delivery framework. Mark is co-author with Scott Ambler of The Executive Guide to Disciplined Agile: Winning the Race to Business Agility, Disciplined Agile Delivery:  A Practitioner’s Guide to Agile Software Delivery in the Enterprise, and an Introduction to Disciplined Agile Delivery. Mark helps clients around the world transform from traditional to lean and agile enterprises. He is a frequent speaker at industry conferences and blogs about Disciplined Agile at DisciplinedAgileDelivery.com. He is also a founding member and President of the Disciplined Agile Consortium (DAC), the certification body for Disciplined Agile.