Lean Data Governance Strategies

The goal of data governance is to ensure the quality, availability, integrity, security, and usability within an organization. The way that you go about this is up to you. Many traditional approaches to data governance seem to struggle in practice; I suspect it is partly because of the cultural impedance mismatch, but also partly because traditional IT governance struggles in general. The bureaucratic command and control approach typical of traditional governance strategies is a lot like herding cats— you do a lot of work but not much gets accomplished in the long run because it’s far too easy for the cats to simply go around whatever you want them to do. Agile/lean governance, on the other hand, is focused on motivating people to do the “right things,” enabling them to do so, and then monitoring and guiding their efforts where needed.

The goal of agile/lean data governance is to enable development teams to maintain and develop high-quality data assets within your overall IT ecology. A lean data governance approach promotes a healthy, collaborative relationship between data professionals and the teams that they’re supporting. In the Disciplined Agile (DA) toolkit we have explicitly adopted proven strategies for lean governance both of development teams and the organization in general. The approach is based on the observation is that the most effective way to govern the actions of intellectual workers is through motivating and enabling them, not by a command-and-control process. In 2006, the Dr. Dobb’s Journal’s Current State of Data Quality Techniques survey which found that a collaborative approach data management is more effective than a command-and-control approach, which in turn is better than no approach at all. Unfortunately, traditional approaches to IT governance are often implemented in a command-and-control fashion.

This article is organized into the following topics:

1. Potential challenges with traditional data governance
2. Agile/lean data governance strategies
3. Data governance success factors

1. Potential Challenges with Traditional Data Governance Strategies

Traditional data governance strategies often suffer from one or more common problems:

1. Data governance doesn’t fit into the overall governance effort. In a way it’s sort of questionable to even consider data (development) governance, financial governance, or a myriad of other governance efforts outside the scope of the overall organizational governance strategy. You need to optimize the whole governance strategy, not just individual parts.
2. Data governance efforts are ignored. The 2006 DDJ survey into the current state of data management practices showed that 66% of development teams will choose to “work around” their organization’s data group. Since then the situation has improved, 2016 Current State of Data Quality survey found that this was happening in only 52% of organizations. Regardless, this is clearly problematic and an indication that your data governance efforts won’t succeed if you can’t find ways to collaborate effectively with development teams.
3. Traditional data governance is too onerous. When it is difficult to work with your data governance team, it can be motivating for people to avoid working with the team whenever possible. This can reduce the likelihood of them being compliant to your organizational conventions.
4. Traditional data governors are too slow to respond. The 2016 study found that 62% of development teams find that their data groups work too slowly and 46% find their data groups are too difficult to work with. This motivates developers to do what they believe is best (even though they might not actually know what the best course of action is).
5. Data teams aren’t perceived as providing value. The 2016 study also found that 46% of development teams find their data groups have little value to offer them.

2. Agile/Lean Data Governance Strategies

In addition to supporting proven agile database practices, the agile/lean development governance practices which you should consider adopting for your data governance efforts are:

1. Valued corporate assets. Guidance (such as database design conventions, modeling style guidelines, data naming conventions, and report design guidelines), metadata definitions, and reusable assets such as frameworks and components, will be adopted if they add value to developers. You want to make it as easy as possible for developers to comply to, and more importantly take advantage of, your corporate IT infrastructure. When data standards are sensible, easy to understand, and easy to access then there is a significantly greater chance that people will actually follow the standards in practice. When you force people to conform to standards, when it makes it onerous for them to do so, then you reduce the chance that they will actually do so.
2. Usage-driven development. The whole cannot be defined without understanding the parts, and the parts cannot be defined in detail without understanding the whole. By taking a usage-driven approach you can understand how people will actually use your system, thereby enabling you to build something that meets their actual needs. A common mistake made by traditional data approaches is that they take data-driven approaches (understandable, given their biases) which gets them in trouble because data is too narrowly focused to drive things and it doesn’t reflect the needs of your overall governance effort.
3. Include data professionals as active participants on development teams. When your DM group is external to project teams it can foster a “them vs. us” mentality within your IT organization if you’re not careful. You don’t need to have an external group to run your data governance activities; instead, individual data professionals can do so as part of their responsibilities on development teams in a collaborative and timely manner. This is one of the fundamental concepts of the Agile Data method.
4. Educate developers. Developers need to understand why your MDM efforts are important, what the benefits are, and how to work together with your DM team. When they know why something needs to be done and how to do it effectively, then the chances  that they’ll actually do it are much better.
5. Allow teams to choose their way of working (WoW). Because teams vary in size, distribution, purpose, criticality, need for oversight, and member skillset, one process size does not fit all. The implication is that your approach to support data-oriented activities, including governance, will vary by team.
6. Align team structure with architecture. The organization of your data team should reflect the enterprise architectural structure. For example, a centralized data team will struggle to support an environment with a decentralized architecture.
7. Align HR policies with IT values. You need to establish specific incentives/rewards appropriate for the mindset of your technical staff to ensure that they follow your data governance strategy.
8. Align stakeholder policies and IT values. Your development efforts are driven and constrained by your stakeholders. Your stakeholders must be realistic in their demands on IT, and understand the implications of their decisions (you’ll need to educate them).
9. Business-driven pipeline. You should invest in the activities that are well-aligned to the business direction, return definable value, and match well with the priorities of the enterprise. This includes data-oriented activities as well. Unfortunately, many traditional data governance strategies often seem to reflect the priorities of the data bureaucrats among us, not the priorities of the business, resulting in data warehouses and others being built, but are underused.
10. Embedded compliance. It is better to build compliance into your day-to-day processes instead of having a separate compliance process which often results in unnecessary overhead. Automation is critical. For example, Instead of holding reviews to ensure that development teams follow corporate data conventions, a time consuming and expensive endeavor, why not instead run an automated analysis tool against the database schemas on a regular basis to ensure that the data naming conventions are followed?
11. Flexible architectures. Architectures that are loosely coupled and highly cohesive, that implement common architectural and design patterns, lend themselves to greater levels of consistency, reuse, and adaptability.
12. Pragmatic governance body. Effective governance bodies focus on enabling teams in a cost-effective and timely manner. They typically have a small core staff with a majority of members being representatives from the governed groups.
13. Promote self-organizing teams. The best people for planning work are the ones who are going to do it. Knowledge workers should be respected as intelligent people who can determine their own strategies for working together. When given a bit of coaching and guidance, they can plan their work within established parameters, such as iteration boundaries. Self-organization doesn’t mean that the team is out of control, any given team must conform to your overall governance strategy, enterprise architecture, and so on.
14. Risk-based milestones. You want to mitigate the risks of your project, particularly business and technical risks, early in the lifecycle. You do this by having project several milestones throughout that the teams work toward. The goal of each milestone is to address one or more risks. For example, Disciplined Agile Delivery (DAD) explicitly includes several light-weight milestones in its lifecycles (it supports several), one of which is “Proven architecture” that requires your architecture be proven through working code before Construction begins, thereby lowering overall technical risk.

3. Data Governance Success Factors

I’ve found that the following factors are critical to the success of your data governance efforts:

1. Recognize that IT governance, and better yet corporate governance, is the real goal. Data governance is just one part of your IT governance program, and it’s highly coupled with other aspects such as development governance, security governance, and so on. Focusing only on data governance puts you at risk of optimizing data governance in such a way that it doesn’t work well with the rest of your governance efforts, putting the entire governance program at risk. Remember the first agile data philosophy that data is only part of the overall picture.
2. The governance effort must be owned. If someone is not responsible for the governance effort, it will very likely die a quick death in your organization. My experience is that the people most suited to be the owners of IT governance are the executive business stakeholders. When it comes to data governance, the people least suited to govern are data management professionals, since they’re the ones being governed (amongst others). In short, don’t let your data governance efforts devolve into yet another political ploy of your data management group to try to grab additional power.
3. Have clear, quantifiable goals. What are you trying to achieve? Improved quality? Improved productivity? Improved time to value? Improved stakeholder satisfaction? Combinations thereof?
4. Measure and honestly report the results. It’s easy to talk about quantifiable goals, but it takes a fair bit of integrity to live up to your promises (and I’ll let the very serious lack of data around the effectiveness of data governance efforts speak for itself). It’s fairly easy to manage direct costs such as the salaries of the people involved with the governance effort, but a bit more difficult to measure indirect costs such as the opportunity cost of potentially lengthening decision and development life cycles due to increased governance (this issue is particularly acute for traditional governance strategies). Measuring the benefits can also be challenging, although as Doug Hubbard points out in How to Measure Anything it is possible if you think outside the box a bit. Automating metrics collection is an important aspect of lean governance.
5. Less is more. You need a lot less governance than what the pro-governance people believe, although probably a bit more governance than what the anti-governance people think. If you find that you need more governance, it’s a lot easier to add it than it is to remove unnecessary governance activities once they’re in place.
6. Educate the people affected. If the people involved, including those being governed, don’t understand what you’re trying to achieve and also believe that any additional effort on their part is worthwhile then your governance effort will quickly fall apart.