My company led a Data Management Assessment for a government agency this spring, using the Data Management Capability Assessment Model (DCAM), and delivered the final report at the end of May. In my last column comparing the DCAM to the DMM, I said that I’d share my observations with you so that you can get an idea of how it was conducted and how the DMM-based method was adapted for the DCAM. A secondary objective is to assist you in planning for conducting an internal Data Management Assessment, or to prepare for an externally facilitated Assessment.
My transition to employing the DCAM as an assessment yardstick, versus the 35+ Assessments I’ve led using the DMM, was enabled by obtaining DCAM certification, becoming an EDM Council Partner, working with the Council and Solidatus on the DMM to DCAM model mapping exercise in 2022, and being awarded a contract for my first DCAM-based Assessment engagement with a government agency.
Prior to this engagement, I led over 35 DMM-based Assessments for organizations in many industries, as the foundation for capability scores, recommendations, and a multi-year roadmap. My experiences proved that a workshop-centric method, developed in parallel with the DMM’s creation, helped forge a cohesive, consensus vision within the organization, produced solid, comprehensive results, and accelerated the implementation of data management programs. The deliverables engendered a high level of satisfaction from all previous clients, and assessed organizations were able to obtain funding and make rapid progress in program development.
So why change a tested and proven approach? I decided that the DataWise Assessment Team would use the same approach, methods, engagement phases, and activity durations, adapting them as needed for the DCAM.
In this column, I’ll first describe the Assessment steps and activities, including scoping, phases, timing, materials used and developed, and what the deliverable contains. Then, we’ll explore the differences and challenges that arose in conducting an Assessment with the DCAM compared to the DMM, and how they were managed.
Assessment Phases and Activities
There are three phases in the Assessment project:
- Phase 1 – Preparation
- Phase 2 – Assessment
- Phase 3 – Assessment Report
Phase 1 – Preparation
The project begins with the kick-off meeting, held with the executive sponsor and designated client project manager. The first order of business is to confirm the Assessment scope — organizational scope, data domain scope, and model scope — that was shaped by pre-engagement discussions and reflected in the scope of work.
- Organization – What segments of the organization are included? If this is a large organization with multiple locations or distributed business lines, the client may wish to limit the Assessment to specific locations or business lines. The DCAM and the DMM are both quite flexible, in that they can be applied to the entire enterprise, a portion of the enterprise, or to one or more data stores, for example a data lake or data warehouse. For smaller organizations or those with a single primary location, the typical scope is the enterprise.
- Data Domain – Organizations often categorize their data assets into primary domains, for example, Customer, Product, etc. (And they definitely should, both to facilitate data governance/accountability and for data architecture, but that’s another discussion). For instance, let’s say an organization has multiple systems managing customer data and a business priority is to implement a Customer Relationship Management system. It may make sense for them to limit the Assessment scope to this domain. For organizations that do not have a defined enterprise data architecture, the client may determine what information systems or repositories are pertinent. Organizations usually exclude ancillary data managed by supporting functions from the enterprise scope, for example, facilities management.
- Model – What portion of the assessment model will be employed? Both the DMM and the DCAM enable organizations to either conduct the Assessment against the entire model or to select the capabilities of greatest interest to them. For example, they can select a subset of the 8 Components and 38 Capabilities in the DCAM, or a subset of the 25 Process Areas of the DMM. And, in the recently completed DCAM Assessment, the organization did not currently have a defined and implemented data governance function. The decision was to remove DCAM Component 7.0 Data Control Environment from the scope, as those capabilities are dependent upon an established data governance program.
The next topic in the kick-off meeting is to confirm the dates for Phase 2 – Assessment, followed by an explanation of the preparation (pre-work) activities that the organization needs to accomplish prior to “Assessment Week” – workshops, interviews, and work products review. Client responsibilities include:
- Creating introductory slides for the Assessment briefing – Why they are conducting an Assessment and how they plan to use the results.
- Identifying workshop participants and interviewees – For the workshops, determining which 20-40 individuals are key subject matter experts, program managers, and business sponsors who possess knowledge about the data assets, and for the interviewees, senior staff who possess a span of control for portions of the data assets and operate in the context of the organization’s strategic direction.
- Scheduling workshops and interviews – Four three-hour workshops are facilitated, sufficient to interrogate the DCAM’s 38 Capabilities or the DMM’s 25 Process Areas. The EDM Council has scoring mechanisms both for at the Capability level as well as for the 136 Sub-Capabilities, which correspond to the DMM Functional Practices — in short, they are requirements that if met, collectively describe a fully implemented data management program.
- Identifying work products – The DCAM has a large list of artifacts / work products associated with Capabilities as well as for each Sub-Capability. I haven’t counted them, but I believe that there are, in total, more than the 596 included in the DMM.
To make the artifact discovery and gathering task less burdensome, the Assessment Team reviewed the entire DCAM and created an annotated list of key work products for each Sub-Capability that are expected for evidence of DCAM Score Levels 4-6.[1] This was provided to the client for their internal team to use.
The EDM Council recommends that key participants in an Assessment take the DCAM certification course prior to the commencement of the project. If that is not possible, a pre-Assessment briefing can be conducted by the Assessment Team to familiarize those involved with the material, and the organization can be provided the scoring mechanism template as well. If the organization is a Council member, they can also download the DCAM document.
The Assessment Team engaged to lead the Assessment provides support to the client throughout this phase, as needed. The duration of Phase 1 for most organizations is 4-6 weeks.
Phase 2 – Assessment
Phase 2 is an intensive period of one week, during which workshops are conducted, interviews are held, and data management work products are reviewed by the Assessment Team.
- Workshops – These begin with providing instructions to the participants about how the model will be interrogated. The Assessment Team starts with the first DCAM Component and facilitates discussion about the current state of each requirement (Sub-Capability).
The participants provide their opinion about which lifecycle stage the organization has achieved at this time, and illustrate their point of view with project or program examples. The facilitator then reflects the consensus score. If there is disagreement that cannot be easily resolved, or a lack of information, e.g., “Only Tom can answer this and he’s on vacation,” the item is noted and followed up at a later time. (Note that this rarely happens if the client has been successful in assembling a group of subject matter experts across the selected scope). As a part of notetaking, the Assessment Team highlights recent accomplishments, challenges, and work in progress.
The score chart is illustrated below.[2] Challenges in arriving at the final score for a Sub-Capability are explored in the next section.
| DCAM Score Level | Sub-Capability Score Description |
| 1 – Not Initiated | Data management activities are ad hoc, and likely to vary project to project, documentation slim to non-existent |
| 2 – Conceptual | The organization has considered and is thinking about or planning an effort to establish or improve a capability |
| 3 – Developmental | There is a project or effort funded and underway, with IDed stakeholders, developing policies, processes, and standards |
| 4 – Defined | Active governance participation, approved policies, processes and standards, data is defined and prioritized, funding is sustained, compliance program is defined |
| 5 – Achieved | Mandated by executive management, full business engagement, adherence audited |
| 6 – Enhanced | Fully embedded across the organization, continuous improvement in capabilities |
The duration of the four workshops is typically three hours each, either for the full DCAM or the full DMM. This duration will be shorter if the client has selected the Capability Assessment level, versus the more usual Sub-Capability Assessment.
- Interviews – The goals of the interviews are to: obtain a broader perspective of the state of data management; capture the business context; explore the enterprise or business area objectives; and surface aspirations and challenges. These are open-ended interviews structured around a few initial questions: what is your role; what are the data assets you create, control or use; what data-related challenges are you experiencing; if you had no data issues, what would you be able to accomplish? The Team also solicits solution recommendations, posing questions like ‘If you were the sole authority and had sufficient resources, how would you improve the situation?’
In most cases, 10-15 interviews are conducted, ranging from one-half to one hour each. This will vary with the client and the number of senior individuals who have broad responsibilities for data assets.[3]
- Work Products Review – The client’s designated team will have located work products during Phase 1 and organized work products by DCAM Component or DMM Process Area, usually in file folders or links to data management toolsets. A point of contact will take the Assessment Team on a ‘tour’ of the work products, and evidence of satisfaction of the assessed Score Levels will be noted, as well as deficiencies that may affect the score.
The work products themselves are not scored, and for either the DCAM or the DMM, score points are not granted or removed because of the existence of a work product. However, a ‘preponderance of the evidence’ standard is applied. For example, if a requirement statement specifically calls for a work product, such as DCAM 1.1.1 – “The Data Management Strategy is developed, documented, and consolidated.” and the workshop participants agreed on a score of 4 – Defined, the Assessment Team will expect to see a completed document that is approved by relevant stakeholders. If it doesn’t exist, or is still in development, the participant score will be discussed with the client project manager and will probably change. This is not a typical occurrence, since the participants (collectively) should know the status of key internal deliverables.
The duration of this activity will vary, depending on the number of work products the organization is able to produce. For organizations just getting started in their data management journey, there may be hardly any work products to review; while for organizations which have been making steady progress for years, there may be hundreds (for one assessment, I encountered over 650). A tip is to encourage the organization to be diligent about discovering work products during Phase 1. After all, the collection represents their enterprise data management asset library. (Even if they complain, the organization will benefit by the consolidation of knowledge).
Phase 3 – Assessment Report & Briefing
Phase 3 is where ‘the magic happens.’ The Assessment Team, armed with the completed and annotated scoring mechanism, interview notes, work product review conclusions, and the running list of challenges, accomplishments, data management plans, and business priorities, builds the Assessment Report from the bottom up. The report is usually about 100 pages in length, including a three-year roadmap. As sections of the report are completed, they are provided to the client project manager for review.
The duration of this phase is from 4-5 weeks, depending on the scope that was determined in the scope of work and confirmed in Phase 1. Here is a list of the report sections. The only new document section added for the DCAM was the comparison of the organization’s scores with the Council’s 2023 Global Data Management Benchmark capability scores.
Assessment Report
1.0 Introduction
- Organization Mission and Key Goals
- DCAM Assessment
- DCAM Assessment description – What it is
- DCAM Model and Scoring – Model Components, scoring approach
- Assessment Methodology – How we conducted the Assessment
- Staff Engagement – How many, what roles?
- Assessment Summary – High-level description, the radar charts for Capabilities and Sub-Capabilities, discussion
- Global Data Management Benchmark Comparison – How they scored by Capability compared to the EDM Council’s Global Benchmark – Discussion and chart
- Organizational Themes – Observed organizational factors that may affect the data management program
2.0 Strengths and Gaps
- Accomplishments – A summary of data management achievements in the recent past
- Strengths and Gaps – A chart of highest-scoring and lowest-scoring Capabilities, discussion of gaps
3.0 Initiative Recommendations
- Key Projects to Expand Program Capabilities – This section consists of a slate of projects — each with a beginning, middle, and end — synthesized from all sources, and customized to the organization. You can describe these projects collectively as ‘This is what we would do if we were in charge of your data management program.’
Projects are described in brief business cases — the rationale, why it’s a priority, a description of what is to be accomplished, the organizational unit that should lead it, and the activity steps (sequential activities, intended as a high-level project plan). Some project descriptions may be illustrated by pictures or charts, for example, an organization chart for a proposed Data Management Organization, accompanied by position descriptions and intended responsibilities.
This section is typically from 30-40+ pages in length, representing 8-12 projects over roughly a three-year period. If you propose too many projects, the client can be put off — ‘we can never do all that’ — and it may create resistance to moving forward. By the time you formulate these projects, based on the organization’s goals, the current state of data management practices, and your professional knowledge, the ‘bespoke Saville Row suit,’ you should have a good sense of what they can tackle and absorb.
At the end of this section, a one-page diagram of the three-year roadmap is presented, as shown in this sample:
4.0 Assessment Findings and Remediations
This section presents the results of the interrogation of the DCAM’s Sub-Capabilities across all selected Components. It consists of the Score Level for each Sub-Capability, with detailed observations based on workshops and artifact reviews, and suggested recommendations (in DCAM terms, remediations) which, if achieved, would raise the score. Charts are provided to show Sub-Capability scores and Capability level averages.
This section is typically about 30 pages in length; scoring notes and interview content provides examples and context. At the end of the section, there is a chart and description of ‘Quick Wins,’ tasks that have few or no dependencies and are relatively low effort, usually from 10-15 recommended activities. In many cases, the quick wins consist of documenting and gaining approval for sound practices that are currently informal, or may pertain for a single major project and would be beneficial to formalize and leverage.
Executive Briefing
The final deliverable is the Executive Briefing, highlighting the results, recommendations and near-term projects identified in the roadmap. It highlights results, organizational themes, benchmark comparisons, key accomplishments, and major gaps, as well as the sequence plan for proposed project initiatives. 3-5 high priority initiatives are further explained and discussion is encouraged.
The briefing is the culmination of the Data Management Assessment. It is your opportunity to build executive advocacy and help your client get the funding needed to accelerate program progress. You need to prepare thoroughly and weave in the priorities that executives expressed in the interviews, ensuring that their objectives are reflected within the context of the proposed projects. It is presented to executives identified by the client, following delivery of the Assessment Report, and is usually about 15 slides long.
DCAM – DMM Assessment Comparison
As discussed in my previous column, the model concepts of the DMM and DCAM are essentially the same. Some topics are more detailed in one model versus the other. The essential difference is structural, in that the Sub-Capability statements in the DCAM are stating what should be implemented, regardless of the scope of the organization or data domain scope.
The DMM, by contrast, builds in the scope of application, roughly translated as project (Level 1), program or business line (Level 2) and organization-wide (Level 3), and each practice is either met, partially met, or not met. The DCAM takes a lifecycle approach to scoring; if the statement indicates ‘you should do XYZ,’ the score levels correspond to ‘how far you have gotten with implementing XYZ.’
Explaining the Model
For the DMM Assessment, one of the preparatory materials provided is an abridged version of the model, Introduction to DMM Concepts, without the practice statements. Stakeholders and participants can review this content and have a good idea of the topics to be discussed. If the organization is an EDM Council member, they have access to the DCAM model file, but if not, you will need to brief them beforehand. The Council recommends that participants take the DCAM certification course prior to being involved in an Assessment. This is ideal but cannot always be achieved.
The DCAM is highly engineered, integrating core practices that every organization should be performing as well as the support practices and work products that they should be implementing or creating. Its origins in the financial world, aimed at articulating and substantiating data management programs as a yardstick for use by auditors and regulators, is an unstated premise. That is, it seems to assume that organizations will have ample resources to develop, define, implement, and enforce a large set of data management practices. Most organizations do not. I found that I needed to make it clear that ‘you are where you are’ and to state that the DCAM’s practices, taken as a whole, were unlikely to be fully implemented other than for large organizations.
Facilitating the Workshops
Overall, the DCAM, by design, employs more abstraction in the Sub-Capability statements than one finds in the DMM’s Functional Practices. The facilitator needs to be ready to offer real-world implementation examples to help the workshop participants interpret the model in the light of their organization’s situation.
Both models have a similar issue with the topic areas of Data Management Strategy, Business Case, and Program Funding, in that subject matter experts are not usually expected to have an enterprise-wide perspective. While facilitating these DCAM topics, just as for the DMM, I learned that I had to encourage the participants to ‘put on the enterprise hat’ and pretend that they were the Chief Data Officer with authority over the program. Otherwise, they tended to tune out, coming to life again in the areas of governance, data quality, data architecture, and technology. However, I think that this is a feature of data management reference models, not a bug, as we do want staff to work through the big picture as well as focus on their areas of expertise. (The ‘data culture,’ right?)
Navigating the Model
When conducting a DMM Assessment, the facilitator starts from Level 1 statements, moves to Level 2 statements, and then to Level 3, where Level 3 overall represents a sound set of practices (defined activities, policies, processes, and standards) that span the scope of ‘enterprise data.’ This means that most organizations you will encounter have not yet met all of the practices in Level 3, and most will not be operating at Level 4. However, if you are conducting a DCAM Assessment against all eight Components, you must interrogate all 136 Sub-Capability statements. I found that in the workshops, it was important to be careful about time management to allow meaningful discussion about all of the content.
Assessment Materials
The Partner materials for Assessments produced by the Council are well structured and professionally presented. However, since most EDM Council Partners conduct DCAM Assessments using a survey-based approach, I had to augment the scoring mechanism to accommodate the workshop-based method, such that my team member could both score and take notes in one spreadsheet.
Scoring Challenges
For the DMM, there are typically discussions about functional practices that require distinguishing between a project scope (Level 1) and a program scope (Level 2), as well as between Level 2 and an enterprise scope (Level 3). These determinations are made, by consensus, in the workshops.
For the DCAM, the challenge is both in determining what constitutes Score Level 3, and differentiating between Score Level 3 and Score Level 4. The guidance for Score Level 3 is high level, so the group needs to chew through considerations such as: ‘Is there a formal project, has the Steering Committee approved a business case, is a data working group focusing on this, has a document been started?’ If not, the group scores the Sub-Capability a 2.
In terms of differentiating between Score Levels 3 and 4, I found that I missed DMM’s Level 2 (program scope) construct, in that accomplishments in one business line or region may be more advanced, and the DMM assumes that if the organization is successful in one area, it can replicate this success across the enterprise. These types of circumstances are not as easy to deal with in the DCAM; I wished that there was a ‘Score Level 3.5’ to reflect this situation. For example, policies, processes, and standards implemented for a data warehouse are typically documented and relatively rigorous, representing industry best practices. However, if the scope of the DCAM Assessment is enterprise-wide, there is no score-related recognition of that strength if it doesn’t attain 4-Defined becauseit is not documented or approved organization-wide.
Choice of Assessment Method
The first assessment executed using an early version of the DMM was conducted by Citi in 2011, encompassing 20+ regions. Citi used a survey and distributed it across all regions. The end result was quite positive; the organization immediately began to enhance data governance, document its data assets, and implement a data quality program.
However, it took approximately six months to complete, and the Assessment Team found that the model was not self-explanatory to many of the recipients, resulting in many emails, phone calls, and web meetings aimed at explaining the meaning and synthesizing some consistency to the responses. They also had to put in significant effort into writing the final report, harmonizing, linking, and levelling conclusions on the back end.
When I reviewed the EDM Council’s Partner assessment assets for the survey-based method, I was impressed with the quality of materials and the level of detail, but still wondered how consistency in interpretation would be managed. Not all of the respondents are likely to have taken the DCAM certification course, and not all of them are data management experts.
That was a factor in my decision to employ the consensus / workshop approach for information gathering. Because I knew from experience that I could assist participants to reach consensus, whereas if I were to receive responses about the same topics from ten different groups, it could be very challenging to knit it all together. In addition, from the materials provided, I estimated that the duration of the Assessment would increase, versus delivering a comprehensive report and briefing in six weeks.
Finally, one of the greatest benefits of the workshops is what many participants have said over the years, such as ‘It was very valuable to discuss these important matters all together.’ The biggest challenge of this approach is the difficulty clients have in carving out 12 hours in a week for busy staff members. But it is worth it to forge a common vision, learn of opportunities to leverage peer accomplishments, and share a consistent point of view about the data assets.
That said, if I were leading a DCAM Assessment for a very large, heterogeneous, or geographically distributed organization, I think it would be advisable to consider a survey-based approach, or hold workshops focused on common core topics, for example, Components 5.0 Data Quality Management and Component 6.0 Data Governance, and employ the surveys as an augmentation for topics that may lend themselves to a smaller set of knowledgeable people.
I greatly enjoyed the challenge of conducting my first DCAM-based Data Management Assessment and am looking forward to my next engagement. With the caveats I’ve mentioned, the DCAM model works smoothly in practice as a cohesive reference model, and my team was able to deliver a comprehensive report and cogent executive briefing. In sum, there are no significant differences in how the Assessment is conducted, using the approach I’ve described.
So if your organization previously
employed the DMM and is transitioning to the DCAM, have confidence, and get
going with your Data Management Assessment.
[1] At Score Levels 1-2, there aren’t many client deliverables. At Score Level 3, Developmental, the Assessor would expect to see an approved business case, project plan, or ongoing progress reports, but not a defined and approved artifact.
[2] In my previous TDAN column “Data Management Assessment Capability Model
and the Data Management Maturity Model – A Comparison” I discuss DCAM scoring in detail.
[3] It is very important to take thorough notes, as business explanations, examples or solutions offered, and issues expressed will be analyzed in Phase 3 and reflected in the Assessment Report and roadmap.
