In the first three articles of this series, we discussed some of the essential building blocks of customer data integration, from the benefits of a closed-loop customer data management environment,
to the importance of data quality, to selecting the right data quality solution to meet your organization’s unique business needs. In this article, we will turn our attention to the customer data
conversion project.
The objective of data conversion projects is clear: to accurately move desired data elements from legacy to source, on-time and on-budget. How best to achieve this objective becomes less clear when
the data in question is customer data.
Customer data is an important component of most business-driven conversion projects. As such, the true measure of success for a customer data conversion depends on satisfying the goals of the
Business users, as well as the IT group, involved with the project. Consequently, customer data integration projects present complex challenges to even the most technology-savvy organizations. True
project success requires both technical and political know-how.
This article will present an approach for creating an incremental conversion strategy that incorporates both Business and IT goals; delivers predictable timeframes, budget projections, and resource
requirements; and, most important of all, accurate customer data.
A Savvy Conversion Strategy – Technically and Politically
Recognizing the dual, and sometimes dueling, demands of customer data conversion projects, it is important to develop an up-front project plan that defines success from both the Business and IT
perspectives.
The plan should outline an incremental approach for testing and analyzing conversion rules and results. An incremental approach is essential, because it enables the IT group to actively engage
their Business counterparts at critical decision points during the conversion planning and implementation process. Such joint involvement helps to ensure the reliability and accuracy of the
customer data throughout the conversion process, while at the same time making sure that both Business and IT objectives are being met.
It is well known that many Enterprise Application implementations are not only completed late and over-budget, but also fail to deliver the promised return on investment. There can be many reasons
for this, such as:
- Poor and unreliable meta data on the source systems
- Quality of the source systems is much worse than anticipated
- Unclear accuracy definitions and thresholds for the target system
- Contradictory IT and Business goals
Any one of these problems can derail your conversion project. When combined, they may result in project failure. Instead of falling victim to one, or all, of these pitfalls, your organization can
reduce the risk of project failure by implementing the following six-step conversion approach:
Step 1 – Get the Right Consultant
One of the best ways to set your customer data conversion project on the right path is to select a senior data conversion consultant to join your project team. This consultant should bring to the
project the knowledge and expertise gained from working on at least 20 previous customer data conversions that were similar to yours in size, scope, or both.
An experienced consultant will be able to facilitate a short, but valuable, data conversion workshop that identifies the business-driven measures of conversion success, and, consequently, the
technology resources required to support this effort. The result of this workshop should be a high-level conversion plan that identifies and aligns the project’s Business and IT objectives, and
establishes estimates for the project’s time, costs and resources. It is helpful to note that although there is a “standard” set of tasks for most data conversions, there are many different
permutations in the order in which these tasks can be executed, especially in relation to customer data conversions. It might also be determined that some of these standard tasks may be left out
altogether, or performed after the conversion.
For example, the customer data conversion’s data quality rules and transformations, and the steps taken to meet desired data accuracy levels, should vary by project. Determining just how clean the
customer data must be when it is loaded into the source system is a decision driven in large part by whether the target system is operational or analytical in nature:
- If the target is an operational system, the conversion plan should include steps for ensuring that the data accuracy levels at the time the data is loaded meet the organization’s operational
requirements, usually at 95% and above. - If the target is an analytical system, the conversion plan may make concessions for initial data accuracy levels of 90% in order to meet other business-driven and time-sensitive requirements,
allowing for follow-on data quality measures to be taken after the conversion and load.
The key is to understand how best to sequence the standard conversion tasks to meet the goals and objectives of your particular project. Your consultant should offer alternative approaches to the
standard sequence, highlight their advantages or disadvantages, and ultimately provide informed recommendations that help to reduce the risks of the conversion falling victim to common project
pitfalls.
Step 2 – Investigate the Legacy Data
Now that the high-level plan has been established and accepted, it’s time to prove or disprove the plan’s most basic assumptions about the source data. Although a critical and necessary guide for
the project, the conversion plan was developed as a tool for keeping the project on course and on target, not as a tool for defining specific data conversion rules.
The next project step requires a detailed investigation of the legacy data in order to make informed decisions about the project’s mapping specifications and data quality transformations. The most
precise, cost-efficient way to understand the legacy system’s true content, quality, structure and relationships is with an automated data profiling system. Data profiling software identifies
underlying data problems and, consequently, enables you to save significant time, money and IT resources in making the necessary data repairs before the conversion takes place.
The results of the data investigation should be reviewed with the project team’s Business representatives so that they understand the data issues that exist, and help to develop the appropriate
business rules for improving the data during the conversion process. For example, data profiling may identify that the source system has a problem with duplicate customer records. The Business user
should have specific input as to how the duplicate records will be handled, based on the nature of the target system:
- If the target is an operational system, the team may determine that a special matching and review step should be added to the project plan to ensure that only unique customer records are loaded
in the system. In this case, the team is likely to opt for a thorough review process to make certain that only true matches are considered duplicates and ultimately consolidated. - If the target is an analytical system, the team may determine that the review step is not necessary, and that more liberal matching and merging rules can be applied.
In any event, it is important to note that the conversion rules and specifications should be made only after a detailed analysis of the actual data, and should not be based on assumptions of the
state of the legacy system’s data. This detailed data analysis has the added benefit of further bringing the project’s IT and Business representatives into alignment as they work together to
understand the current condition of the data and develop strategies and business rules for its conversion.
Step 3 – Stage a Dress Rehearsal
After the underlying source data has been analyzed, data issues and errors have been revealed, and specific conversion plans for addressing these data conditions have been developed, it is time to
find out if your team’s conversion plan is ready for “prime time.” However, before executing a large volume conversion test, it is critical to stage a practice run – actually, a series of
small-volume tests – of the conversion plan. This testing effort is commonly referred to as the conversion’s dress rehearsal.
By selecting statistically significant data samples, the dress rehearsal simulates the large-volume test, and ultimately the final conversion process. Unlike the large-volume conversion test, the
dress rehearsal tests involve small data sets that permit detailed review and analysis of the tests results, particulary the data quality results. Taking multiple data samples, the project team can
better verify the consistency and accuracy of data conversion results across the data set. If the results are not satisfactory, rules and results can be reviewed in more detail and adjustments can
be made – prior to committing expensive resources to a flawed, large-volume test or even full-scale conversion effort.
This testing process may need to be iterative. However, since the team is testing samples rather than the full data set, this iterative testing approach can be accomplished quickly, during the
early weeks of the conversion project. Once the desired results are achieved, it is imperative that the conversion project’s Business team members “sign off” on the dress rehearsal results and
data quality levels before moving forward with the next step in the plan.
Step 4 – The Reality Check
At this point in the conversion process, it is necessary to revise the initial conversion plan, armed with the knowledge gained from the data analysis and dress rehearsal phases. The updated plan
should include estimates and projections of the project’s anticipated timeframes, resource requirements and necessary data quality levels. Before committing significant time and money to the
large-volume data conversion test, it is necessary to ensure that the results of the revised data conversion plan provide the internal project sponsor and Business users with the outcomes that they
require.
To avoid the dilemma of delivering what was requested, but not what is needed, this plan refinement may simply restate the project’s initial goals, but in more detail. Or, it could instead bring
to light some significant changes in project direction, and either bring the project sponsors on-board to this realignment – or serve to turn the ship before the team heads in the wrong direction.
Step 5 – The Volume Test
If all goes well during the rehearsal and subsequent results review, the next recommended step of the conversion plan is the Volume Test. As its name implies, this test goes beyond the manageable
and statistically significant data samples of the dress rehearsal. The size of the Volume Test varies by organization, with some demanding that the entire source system be included, while others
with lower risk thresholds choosing to test only 25 – 50% of the source data.
The Volume Test identifies important issues for capacity planning and cycles on the mainframe. For teams that executed steps 1-4 of this recommended approach, the volume test and upcoming
Production Implementation should be uneventful, thanks to the preliminary planning, analysis and testing during the project’s earlier phases.
Step 6 – Lights, Camera, Action!
Regardless of how well the team executed the project plan, this final step – the actual conversion – is always the most nail-biting. Even though in theory this should be the least anxious time
because of all the upfront planning and testing, there are always variables that cannot be totally accounted and planned for, like the computing infrastructure. But, what you don’t want is for
these last-minute concerns to be data-related, because unlike some of the other conversion variables, the data-related issues – particularly the quality of the customer data – can be controlled
with proper planning, testing and execution.
In Summary
A customer data conversion can be considered successful only if the resulting data meets the needs of both its Business users and IT team. To help achieve this objective, both groups should be
included on the conversion planning and implementation team. To help ensure that the conversion is delivered on-time and on-budget, the project should include 6 critical steps: (1) select an
experienced data conversion consultant to lead the project; (2) use an automated data profiling tool to thoroughly investigate the data; (3) perform a dress rehearsal using small, statistically
significant data samples; (4) based on the results of the dress rehearsal, update the project estimates and projections; (5) conduct a large-volume conversion test; (6) run the conversion.
By working together, the Business and IT users are very nearly guaranteeing a successful data conversion implementation – from both standpoints.
What’s Next …
In this series we have laid the foundation for effective customer data integration by presenting best practices for enterprise customer data management, data quality and customer data conversions.
Our next article will address the convergence of customer data integration and regulatory compliance, and discuss the benefits an organization can gain by leveraging its customer data management
infrastructure for a host of priority compliance initiatives, including U.S.A. PATRIOT Act, OFAC (Office of Foreign Assets Control), SOX (Sarbanes-Oxley), Basel II, Do Not Call lists, internal risk
lists, and others.
To be successful, organizations serious about compliance must also be serious about data quality, because the core of any reliable screening and compliance program is quality data. Beyond this,
organizations should also understand what compliance-specific functionality not commonly provided in traditional data quality solutions is required in order to effectively reduce the time, cost and
risk associated with suspect screening and customer identification and verification. The next article in this series will present a proven data quality approach to regulatory compliance.
