“…quite simply, the better and more accessible the data is, the better the decisions you will make.” – “When Bad Data Happens to Good Companies,” (environmentalleader.com)
The Business Impact of an organization’s Bad Data can cost up to 25% of the company’s Revenue (Ovum Research)
Bad Data Costs the US healthcare $314 Billion. (IT Business Edge)
Bad Data Costs the U.S. Economy Over $3 Trillion a Year. (HBR)
Many IT managers would cringe at these headlines. After all, IT managers are the stewards of the data used in their businesses. That stewardship means they must develop strategies and solutions to prevent bad data from severely damaging the business. And in a cost-constrained economy, they have to present economically compelling arguments for the solutions they recommend.
Data is both a blessing and a bane to modern business. The amount, complexity, and time criticality of the data in our information driven world continues to expand rapidly (currently at a 61% annual growth rate according to the IDC).
“IDC predicts that the collective sum of the world’s data will grow from 33 zettabytes this year to a 175ZB by 2025, for a compounded annual growth rate of 61 percent.”
Determining where and how much investment to place in information technology is a large challenge in today’s complex business-technology environment. Business executives and managers are faced with exponential growth in the information needed to support their enterprises. At the same time, the global economy is recessionary, impacting budgets for IT in both the private and public sectors. This unprecedented growth challenge places a premium on making sound investment decisions in the IT technologies that will best fulfill their needs. From the perspective of these business executives, the paradigm of having the right data at the right time in the right place is essential.
The concept of gaining a positive return on an investment is fundamental to business thinking. But investing in IT is not as simple as investing in real estate property that is usually expected to appreciate in value over time. Investing in IT is investing in the “software factory” and information infrastructure that supply the data, software products, and services that allow businesses to successfully provide and distribute their products and services. If an IT factory is equipped with modern tools, it is more likely to fulfill the business’s needs than if it is using obsolete practices and technology. And since IT factories must support the most complex aspects of business, they are inherently complex themselves. Changes in the business drive changes in IT systems. From the business perspective, some of the relevant issues that place demands on IT include business expansion, changes in customer or market dynamics, and introduction of new products and services. Underlying all of these issues is the quality of the delivered products and services and the ability to retain customers.
These changes result in the need for periodic upgrades and modernization initiatives in the IT factory. In today’s world, our factories – IT included – must stay modern or they will die. The business executive and the IT Manager are in a necessary partnership to prioritize business challenges and supporting IT initiatives. These partners must not only choose the investments and the technology that will provide a positive return, but they must also establish their relative importance to the business’s goals. Due to the complexity of both the business and the IT factory, the difficulty of quantifying the return and priority of an investment is an ongoing challenge.
An excellent starting point for this kind of partnership is developing a mutual understanding of what is important to the business and its IT factory. Growth in market share, revenue, and profitability are generally what is important to the business. For any given business situation, these top level goals may be best supported by more detailed goals, such as shorter time-to-market of products and services. In another situation, quality and customer service issue resolution is of paramount importance. In a third situation, productivity improvement to reduce costs may be the prime goal. At one time or another, all of these are likely to be of interest to businesses of almost any scale. How IT strategies are optimized will, or should, be determined by the business strategies and needs. And what is measured in that process of optimization will determine what is achieved.
Technical progress notwithstanding, IT managers will find it difficult to justify their IT strategies unless they can effectively articulate the economic benefits and costs of their recommendations.
Sources of bad data are identifiable
Data is the lifeblood of the business, and one of the core problems facing the business-IT partnership is establishing and maintaining the integrity of the business’s data. Most businesses have multiple processes using this data and they range from sales and production, to supply chain, R&D, human resources, finance, and beyond. In many organizations, the methods and systems for collecting, storing, processing, and distributing data have evolved at different times in the history of the business, and they generally use different technology. In some cases, spreadsheets sit next to sophisticated information portals with key data migrating to the spreadsheets; it is then processed outside the application domain. The result is that the operational integrity of the business is directly threatened by these kinds of actions.
Data integrity can be defined as “…data that has a complete or whole structure. All characteristics of the data including business rules, rules for how pieces of data relate, dates, definitions and lineage must be correct for data to be complete.” When the integrity of data ceases to exist, the business is in trouble. One story illustrates:
A major outsourcing provider, let’s call it “Company X,” had a legacy payroll system with which they wished to share employee data with the Company’s human resource’s employee database. After the “integration” of the two systems, when the company went to hire new employees for an outsourcing contract, they experienced a stunning result: approximately 50% of the employment “hiring actions” were in error. This company typically hired 200 to 400 new employees per week. With their integrated payroll-HR system, they consistently encountered errors in the new employee’s deductions for benefits and withholding taxes. The process to resolve these errors typically took a cycle time of 7 to 22 weeks, and involved complaint processing, research, and additional check runs. The internal business processes involved had become bloated with error correction procedures, and stealth spreadsheet-based databases had cropped up to keep track of the “exceptions.” It was noted that when operated on their own, the payroll and human resources applications seemed stable and held a manageable error level. The integration however was a disaster, with both the internal and external customers threatening to outsource the entire process.
“Bad data” was the symptom of Company X’s problem, but the root cause was not. A modeling analysis of the data and its flow revealed several critical technical and management factors that caused the problem.
Technical root causes:
- Data Corruption: Two different definitions and usages of an attribute of the employee were unsuspectingly being used.
- Data Access Violation: There were multiple access points to the data resulting in uncontrolled and conflicting data updates.
- Business Rule Violation: A data sharing and migration process between databases was used without understanding and properly implementing the business rules.
In reality, the technical and operational problems were largely attributable to the lack of a sound data life cycle management approach to Company X’s data integration challenge. This deficiency was evident in four areas:
- Data and Process Integration: Procedures, methods, tools or training for performing data-process integration were missing or inadequate.
- Quality and Oversight: Standards and best practices were missing or inadequate to support the task at hand.
- Financial/Risk Management: There was no methodology for anticipating the economic and technical impact of a selected strategy.
- Organizational Responsibility: Multiple IT groups and multiple business users were involved, leading to challenges in communication, requirements definition, and authority.
The lesson for Company X: Without a systematic data modeling and data life cycle management approach, the data that supports the business and the business itself are at risk.
What is astonishing is that almost every organization has stories like the one above. The economic losses of situations like this example are high, both from the perspective of the additional costs of remediation as well as the potential or actual loss of business as a result of customer dissatisfaction. A strategy to correct and ultimately avoid problems of this nature requires actions and initiatives of both a technical and managerial nature.
Many organizations just like yours would like to leverage the benefit of modeling but need to provide a justification (both financial and non financial) to management.
