Published in TDAN.com October 2006
The concept of a “federated community” has started to pop up in a number of instances, and the nature of information sharing within (or across?) a federated community. And as more
organizations begin to explore ways to exploit free-flowing information streams (such as those enabled through services-oriented frameworks), the more important the notion of federation becomes. So
as an advocate for data quality management, I think that it is worthwhile for us to consider the special challenges that are introduced when attempting to define a data quality strategy for
interoperability within a federated community model.
First of all, what is a federated community? At the simplest level, it is a collection of participants (individuals or organizations), each of which under its own administrative domain and
governance, who agree to collaborate in some way that benefits the participants, both individually and as a community. These communities may cross organizational, political, geographic, and
jurisdictional boundaries. An easy way to identify the formation of a federation, at least one based on information sharing, is watching the development of data standards. The need for a standard
exists when two parties need to agree on a way to understand each other; the more parties that join in the activity is evidence that there is general agreement that there is benefit in
collaboration.
In addition, there may be external obligations and expectations with which each participant may be required to comply. Each participant may have different roles and obligations, and the level of
expected conformance to these obligations can vary from voluntary to various degrees of compliance.
Assessing the degree to which participants conform to best practices and the various implementations of best practices introduce interesting challenges, especially in the area of data quality
management. First of all, within an administered environment, policies regarding the quality of information can be defined and enforced, but as data leaves the organization boundary, so too does
the ability to control its quality. Second, the quality expectations for data used within a functional or operational activity within one organization may be insufficient for the needs of the
“extended enterprise.” Third, the existence of data outside the administrative domains suggests the notion of ownerless data, for which no one is necessarily accountable.
Consider this: while data quality cannot necessarily be mandated, the expected benefits of collaboration through data sharing can only be achieved when all participants willingly contribute to
successful data quality management. An important objective of the community is the development of a data quality framework is that encourages participants to willingly conform to and broaden the
integration of data quality. Here are some of the challenges:
Quantification and Measurement
By considering a definition of “quality data” as “fit for purposes,” we can infer that there are few objective metrics for measuring data quality, and that the quality of
data is dependent on business user expectations. Even within individual organizations, there often is no formalized approach for quantifying data quality. In a federated community, the absence of
formal methods for expressing ways to measure the quality of data adds additional complexity. Consistent criteria and measurements are required.
This introduces two kinds of challenges. One challenge lies in the need to express business expectations for data quality that support the various business needs of shared services within the
constraints imposed by the collaborative environment. The other challenge is specifying a universal set of dimensions of data quality for the entire community against which data quality performance
can be measured.
Politics and Organizational Behavior
While shared services provide the ability to access data materialized from numerous supplier sources, the actual components of this virtual record are likely to have been distributed across
multiple organizations, and may be located in different geographical areas and governmental jurisdictions. The challenge involves providing a framework for information sharing across
multi-jurisdictional domains while considering the differences in policies across those different jurisdictions. Since the policies for security, privacy, and management may be defined in different
ways by different jurisdictions, non-governmental boards/standards organizations, and private organizations, the framework must accommodate data quality management while maintaining conformance to
political and organizational policies.
Regulatory
Accompanying the individual political and organizational challenges are the regulatory policies and legislation that may govern the various geopolitical jurisdictions. While it is reasonable to
assume that there will be overarching policies regulating the use, sharing, and privacy of the data exchanged through the use of shared services, it is also likely that each jurisdiction’s
policies may have slight differences. The challenge is to provide a means of data quality policy management that maps the various policies into business and data rules for validating that the
policies are being consistently conformed to, while accommodating variant rules as information crosses jurisdictional borders.
Technical
Typically, the collaboration is an attempt to exploit the abilities of production or legacy systems, and layer service-oriented functionality on top of the existing applications. The combination of
existing application systems implemented across a distributed environment consisting of heterogeneous systems introduces a need to provide a data quality infrastructure that can accommodate
existing systems while ensuring alignment with future systems. Differences between hardware platforms, operating systems, data storage, and database management systems can introduce challenges in
consistency of the data and integration challenges for the conceptual consolidation of many data sources into a virtual “master data source.”
Operational
The ability to best use shared information services relies on the ability to provide accurate and current data from trusted data sources. To ensure that the quality of this data is maintained,
there is a need to establish a process for “qualifying,” as well as continuously monitoring these data sources and establish their trustworthiness for ensuring quality across the
relevant dimensions of data quality.
A lot of these issues boil down into some more fundamental questions regarding the definition of a governance framework for data quality management that is implementable (and doesn’t impose
too much of an impact on the participants), acceptable (i.e., maintains the level of data quality high enough for all to benefit), and operational (i.e., can provide continuous measures of
community-wide conformance to data quality standards). The trick lies in gaining acceptance for two ideas:
- Integrating data quality management and monitoring within the service-oriented architecture. The technical components supporting community-wide data quality management must also be provided as
component services that can be embedded within the architecture. - Providing a forum for balancing the differences in administrative and jurisdictional policies, certification, and participant requirements, to provide methods for managing and implementing the
numerous policies that reflect how data quality expectations are validated. This implies establishing protocols for configuring and deploying the shared and collaborative management and measurement
of data quality.
It is inevitable that data quality issues will erupt, and developing a governance program that is built on cross-organizational consensus will help in developing policies and protocols to address
those issues. And what I find funny is that, the more we look at the issues associated with federated communities, we will start to see (in an almost fractal way) similarities within our own
organizations’ boundaries – cross-division, cross-region, cross-department, cross-application, cross-database… Perhaps the federation concept might not be such a new one after
all.
Copyright © 2006 Knowledge Integrity, Inc.
