A Route to Streamlined Performance Reporting and Stronger Information Systems

In August of this year, several hundred stakeholders from around the country and beyond met in Philadelphia for a conference of the National Information Exchange Model (NIEM). The NIEM is a process and technology framework designed to help organizations develop standardized exchanges of data. With roots in the justice system—it grew out of the Global Justice XML Data Model (GJXDM)—the NIEM has already expanded into public safety, homeland security and emergency management work. At this year’s conference, the most important news was the advance of NIEM into the human services.

The human service sector is vast and multifaceted. It includes public assistance and low-income health insurance, job training programs, services for the elderly, and efforts that address child abuse and neglect; domestic violence; homelessness; mental illness and substance abuse. The sector’s resources flow through a labyrinth of funding streams originating in and passing among federal, state and local treasuries, supplemented by private philanthropy. And a host of nonprofit agencies carry out a large part of the hands-on work with clients.

With human service efforts fragmented among so many different players, information silos are a major challenge to effectiveness and efficiency. The NIEM is beginning to enable government agencies to break down those barriers. When agencies agree that a particular business process requires an exchange of data, the NIEM provides the framework for them to develop an Information Exchange Package Documentation (IEPD) that will guide all the agencies in how to make their information systems talk to each other.

Use of the NIEM for the human services is in its infancy, but there are already important successes. For example, New York City has an IEPD that allows its web portal ACCESS NYC to transmit clients’ online food stamp applications to the city agency that processes them. Texas has an IEPD that allows the state attorney general, a court or a child support office to notify other agencies of delinquent child support payments that may affect eligibility for benefits. And there is an IEPD for exchanging information between courts and child welfare agencies about clients’ service plans.

So far, use of the NIEM has been limited to exchanges in the public human services, i.e. among government agencies. But with such a large portion of work contracted to independent nonprofits, it is only a matter of time before the NIEM will impact their world as well. It makes sense for nonprofit leaders to begin thinking now about the possible future benefits and challenges of the NIEM.

In certain sub-sectors—e.g., child welfare—nonprofit human service providers often directly interact with government information systems. Nonprofits may find that such systems become more agile as the NIEM facilitates better communication among government agencies. The more pressing questions, though, will be how the NIEM may come to involve the nonprofits’ own information systems and how it may change their relationships with funders.

The most obvious use for the NIEM would be as a vehicle for organizing how nonprofits report performance data. Over the last twenty years, expectations in that arena have changed dramatically. In the past, each funder would specify a paper form that collected a few aggregate numbers about clients and services—what the performance measurement field refers to as outputs. Today there is a much greater focus on outcomes, indicators of the positive effects that a program has in clients’ lives; there is movement toward defining common indicators that can be used to benchmark the performance of similar programs no matter who funds them or where they may operate; and it is common for funders to collect electronic data at the client and service level.

The NIEM would clearly be a good tool set for specifying what client and service-level data to report. Large federal programs have already proven the feasibility and value of XML-based standards in areas such as homelessness services and HIV care. But such data exchanges, if built collaboratively with the entire sector in mind, could also have the potential to help nonprofit human service agencies solve one of their most intractable problems.

Nonprofits—especially multi-service agencies that run more than one kind of program—are in a predicament. Many government and philanthropic funders mandate that grantees use funder-provided software systems for reporting. Multiple funding streams mean that nonprofits must enter the same clients’ information into multiple systems. This increases the burden of data entry, a cost that nonprofits can ill afford. At the same time, it causes clients’ information to be fragmented among multiple databases in a way that confounds integrated delivery of services. In short, funders’ requirements tend to prevent nonprofits from acquiring information systems that comprehensively meet their own needs.

The problem can be seen as two aspirations for holism in competition with each other. A funder wants its many grantees to all collect and report information in the same way so that the funder can holistically understand the entire portfolio of work that it supports. But in order to serve clients holistically, a grantee wants all the data about its work to be in one system, though the work may be supported by multiple funding streams. Because funders have the power to mandate information systems, the nonprofit’s aspiration must usually be put on hold.

This has become a point of considerable tension between funders and grantees. For example, a well-established technical assistance provider argues that funder-mandated systems are generally not useful to program managers, while noting that commercial software packages which focus narrowly on particular areas of clinical practice (e.g. substance abuse treatment) cannot comprehensively meet the needs of multi-service organizations (Blumenthal, 2010). In the same vein, an organization that incubates and evaluates social programs asserts that evaluations often do not produce information useful to programs. It proposes that practitioners be involved in design efforts to build data collection systems that will reflect program management priorities, and believes that those systems will also be more useful to funders and researchers as well (Public/Private Ventures, 2011).

Development of NIEM data exchange protocols could solve that problem. In each sub-sector of the human services, it should be possible for stakeholders interested in performance measurement and program evaluation to agree on a common protocol, to be required by all funders, for collecting and transmitting anonymized client and service-level data. There could be one protocol for job development programs, another for substance abuse programs, etc. The existence of such protocols could free non-profits from funder-mandated databases, as the requirement for grantees would be to produce the specified data rather than to use a particular system. It would also free funders from having to develop and support software. And the protocols could point software development firms toward a new generation of human service information systems that would more comprehensively support multi-service organizations, producing the required data extracts for all funders while managing all of a given client’s data, across all programs, in one undivided store.

Seeking common agreement about data for a particular type of program is not a new idea. So far, though, efforts of this kind have usually focused on aggregate performance indicators, not client and service-level data. A well-known early example is the Cultural Data Project, which collects financial and other indicators on arts programs. New York City’s Human Services Data Project is currently developing specifications for common indicators that will be required in the city’s contracting for workforce development, senior services and alternatives to incarceration. Projects such as these are paving the way toward a human service sector that will have far better information about the comparative value and cost effectiveness of funded programs.

But aggregate indicators can only reveal surface differences. When indicators show variation between programs, stakeholders immediately begin to ask more probing questions that can only be answered by disaggregating, slicing and dicing the data in an exploratory spirit. To do that, warehouses that store client and service-level data are required. Developing common reporting protocols using the NIEM could allow those warehouses to come into being.

In addition to making information about nonprofits’ performance more accessible to funders, warehouses of linkable data could be used to make government agencies more accountable. For example, the criminal justice system comprises autonomous agencies in a long sequence from police to courts, probation, corrections and parole. Its financial costs are high, and the human costs of arbitrary or unnecessarily harsh sanctions are even higher. In many jurisdictions, nonprofits provide alternative programs—substance abuse treatment, community supervision, etc.—that can prevent minor offenses from leading to deeper involvement in the justice system. But how consistently do justice system agencies employ the alternatives that nonprofits offer? To answer that, what is necessary is a unified record that represents the entire trajectory of decision points of every agency from arrest through disposition and after, including the results of any alternative sanctions that nonprofits provide. Right now, though, it is extremely laborious to link multiple agencies’ data. When New York State convened a group of agencies to develop a panoramic statistical overview of the juvenile justice system, the costs of linking data were prohibitive, so the project limited itself to juxtaposing statistics from individual agencies along the sequence of case flow (New York State Task Force on Juvenile Justice Indicators, 2007). That is the kind of barrier that the NIEM could help overcome. If criminal justice agencies and nonprofit human service providers were to provide data to be linked in a common warehouse, it would become much less expensive to analyze the justice system’s use of detention, incarceration and alternatives.

The possibilities outlined above are exciting. But there is important groundwork that needs to be done before good data exchange protocols—or a new generation of more comprehensive information systems—can be designed. The sector currently suffers from significant weaknesses in how it plans for and even conceives of its data.

One problem arises from the gaps between what people in different functional roles need to know. Front-line human service workers relate to data from a different perspective than their supervisors do, performance indicators required by funders may not be the same ones desired for internal use, and evaluators who perform complex statistical analyses have a set of needs that overlaps only partially with those of other stakeholders. The sector so far has very limited experience in designing information management tools that meet the needs of multiple constituencies.

An even deeper problem is that the very concepts by which the human service sector organizes its work do not provide clear, specific and intuitive guidance for designing information systems. What is a “program” anyway? From a federal government perspective it is perfectly clear, as there is a Catalog of Federal Domestic Assistance that defines the goals, boundaries and funding of each one. For nonprofit service providers, though, a program is a way of thinking of a more or less coherent set of internal practices which may have staff and other resources allocated to it. The reality is that many nonprofits cobble together a single program’s resources from different funding streams— i.e., from different programs. There is therefore a disjunction between the two uses of the term. An even more ubiquitous and troublesome example is the “case” which has been rightly criticized as “an egregious transfer of an analog concept to the digital environment” (Fitch, 2007). The notion of a case does fulfill certain functions. It delineates who—client, family members, social worker—is involved; it pulls together the substantive components of the matter, such as client history and issues and service plan; and it traces the trajectory of the client from intake to discharge (Coursen and McMillan, 2010). But for the purpose of information system design, the case is a very poor way to reflect these aspects of the human service setting. A comprehensive system ought to model the entire ecosystem of people and organizations who are involved in all matters across time (Coursen, 2006). Client trajectories can be modeled more completely and flexibly if the program is conceived as a set of paths by which clients transition from one status to another (Coursen and Ferns, 2004a, 2004b). Other aspects of human service settings, such as clients’ issues and referrals to other services, stand in need of being re-conceptualized and re-modeled as well.

Another way to describe the situation is that the sector has not developed strong domain models. A domain model is a framework that conceptually represents the significant entities in a business setting and how they relate to one another. It usually implies an aspiration to represent what is common among all settings of a particular type. Major resources (e.g. Hay, 1996, 2011; Silverston, 2001a, 2001b) offer patterns for many domains such as commerce, accounting and manufacturing—but not for the human services. And it does not appear that any professional institution has yet attempted to build consensus and clarity toward human service domain models. Without laying that groundwork, it will far more difficult to satisfy the sector’s growing impatience for integrated and meaningful data to facilitate holistic program planning, service delivery, and analysis.

References

1. Blumenthal, B. (2010) ‘Is your agency (or grantee) starved for data?’, Community Resource Exchange Blog, 8 April. Available at http://www.crenyc.org/_blog/News_and_Views/  [Accessed 21 April 2011].

2. Coursen, D. and Ferns, B. (2004a) Modeling participant flows in human service programs. Journal of Technology in Human Services, 22 (4), 55-71. Available at https://files.nyu.edu/dac229/public/CoursenFerns_JTHS_modeling_participant_flows.pdf [Accessed 21 September 2011].

3. Coursen, D. and Ferns, B. (2004b) Modeling Participant Flows as a Basis for Decision Support in Human Service Programs. Tenth Americas Conference on Information Systems, New York, NY. Available at https://files.nyu.edu/dac229/public/SIGDSS03-1476.pdf [Accessed 21 September 2011].

4. Coursen, D. (2006) An ecosystems approach to human service database design. Journal of Technology in Human Services, 24 (1), 1-18. Available at https://files.nyu.edu/dac229/public/Coursen_JTHS_ecosystems_approach.pdf [Accessed 21 September 2011].
5. Coursen, D. and McMillan, J.E. (2010) A Framework for Logical Data Models in Courts. Data Administration Newsletter 14:12 (December). Available at http://www.tdan.com/view-articles/14749 [Accessed 21 September 2011].

6. Fitch, D. (2007) Designing Databases Around Decision Making. In M. Cortes & K. Rafter (Eds.), Nonprofits and Technology: Emerging Research for Usable Knowledge (pp 135 – 147). Chicago: Lyceum Press.

7. Hay, D. C. (1996) Data Model Patterns: Conventions of Thought. New York: Dorset House.

8. Hay, D. C. (2011) Enterprise Model Patterns: Describing the World (UML Version). Bradley Beach, NJ: Technics.

9. New York State Task Force on Juvenile Justice Indicators (2007) Widening the Lens: A Panoramic View of Juvenile Justice in New York State. Available at
   http://www.vera.org/content/widening-lens-panoramic-view-juvenile-justice-new-york-state [Accessed 23 September 2011].

10. Public/Private Ventures (2011) Priorities for a New Decade: Making (More) Social Programs Work (Better), March [bulletin]. Available at www.ppv.org [Accessed 5 May 2011].

11. Silverston, L. (2001a) The Data Model Resource Book, Vol. 1: A Library of Universal Data Models for All Enterprises. New York: John Wiley & Sons.

12. Silverston, L. (2001b) The Data Model Resource Book, Vol. 2: A Library of Universal Data Models for Specific Industries. New York: John Wiley & Sons.