Excerpts from Measuring Program Outcomes: A Practical Approach © 1996 United Way of America

Introduction to Outcome Measurement

If yours is like most human service agencies or youth- and family-serving organizations, you regularly monitor and report on how much money you receive, how many staff and volunteers you have, and what they do in your programs. You know how many individuals participate in your programs, how many hours you spend serving them, and how many brochures or classes or counseling sessions you produce. In other words, you document program inputs, activities, and outputs. Inputs include resources dedicated to or consumed by the program. Examples are money, staff and staff time, volunteers and volunteer time, facilities, equipment, and supplies. For instance, inputs for a parent education class include the hours of staff time spent designing and delivering the program. Inputs also include constraints on the program, such as laws, regulations, and requirements for receipt of funding. Activities are what the program does with the inputs to fulfill its mission. Activities include the strategies, techniques, and types of treatment that comprise the program's service methodology. For instance, sheltering and feeding homeless families are program activities, as are training and counseling homeless adults to help them prepare for and find jobs. Outputs are the direct products of program activities and usually are measured in terms of the volume of work accomplished–for example, the numbers of classes taught, counseling sessions conducted, educational materials distributed, and participants served. Outputs have little inherent value in themselves. They are important because they are intended to lead to a desired benefit for participants or target populations. If given enough resources, managers can control output levels. In a parent education class, for example, the number of classes held and the number of parents served are outputs. With enough staff and supplies, the program could double its output of classes and participants. If yours is like most human service organizations, you do not consistently track what happens to participants after they receive your services. You cannot report, for example, that 55 percent of your participants used more appropriate approaches to conflict management after your youth development program conducted sessions on that skill, or that your public awareness program was followed by a 20 percent increase in the number of low-income parents getting their children immunized. In other words, you do not have much information on your program's outcomes. Outcomes are benefits or changes for individuals or populations during or after participating in program activities. They are influenced by a program's outputs. Outcomes may relate to behavior, skills, knowledge, attitudes, values, condition, or other attributes. They are what participants know, think, or can do; or how they behave; or what their condition is, that is different following the program. For example, in a program to counsel families on financial management, outputs–what the service produces–include the number of financial planning sessions and the number of families seen. The desired outcomes–the changes sought in participants' behavior or status–can include their developing and living within a budget, making monthly additions to a savings account, and having increased financial stability. In another example, outputs of a neighborhood clean-up campaign can be the number of organizing meetings held and the number of weekends dedicated to the clean-up effort. Outcomes–benefits to the target population–might include reduced exposure to safety hazards and increased feelings of neighborhood pride. The program outcome model depicts the relationship between inputs, activities, outputs, and outcomes. Note: Outcomes sometimes are confused with outcome indicators, specific items of data that are tracked to measure how well a program is achieving an outcome, and with outcome targets, which are objectives for a program's level of achievement. For example, in a youth development program that creates internship opportunities for high school youth, an outcome might be that participants develop expanded views of their career options. An indicator of how well the program is succeeding on this outcome could be the number and percent of participants who list more careers of interest to them at the end of the program than they did at the beginning of the program. A target might be that 40 percent of participants list at least two more careers after completing the program than they did when they started it.

Program Outcome Model

Resources dedicated to or consumed by the program money staff and staff time volunteers and volunteer time

facilities equipment and supplies

Constraints on the program laws regulations funders' requirements

What the program does with the inputs to fulfill its mission feed and shelter homeless families provide job training educate the public about signs of child abuse counsel pregnant women create mentoring relationships for youth

The direct products of program activities number of classes taught number of counseling sessions conducted number of educational materials distributed number of hours of service delivered number of participants served

Benefits for participants during and after program activities new knowledge increased skills changed attitudes or values

modified behavior

improved condition altered status

Why Measure Outcomes?

In growing numbers, service providers, governments, other funders, and the public are calling for clearer evidence that the resources they expend actually produce benefits for people. Consumers of services and volunteers who provide services want to know that programs to which they devote their time really make a difference. That is, they want better accountability for the use of resources. One clear and compelling answer to the question of "why measure outcomes?" is to see if programs really make a difference in the lives of people. Although improved accountability has been a major force behind the move to outcome measurement, there is an even more important reason: to help programs improve services. Outcome measurement provides a learning loop that feeds information back into programs on how well they are doing. It offers findings they can use to adapt, improve, and become more effective. This dividend doesn't take years to occur. It often starts appearing early in the process of setting up an outcome measurement system. Just the process of focusing on outcomes–on why the program is doing what it's doing and how participants will be better off–gives program managers and staff a clearer picture of the purpose of their efforts. That clarification alone frequently leads to more focused and productive service delivery. Down the road, being able to demonstrate that their efforts are making a difference for people pays important dividends for programs. It can, for example, help programs:

• Recruit and retain talented staff • Enlist and motivate able volunteers • Attract new participants • Engage collaborators • Garner support for innovative efforts • Win designation as a model or demonstration site • Retain or increase funding • Gain favorable public recognition

Results of outcome measurement show not only where services are being effective for participants, but also where outcomes are not as expected. Program managers can use outcome data to:

• Strengthen existing services • Target effective services for expansion • Identify staff and volunteer training needs • Develop and justify budgets • Prepare long-range plans • Focus board members' attention on programmatic issues

To increase its internal efficiency, a program needs to track its inputs and outputs. To assess compliance with service delivery standards, a program needs to monitor activities and outputs. But to improve its effectiveness in helping participants, to assure potential participants and funders that its programs produce results, and to show the general public that it produces benefits that merit support, an agency needs to measure its outcomes. These and other benefits of outcome measurement are not just theoretical. Scores of human service providers across the country attest to the difference it has made for their staff, their volunteers, their decision makers, their financial situation, their reputation, and, most important, for the public they serve.

Eight Steps to Success

Measuring Program Outcomes provides a step-by-step approach to developing a system for measuring program outcomes and using the results. The approach, based on methods implemented successfully by agencies across the country, is presented in eight steps, shown below. Although the illustration suggests that the steps are sequential, this is actually a dynamic process with a good deal of interplay among stages.

Example Outcomes and Outcome Indicators for Various Programs These are illustrative examples only. Programs need to identify their own outcomes and indicators, matched to and based on their own experiences and missions and the input of their staff, volunteers, participants, and others.

Type of Program Outcome Indicator(s)

Smoking cessation class

Participants stop smoking. • Number and percent of participants who report that they have quit smoking by the end of the course

• Number and percent of participants who have not relapsed six months after program completion

Information and referral program

Callers access services to which they are referred or about which they are given information.

• Number and percent of community agencies that report an increase in new participants who came to their agency as a result of a call to the information and referral hotline

• Number and percent of community agencies that indicate these referrals are appropriate

Tutorial program for 6th grade students

Students' academic performance improves.

• Number and percent of participants who earn better grades in the grading period following completion of the program than in the grading period immediately preceding enrollment in the program

English-as-a-second-language instruction

Participants become proficient in English.

• Number and percent of participants who demonstrate increase in ability to read, write, and speak English by the end of the course

Counseling for parents identified as at risk for child abuse or neglect

Risk factors decrease. No confirmed incidents of child abuse or neglect.

• Number and percent of participating families for whom Child Protective Service records report no confirmed child abuse or neglect during 12 months following program completion

Employee assistance program

Employees with drug and/or alcohol problems are rehabilitated and do not lose their jobs.

• Number and percent of program participants who are gainfully employed at same company 6 months after intake

Homemaking services

The home environment is healthy, clean, and safe. Participants stay in their own home and are not referred to a nursing home.

• Number and percent of participants whose home environment is rated clean and safe by a trained observer

• Number of local nursing homes who report that applications from younger and healthier citizens are declining (indicating that persons who in the past would have been referred to a nursing home now stay at home longer)

Prenatal care program

Pregnant women follow the advice of the nutritionist.

• Number and percent of women who take recommended vitamin supplements and consume recommended amounts of calcium

Shelter and counseling for runaway youth

Family is reunified whenever possible; otherwise, youths are in stable alternative housing.

• Number and percent of youth who return home • Number and percent of youth placed in alternative living arrangements who

are in that arrangement 6 months later unless they have been reunified or emancipated

Camping Children expand skills in areas of interest to them.

• Number and percent of campers that identify two or more skills they have learned at camp

Family planning for teen mothers

Teen mothers have no second pregnancies until they have completed high school and have the personal, family, and financial resources to support a second child.

• Number and percent of teen mothers who comply with family planning visits • Number and percent of teen mothers using a recommended form of birth

control • Number and percent of teen mothers who do not have repeat pregnancies

prior to graduation • Number and percent of teen mothers who, at the time of next pregnancy, are

high school graduates, are married, and do not need public assistance to provide for their children

Glossary of Selected Outcome Measurement Terms

Inputs are resources a program uses to achieve program objectives. Examples are staff, volunteers, facilities, equipment, curricula, and money. A program uses inputs to support activities. Activities are what a program does with its inputs-the services it provides-to fulfill its mission. Examples are sheltering homeless families, educating the public about signs of child abuse, and providing adult mentors for youth. Program activities result in outputs. Outputs are products of a program's activities, such as the number of meals provided, classes taught, brochures distributed, or participants served. A program's outputs should produce desired outcomes for the program's participants. Outcomes are benefits for participants during or after their involvement with a program. Outcomes may relate to knowledge, skills, attitudes, values, behavior, condition, or status. Examples of outcomes include greater knowledge of nutritional needs, improved reading skills, more effective responses to conflict, getting a job, and having greater financial stability. For a particular program, there can be various "levels" of outcomes, with initial outcomes leading to longer-term ones. For example, a youth in a mentoring program who receives one-to-one encouragement to improve academic performance may attend school more regularly, which can lead to getting better grades, which can lead to graduating. Outcome indicators are the specific items of information that track a program's success on outcomes. They describe observable, measurable characteristics or changes that represent achievement of an outcome. For example, a program whose desired outcome is that participants pursue a healthy lifestyle could define "healthy lifestyle" as not smoking; maintaining a recommended weight, blood pressure, and cholesterol level; getting at least two hours of exercise each week; and wearing seat belts consistently. The number and percent of program participants who demonstrate these behaviors then is an indicator of how well the program is doing with respect to the outcome. Outcome targets are numerical objectives for a program's level of achievement on its outcomes. After a program has had experience with measuring outcomes, it can use its findings to set targets for the number and percent of participants expected to achieve desired outcomes in the next reporting period. It also can set targets for the amount of change it expects participants to experience. Benchmarks are performance data that are used for comparative purposes. A program can use its own data as a baseline benchmark against which to compare future performance. It also can use data from another program as a benchmark. In the latter case, the other program often is chosen because it is exemplary and its data are used as a target to strive for, rather than as a baseline.