In this step, the people who would be affected by the program evaluation are identified, along with people who are in a position to use it to improve the program or act to continue progress toward the program’s goals.

The stakeholders include CDSMP course leaders and master trainers, their sponsoring agencies, and participants in their classes.  The focus of this component of the CDSMP project evaluation is on the course leaders, who were asked to distribute participant evaluation forms, collect some information (age group, sex, chronic disease) from participants, and send completed packets to CDPHP.

Program evaluation depends on a careful, specific description of the program and the relationship between the program and its intended affect.

CDSMP is a series of six two-hour sessions presented over six weeks.  The ideal group size is 10 to 14 participants.  Each session includes presentations on common concerns for people with a chronic disease such as managing pain and fatigue, healthy eating, and working with a health care team, and provides time for developing and discussing the success of participants’ self-determined action plans for changing a behavior.  Each week, participants choose a buddy from the class; the buddies agree to check in with each other between sessions to find out about and support progress toward each other’s action plans.

A simple logic for this project is:

In this step, attention shifts to what is wanted from the program evaluation.  Questions to consider include: Generally, why are you evaluating?  Who will use the evaluation results, and how will they be used?  What data collection strategy(ies) will you use?

The purposes of this program evaluation are to support project accountability and to provide constructive feedback to course leaders, master trainers and the agencies sponsoring CDSMP.

The specific evaluation question for this example is: Is CDSMP being implemented as planned?  To answer this question, data were collected from participants using a questionnaire.

Now, with all of the background information from the first three steps, comes the question of exactly what information is needed for meeting the evaluation needs identified in step 3.

CDSMP sessions are guided by a pair of leaders, who use a script.  The script is very self-contained, which means that leaders do not need to have a medical background to be effective.  Since leaders agree to follow the script when they sign an agreement that allows them to lead CDSMP sessions, CDPHP decided that the presentation of material did not need to be assessed.  Instead, CDPHP decided to use between-session support provided by the buddies as an indicator for the quality of CDSMP implementation.

CDPHP developed questionnaires for participants as part of the program evaluation materials.  After completing participant feed-back questionnaires, which are distributed at the end of the last session, participants put them into an envelope that is then closed before it is returned to the leader, who then bundles it with other participant and class information and sends the whole package to CDPHP.

The questionnaire asks participants to rate their contact with their buddy(ies) from none (1) to frequent (10).  Between 3/1/06 and 12/31/06, forty-six participants completed the feed-back questionnaires, with a mean, or average, of 4.7 for this question, indicating a minimal use of the buddy system.

Once data have been collected, they must be analyzed and interpreted.

This is a disappointing result, since it suggests that these CDSMP participants missed important support for their behavior change efforts.

There are several limitations that prevent a more conclusive interpretation.  They include:

• Lack of a comparison.  Since there are no program standards for the buddy system aspect of CDSMP, there is no way to determine if the 4.7 result is lower or higher than or the same as other programs.
• These data could be biased.  They are self-reported, and answers could be affected by poor memories (which might reduce the rating) or by a desire to please (which might increase the rating) or other factors.
• These data were only collected from people who were at the last session.  There is no way to know if the respondents were more or less likely to have benefited from the buddy system than participants who did not complete a questionnaire.
• Only 46 people completed a survey, which means that very low or very high ratings by one or two individuals could have a substantial impact on the average.

The final step in the Framework is similar to the Action Plan component of the Results Accountability process.  It involves distributing the results and engaging stakeholders in using the results to move forward.

CDPHP wrote and distributed an evaluation report, including this finding, to all CDSMP master trainers and leaders in Alaska.  A report on this program, including all process evaluation findings, was also published as the first CDPHP Chronicle in September, 2007.

As a result of drafting this excerpt from the program evaluation, CDPHP has identified the need to take further action.  CDPHP will encourage master trainers to emphasize the added strength that the buddy system can lend to the success of CDSMP in their leader training sessions.  CDPHP will also use publication of the Chronicle to directly encourage leaders to promote buddy interaction.

Results from the first year of CDSMP program evaluation will be used as a baseline and compared with subsequent periods.