What do you really want to learn from this assessment? A program evaluation
design identifies:
A logic
model can help you respond to design questions related to evaluation
reasons, questions and methods.
A complete evaluation design also includes specific agreements that include the
answers to these questions and define the roles and responsibilities of all involved.
Evaluation users - Who will use the evaluation results?
The people who will use the evaluation are a sub-group of your program’s stakeholders. The
evaluation results must meet their needs to be useful, so they need to be involved in
designing it.
Evaluation uses - How will the evaluation results be used?
An evaluation is effective when it is used in program decisions. Different evaluation
users may use the evaluation for different purposes, which should be addressed in the design.
Evaluation purpose - Generally, why are you evaluating now?
Program evaluation often corresponds to a need to find out:
- About the quality of program activities. Also, are they effective and efficient?
- Does it make sense to change program activities?
- Are program activities having their expected responses? (Only appropriate for mature programs.)
- About the continued value of program activities for stakeholders.
Other reasons to conduct an evaluation are to enrich trust among participants, develop
program evaluation capacity and provide a foundation for self-directed change,
Evaluation questions - What specific questions will the evaluation answer?
- What do you want to learn?
- Which components of your logic model will be assessed?
- What period of time will be used?
This is an important step toward a feasible evaluation design. Stakeholders
will have different interests and priorities for the evaluation results; and you
will not be able to address them all. If you can get key stakeholders to
agree on which are most important, your evaluation will be more successful.
Evaluation methods - What data collection strategies will you use?
You need specific information to answer your evaluation questions. The data collection
methods you use must meet your information needs and be feasible with the resources you have available.
What kinds of comparisons will you make?
- If the evaluation question concerns
outputs
(e.g., products or services), you may be comparing what was done or provided
against a defined
standard.
- If an evaluation question is about
outcomes
(or what effect the outputs had), you may be looking for change. Change may be
harder to measure than outputs. It can be very difficult to prove that a
product or
service
caused a difference that you might find. Logic models or other methods for carefully explaining
how your program is expected to cause certain changes will help.
What kind of data do you need to collect to answer your evaluation questions?
- Quantitative data can be counted: such as the number of visits, or the number of
people with blue eyes, or the number of minutes of physical activity.
Quantitative data are relatively easy and less expensive to collect and analyze.
They are simple to use and understand.
- Qualitative data are people's own words or stories; they are collected and used for
exploring opinions or attitudes. Collecting and analyzing qualitative data
generally takes much more time than quantitative data, which often means that
information is collected from fewer people. Qualitative data are particularly
helpful for understanding complex issues, such as why people change behavior or how
they feel when they heard a certain message.
How will you collect the data you need?
- Common strategies include surveys, focus groups, interviews, observations and program data.
Since each evaluation method has strengths and weaknesses and
because change is hard to measure, it makes sense to use several strategies to
assess most evaluation questions. As your program and your evaluation
progress, you might find that different methods work better than the ones in your
design. Use the methods that work best to answer your evaluation questions.
Evaluation agreements
Review your evaluation design and its expected products, timeline and budget with
your key stakeholders. Work with them to develop a contract, memorandum
of understanding, or detailed protocol that includes everyone's roles and
responsibilities and their commitment to provide the support and resources
that you will need for your evaluation.
