Glossary of evaluation terms and concepts:

Evaluate: 1. To determine or fix the value, 2. To examine carefully. (Webster's II New Riverside University Dictionary ©1998; Houghton Mifflin Co.)

Types of evaluation:
      Program evaluation has different names, depending on its purpose and context.  Different people may use different terms with similar meanings.  Formative and process evaluations have a roughly parrallel purpose, as do summative and outcome evaluations.

Formative evaluation asseses the worth of a program while the program activities are forming or happening.  Formative evaluation focuses on the process.  (http://www.sil.org/linguaLinks/literacy/ReferenceMaterials/GlossaryOfLiteracyTerms/WhatIsFormativeEvaluation.htm, printed 9/19/07)

Outcome evaluation looks at impacts/benefits/changes to your clients (as a result of your program(s) efforts) during and/or after their participation in your programs.  Outcome evaluation may examine these changes in the short-, intermediate- and/or long-term.  (United Way; http://www.managementhelp.org/evaluatn/outcomes.htm, printed 9/19/07)

Process evaluation considers the extent to which your treatment service or system is serving the people for whom it was intended, as well as program operation and delivery.  (WHO; http://whqlibdoc.who.int/hq/2000/WHO_MSD_MSB_00.2e.pdf, printed 9/19/07)

Summative evaluation determines the worth of a program at the end of the program activities.  The focus is on the outcome.  (http://www.sil.org/linguaLinks/literacy/ReferenceMaterials/GlossaryOfLiteracyTerms/WhatIsSummativeEvaluation.htm, printed 9/19/07)

Robert Stakes "When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative."  (http://jan.ucc.nau.edu/edtech/etc667/proposal/evaluation/summative_vs._formative.htm, printed 9/18/07)

Trochim WMK.  Research Methods Knowledge Base
http://www.socialresearchmethods.net/kb/intreval.php - a nice discourse on evaluation types and the kinds of assessment activities that fall under the formative and summative evaluation headings.

Measure: n. ... 5. A device, as a marked tape or a graduated container, used for measuring; 6. An act of measurement; 7. A basis of comparison: CRITERION ....

v. 1. To determine the dimensions, quantity or capacity of.... (Webster’s II New Riverside University Dictionary ©1988; Houghton Mifflin Co.)

Types of measures:
      There are also lots of terms for the key criteria used in program evaluation.  Although there may be some variation, these concepts may be used in all kinds of evaluations, regardless of the purpose or context of the assessment.

Activity for evaluation purposes means a measurable component that is produced by the program being assessed, such as a meeting.  Products and services may be other measurable components.

Benchmark is a comparison value used to assess progress on a certain measure.  It is often the starting point, or baseline.  For example, one Healthy Alaskans 2010 objective is to increase the proportion of adults aged 18 or over with diabetes that have at least an annual foot examination; its 1999 baseline is 79%.

Credible

Confirmable

Dependable

Goal is a broad statement about what a program hopes to achieve.  A well-written goal is describes the change that will have happened because of the program’s success.  For example, one of the goals of the Alaska Tobacco Prevention and Control Program is to eliminate exposure to environmental tobacco smoke.  (A good guide for developing goals and objectives is: http://www.cdc.gov/dhdsp/state_program/evaluation_guides/smart_objectives.htm, printed 9/19/07)

Impact describes the anticipated long-term change that happens in a community because a program has been successful.  For example, if a program successfully increases the proportion of adults that exercise regularly from 25% to 50%, one long term impact might be that more children and teens exercise regularly as physically active family events become more common.  In logic models, impact describes the furthest ripples that correspond to a program’s outputs.

Incidence is a measure used in public health.  It is the number of newly identified individuals with a certain characteristic divided by the total number of people that could have had that characteristic.  For example, 630 of the 10,372 babies born in Alaska in 2005 were low birthweight, for a low birthweight incidence rate of 6.0%.

Indicator is a specific characteristic or event that is used to assess change.  Well-defined indicators have a self-evident connection to the program objectives, measure an important aspect of program success, focus on something that can be changed, and are easily counted.  For example, one indicator for the severity of diabetes complications is the number of reported lower extremity amputations in a certain period of time.

Input refers to the specific investments made by a program or agency to generate the activities, products or services that will cause a result.  Inputs are often personnel time, financial support or raw materials.  Logic models often use inputs, activities, outputs, outcomes and impact to organize and specify assumptions about a program.

Instrument

Measure refers to the specific characteristics or events that are used to describe a program.  Some measures are indicators, but others might be used to identify a program’s target population or the community in which it takes place.  One of the measures monitored by the AK Diabetes Program is the percentage of adult Alaskans with diabetes who report having had a flu shot during the previous year.

Logic models link program investment (inputs) to products (outputs, including activities, services or materials) to anticipated change in the target population (outcomes); models may extend to anticipated change in the larger community (impact).  Logic modeling can be a very useful tool for clarifying assumptions about what would happen between steps, for developing evaluation questions, and for identifying indicators.  There are several websites with good information on how to develop and use a logic model ([logic model tools]).

Objectives are the stepping stones toward a goal.  Well-written objectives are very specific statements about who will be involved, how much of what will happen and when it will be completed.

SMART objectives are specific, measureable, achieveable, relevant and time-limited.
Generally speaking, SMART objectives make program evaluation much easier.
(see: http://www.cdc.gov/dhdsp/state_program/evaluation_guides/smart_objectives.htm, printed 9/19/07)

Objective data

Outcome is an anticipated result of program activities.  Most outcomes are short- (less than six months) or intermediate-term (one to five years).  Generally, logic models use outcomes to describe anticipated changes in a program’s target population.  Long-term outcomes and impacts may refer to changes anticipated in a similar period of time (more than five years).

Outputs are the actions taken by a program that are intended to cause a desired change.  In logic models, an output may be an activity, service, or product, such as patient education materials.

Pretest

Prevalence is another measure used in public health.  It refers to the total number of people who have a certain characteristic as a proportion of the total population, regardless of when they started having it.  For example, 731 of 2778 Alaskans said in 2005 that they had ever been told by a doctor that they had arthritis for a prevalence of 23%.  (Health risks in Alaska among adults.  http://www.hss.state.ak.us/dph/chronic/hsl/brfss/pubs/BRFSS0405.pdf, p 27 (printed 9/21/07))

Product for evaluation purposes means a measurable component that is produced by the program being assessed.  One of the products of the Alaska Arthritis Program is the Alaska Arthritis Resource Guide.  Activities and services may be other measurable components.

Qualitative data are rich in detail and description, usually in a textual or narrative format.  Examples include data from open-ended survey questions (e.g., “How should we solve this problem?”) or collected during conversations.  (http://www.uwex.edu/ces/tobaccoeval/glossary.html (printed 9/21/07))

Quantitative data are counted and used in calculations.  Examples include data collected from yes/no questions on surveys or reports on the numbers of people served.

Rates are used in public health to compare populations; they are important because they use denominators.  For example, the impact giving four people an influenza vaccine is very different when the group that could have been immunized includes 40 individuals (10%) or 200 (2%).

Representative

Reliable

Samples are used when collecting information from everyone in a group would be too difficult or too expensive.  The sample should be the same as the entire group, but smaller.

Sample frame is the group from which a sample is taken.  All of the characteristics used to define this group should be included in the strategy used to select the sample.

Sampling refers to the selection process.  There are many, many good discussions about how samples might be selected, including Probability Sampling: http://www.dsf.health.state.pa.us/health/cwp/view.asp?a=175&q=202058

Service for evaluation purposes means a measurable component that is produced by the program being assessed.  Activities and products may be other measurable components.

Statistical power

Statistically significant

Target is another comparison value used to assess progress on a certain measure.

Transferable

Valid

Value is the quantity of an evaluation measure at a particular time.  It may be a percentage, a number or even a date.

Other online glossaries of evaluation terms:

Local Program Evaluation in Tobacco Control Glossary of Evaluation Terms.  University of Wisconsin Cooperative Extension Program Development and Evaluation http://www.uwex.edu/ces/tobaccoeval/glossary.html (printed 9/21/07)
    This webpage uses straight-forward definitions and examples

The Evaluation Center.  List of Evaluation Glossary Terms.  Western Michigan University.  http://ec.wmich.edu/glossary/glossaryList.htm (printed 9/21/07)
    There are 571 terms in this one, which seems to be intended mostly for educators.

Program Evaluation Glossary.  Environmental Protection Agency.  http://www.epa.gov/evaluate/glossary.htm (printed 9/21/07)
    The reading level on this webpage is pretty high, but it may contain terms not found on the others.

Center for Program Evaluation.  Glossary.  US Bureau of Justice Assistance.  http://www.ojp.usdoj.gov/BJA/evaluation/glossary/index.htm (printed 9/21/07)