Exactly what data do you need to collect to be able to answer the evaluation questions?

Your evaluation results need to be believable, trustworthy and relevent to stakeholders.  If you involve stakeholders during the planning stages of your evaluation, it will be easier for you to make sure that the data you collect will support results that are valued by your stakeholders.

Another important way to increase trust in your evaluation results is to use a variety of data collection methods, analyses and interpretation strategies.

Data considerations:

• Indicators
• Sources
• Quality
• Quantity
• Logistics
• Analysis and Synthesis

Indicators - Which specific, measurable characteristics of your program activities or effects are you going to use in your evaluation?

The best choices will show change within the time period or scope of your evaluation and give you meaningful information for your stakeholders.  Your logic model may help you identify your evaluation indicators.

Sources - What data sources do you have for each of your indicators?

Progress toward many of your indicators could be measured using data from more than one source.  You could also use different kinds of data, depending on the sources you have.  You can increase the credibility of your evaluation by using more than one data source for your important indicators.

Participants, observers and program records may be useful resources.  Once again, your logic model may help you identify data sources and measures that would be useful.

In the end, the measures that may work best for you may be the ones that use data you have or could easily obtain.  Other factors to think about when selecting measures include:

• Which ones would make the most sense to your evaluation users
• Which measures best fit the program activity you are evaluating or the effects you expect it to have, and
• Any characteristics of the people involved that might affect their response to the measure (for example, family income questions might be offensive to some).

Quality - Will you and your evaluation users be able to trust your data?

If you have good data, your evaluation will be useful even if people disagree with your results.  On the other hand, bad data will undermine your entire evaluation effort.  All of the data you collect need to have a clear, specific, anticipated use in your evaluation.

Quality is judged differently for quantitative and qualitative data.

Quantitative data should be:

• valid - they need to match what you want to measure.  For example, you could use the number of cigarettes sold in a community to assess demand for tobacco, but not (by itself) to measure the number of smokers.
• reliable - they need to give you consistent answers.  For example, imagine giving the same pre-test to workshop participants when they registered a week before it happened and again just before you started the workshop.  If their answers were really different between the two times, it would be hard to say what effect your workshop had.  In that case, your pre-test data would be unreliable.
• objective - although you might hope for certain answers, you need to collect data with a completely open mind.

Qualitative data should be:

• credible - your results need to be believed by your evaluation users and the people you ask for comments or stories.  This trait is similar to validity for quantitative data.
• confirmable - other people need to be able to produce similar results, if needed.  This trait is similar to objectivity for quantitative data
• dependable - qualitative data are very sensitive to the environment in which they are collected.  This trait is similar to reliability for quantitative data, and reflects the importance of recording changes in the surroundings that might affect them.
• transferable - this trait is especially important if you hope to match or use someone else's qualitative data results in your evaluation.  In that case, pay close attention to their description of the context of their data collection and the assumptions they made.  This trait is also similar to validity for quantitative data.

Quantity - How much data do you need to collect?

This question suggests that you do not need to include everyone in the group when you collect data.  This is a good thing, especially when you have large groups!

If you do not include everyone, you will need to choose a sample of the entire group you want to learn from.  Your sample must be:

• representative - it needs to match your larger group as much as possible
• frugal - collecting data takes time and resources, both from you and from your data sources.  Work to keep demands by your data collection process as small as you can.
• planned - before you start collecting data, you need to estimate how much data you will need and decide how you will know that you have enough.

Qualitative data numbers depend on how much you think will be required to completely explore the topic.  Qualitative data tends to contain a lot of information per person, so they are often collected from a fairly small number of people.

Quantitative data numbers depend on the level of statistical power you would like to achieve.  When you collect quantitative data from a sample, there is a chance that the results will be different than they would have been if you had obtained data from everyone.  Statistical power refers to the number of records needed to be sure that the true result is within a certain range (or confidence interval).  Generally, you will need more records if you want a smaller confidence interval, and you are more likely to observe change if you have a small confidence interval.  This is a straight-forward calculation, and there are websites that will do it for you (e.g., Sample Size Calculator).

Logistics - What tools will you use to collect data, and how will you use them?

Collecting good data involves using well-designed tools consistently and carefully.

Carefully designed data collection instruments (e.g., surveys questionnaires, interview guides, or forms for recording observations or program records) will reduce the number of errors and misunderstandings that may occur.  Your instruments will be more successful if they are:

• pretested - a really good way to find out if your instrument will collect the data you want is to ask people to try it out for you.  This is particularly important for quantitative data questionnaires, although it is always a good practice.  Your ideal questionnaire testers would be people who are very much like the group who will be involved in the data collection, but are not part of that group.  You need to find out if:
  • they understand your questions the same way you do;
  • you are asking too many questions or you have missed an important question; and
  • you are asking for information that makes them uncomfortable
• easy to use - your instrument can help the people using it to give you the data you want.  Design errors can cause any number of problems if people make mistakes in recording their answers or miss part or all of some questions.  Pre-testing is also a good way to find out if the questions are well-organized.  If you are collecting data during interviews, from records or by observation, the instrument should be tested by the people who will be using it.

The procedures you use for data collection will also have a lot to do with your data quality.  You need to specifically define how:

• you will protect the confidentiality and privacy of the people giving you data while they give you information
• you will assure the security of the data you collect
• you will identify errors during the data collection process
• when you collect data, including how many times you will ask someone to answer your questions, about how many hours you will spend collecting data, and how many days, weeks, months or years will be involved.
• you will address the beliefs that all communities, organizations and families have about who should answer questions about them

Analysis and Synthesis - What methods are you going to use to interpret your data?

Another critical step forward in the evaluation process is thinking about what your data might tell you.

The first part of this process is analyzing the data you collect.  Key questions at this point are:

• How are you going to organize and group your data?
• What calculations will you use with your quantitative data?
• How will you look for ideas or key statements in your qualitative data?

If you keep your stakeholders in mind when you design your analysis, you will be more likely to produce results they believe and can use.  Complex data calculations may not be as understandable as those that require less explanation.

The next part is to determine how results from multiple data sources support or conflict with each other.