NextBillion - An Initiative of the William Davidson Institute at the University of Michigan

Even organizations that are well-intentioned and carry out an impact assessment can end up with unusable data due to implementing a poor methodology. For example, a review of the United Nations Children’s Fund (UNICEF) found that many of their evaluations could not assess impact due to methodological shortcomings.^1,2 Not only were these projects not able to demonstrate any impact, they also wasted time and money. When developing a rigorous impact assessment methodology, it comes down to being an educated user. No methodology is flawless, but you must understand the effects of the decisions you make regarding methodology on the data you will collect.

Last week during our BoP Impact Assessment Workshop at the William Davidson Institute, we discussed some of these issues around measuring impact with participants from around the world and various fields including the private sector, non-profits and academia. Our main objective was to have participants leave the workshop with an action plan to bring back to their organizations on how to implement the BoP Impact Assessment Framework. During the workshop participants developed a strong understanding of the framework, performed a strategic analysis of their organization’s impacts, as well as learned and applied key issues in developing and implementing a robust data collection strategy to their organization. This last aspect highlighted the importance of developing a rigorous methodology to collect impact data at the project level.

So how do you successfully collect impact data at the project level? You must carefully consider several issues around research design, content development, and a data collection process. If one of these components is not done well, then the integrity of the data can suffer.

When thinking about research design, there are a few key decisions to make.

• Will you use a comparison group? If not, will you be able to ensure that the impact you track is actually due to your venture and not something else such as a new law being passed?
• How many data collection points will you use? If you want to show change over time, it is important to consider collecting data before the project affects your potential subjects and again after those affects are measurable.
• How many people should you collect data from in order to be sure you can make conclusions about your data, i.e. will your data be statistically sound?

There are also multiple things to consider when developing the instrument you will use to collect data.

• What impacts do you think are the most important to measure? You want to ensure you don’t burden your respondents with too many questions.
• How are you going to administer the instrument? Using the telephone, mail, in-person, internet or a PDA to administer your survey all have implications on your response rate, and the resources that will be necessary.
• Are your questions are reliable and valid? It is important that you ensure your questions are reliable in that they produce the same result when used repeatedly, and valid meaning that the questions measure what they were intended to measure.
• Will you back-translate the instrument? Translating your instrument into the local language, and then back to the original language allows you to compare two different versions for discrepancies.
• How will you pre-test the instrument? Will you use share the instrument with experts in the field for their feedback, will you test your survey’s content and process with the target population?

Even if you develop a strong research design and survey instrument, you could end up with unusable data due to how it was collected. Some things to think though when developing your data collection process include:

• Who will collect the data? Your results may be questioned if your organization collects the data instead of a third party. If you use in-person interviews, the gender of the interviewer may influence the respondent’s answers.
• When should you collect the data? The time of the year, such as crop harvests may influence the responses.
• Where should you collect data? The environment that you collect the information may also influence responses, such as having on-lookers while the survey is being conducted.
• How will you ensure that the respondent’s answers remain confidential?
• How will you find the respondent again at later data collection points?

Despite these many considerations when developing a rigorous methodology to collect impact data at the project level, it isn’t difficult. The key is understanding and carefully thinking through each of these issues to inform your decisions around your own methodology.

Footnotes:

¹Victoria, C.G. 1995. “A Systematic Review of UNICEF-Supported Evaluations and Studies, 1992-1993.” Evaluation and Research Working Paper Series 3. United Nations Children’s Fund, New York.

²Savedoff, W., R. Levine, et al. (2006). When Will We Ever Learn? Improving Lives Through Impact Evaluation. Evaluation Gap Working Group. Washington, DC, Center for Global Development.