Heather Esper

The Best of 2013: Measure for Measure: Eight tips to optimize your data collection and maximize your impact

Editor’s note: As part of our Most Influential Post of 2013 contest, we are re-publishing the articles that attracted the most reads, social media shares and comments of the year. This article was the most-viewed for July. To see the full list of the most popular posts in 2013 and to vote for your favorite, click here.

So, you want to learn more about your impact.

Or maybe you are trying to get more information about the preferences of your customers at the base of the pyramid.

It’s time for you to run a household-level survey. But how do you get started?

Collecting accurate information via surveys requires developing both effective content and processes to capture data. The integrity of the data – that is, the accuracy and usefulness of your results – depends on the questions that are asked (the content) and the manner in which they are asked (the processes used to collect the data).

If you’re going to invest time and money in a household survey, you want to make sure that you dedicate significant time up front, perfecting the content and processes to make it worth your while. In addition, the experimental design might also be important to you if you are hoping to be able to attribute outcomes to your work. You’ll need to consider things such as the number of data collection points, and you’ll need to explore potential comparison groups.

Both the content and process are influenced by your target population’s characteristics – those from whom you seek to collect information. Characteristics such as their literacy level, culture and health can influence how questions should be worded and how information is collected. If low literacy is prevalent, then you will likely need to conduct the interview in person or over the phone. Culture influences what types of questions are appropriate to ask, and how to word questions to be more sensitive.

Below are some key considerations to keep in mind when developing your impact assessment content and processes using surveys:

Content Development

Define Your Objectives

Stay focused and only collect data relevant to what you intend to measure. Although it is tempting to explore all possible changes that might occur, it is likely you won’t use all the data. Collecting too much data can also create more work and expense due to additional data collection points, data entry and analysis. Furthermore, people may not be willing to devote their time to filling out a longer survey, which can negatively impact the accuracy of your data or result in incomplete surveys. Limiting the data you collect can help you stay focused on the most important information, and actually use it to benefit your company.

Determine Your Format

Different survey formats and types of data are better fits for different population characteristics, such as literacy. They also depend on your sample size, resources available, and the type of information you are trying to collect. Questions to consider when determining your survey format include: Will the respondents fill out the survey themselves, or will an interviewer ask the questions and fill it out for them? Although self-administered surveys are less expensive, they require a higher level of literacy and often can result in missing data. Will you use paper or electronic platforms for data collection? Mobile phone data collection is useful, but is usually limited to one to five questions, whereas paper data collection requires data entry. What type of data will you collect – qualitative or quantitative? For example, qualitative may be richer and capture things you may not have considered, but it requires coding and more resources to analyze. Quantitative data is fast and easy, but not as rich. Thinking carefully about the format will help increase the accuracy of your data as well as response rates.

Develop Your Questions

Questions should be clear and have comprehensive response options. Your target population should be able to easily understand your questions. At the same time, the response options should be comprehensive so there is a way for them to answer accurately. You also want your questions to have both reliability and validity. Reliability ensures the question is consistently understood – that a respondent would answer the same question the same way during the same time period. Validity ensures that the question measures what the researcher intended it to measure. Tests for reliability and validity can be complicated, so often it is best to utilize pre-existing questions – measures and tools that have been tested for reliability and validity. Using reliable and valid questions improves the accuracy of your data.

Design Your Instrument

A strong design will make the survey easy to follow and the respondent comfortable. It is important to give information on participation through a friendly introduction at the beginning of the survey that includes the reason for the survey, why you want their input, who you are, the survey process, how you will use their information, and your contact information if they have further questions. Starting the survey with easy-to-answer questions will build the respondent’s confidence and trust, while placing sensitive questions at the end tends to result in more accurate responses. It also is useful to provide guidance throughout the survey when the question topic or response method changes. A thoughtfully designed survey will help you not only get more responses, but more accurate responses.

Refine Your Instrument

Give yourself enough time for translation and pre-testing, and avoid the tendency to rush to collect data. In refining your survey, you want to make sure it says what you intend it to say. If you are translating the survey, a way to help ensure accuracy is to carefully review the translated instrument with a version of the survey that has been translated back to the original language. In addition to reviewing the translation, it is always valuable to pre-test with members of the target population to learn whether they understand the survey questions in the manner you intended, and the logistics of using the survey. The pre-test also allows you to further adjust the questions and process before you begin collecting data.

Process Development

The most carefully crafted survey instrument will not be useful without establishing an effective process for data collection.

Build a Research Team

Select and train a team to collect data. Potential options include leveraging existing staff, and hiring new staff or a third party. Experience and trustworthiness are important criteria to consider when making this decision. Once the team is hired, they should be carefully trained on the background of the research instrument and purpose, and become familiar with the instrument by conducting the survey on other interviewers first, and then on the target population. A well-trained and experienced team can contribute in a major way to enhancing the quality of data collected.

Ensure Optimal Participation

Establish protocols to maximize participation and reduce attrition at follow-up. Ensuring optimal participation means making the respondent as comfortable as possible, and making it as easy as possible for the individual to participate. Does the survey look official? Will you set appointments in advance to conduct the survey? Where can you conduct the survey with minimal onlookers? Will the respondent trust the survey team? What time of year should you collect data? What time of day should you conduct the surveys to maximize participation? Will you use incentives (payments, raffles)? If you are conducting a follow-up survey, how will you find respondents again? Thinking through these types of questions and addressing them in your protocol will help improve response rates.

Perform Data Management

Develop a strong plan for where data will reside and how it will get to necessary locations for processing. How will the data be safely transported and stored? How will you reduce data entry errors? Who will review, clean and prepare the data for analysis? After all the hard work of collecting data, you need to ensure that you can analyze and benefit from the information that’s gleaned from it.

Accurate data is essential to maximizing both your social impact and your bottom line. By fine-tuning the content and processes of your data collection efforts, you’ll set your business or organization up for success on both fronts.

Education, Impact Assessment
research, social enterprise