Research that Embraces the Messiness of Real Life: How an Innovative, Tech-Enabled Survey Method is Revealing the True Feelings of the Bottom Billion
Of any population in the world, the poor are the least likely to feel free to speak their minds. That raises several challenges for the researchers, enterprises and organizations that use surveys to better understand and meet their unique needs. Too often, open-ended survey questions produce responses that are short, dispassionate, embarrassed or confused. The thornier the topic, the greater the complexity of the response. But though many surveys are designed to avoid or minimize this sort of ambiguity to facilitate data collection at scale, not embracing this “messiness” of real life confounds our efforts to positively impact the lives of the bottom billion.
New tech-enabled research methods may provide a solution to this dilemma by enabling the capture and analysis of open-ended responses — even at the same scale as large surveys.
A Tech-Based Approach to Survey Research
Recently, Fundación Capital teamed with Daryl Collins, author of “Portfolios of the Poor,” and together we put these methods to work to assess the impact of a messaging campaign Fundación Capital had conducted in Paraguay. The campaign included a WhatsApp-based digital advisor, which provided messaging intending to shift gender norms about traditional gender roles, gender violence, child labor, child molestation and the use of contraceptives among low-income women in the country. The digital advisor provided text messages, infographics and short videos via WhatsApp to women enrolled in Abrazo, a national program created in 2005 by the Ministry of Children and Adolescents in Paraguay. Abrazo was created to address the root problems of child labor and street children by providing cash transfer, childcare and other programs for mothers living in poverty. Thanks to support from the International Development Research Centre, Fundación Capital has been supporting the program’s socioeconomic component with the implementation of a gender-transformative poverty graduation program, focused on Abrazo’s women participants. By the end of 2020, we were able to run an early assessment to understand how recipients were reacting to the messaging campaign.
Daryl and her team developed a short questionnaire using a WhatsApp survey application called Outside Voice. This application sends out open-ended questions and allows respondents to send back an audio file recording their responses. The respondent can talk as long as she wants — in fact, the questions and approach are styled to encourage her to talk as long as possible, to generate more data. In this case, because the digital advisor messaging is also delivered via WhatsApp, the women were already used to receiving materials from the digital advisor in this way.
Analyzing the Data to Uncover the True Meaning of Women’s Responses
The first step in analyzing this data was for a professional translator fluent in the languages of Paraguay — Spanish and Guarani — to convert the audio files to text. Then, Daryl’s team used a two-stage Natural Language Processing (NLP) method to detect key topics and phrases to unpack the meaning of the women’s responses. Using these methods, we identified three to four topics and patterns in each of the 10 responses. It is important to emphasize that this method allowed for spontaneous responses from women, rather than a pre-determined and closed-ended set of responses. We then used speech signals extracted directly from the audio files. These speech signals (such as pitch, duration of responses and voice modulation) told us whether the woman responding was engaged in her responses, when she was unsure, and when she gave us a “canned response” — i.e., telling us what she thought we wanted to hear. The NLP process told us what women were saying while the speech signals told us how they were saying it.
Together, we were able to generate segments of responses that unpacked the “grey areas” of each of the 10 topics. For example, the chart below shows some of the responses to a question about child labor. If we were to ask the question in a closed-ended way using a Likert scale, most would have agreed that children should not work. We therefore would have concluded that attitudes towards child labor were no longer an issue in Paraguay. However, using these other methods, we were able to produce a more nuanced set of responses. Just under half the women interviewed had extraordinarily strong responses — not only did their speech signal strong engagement, but they also would name three to four varied reasons why they thought the way they did. But a sizable group gave a “canned response,” which suggested that their true feelings were uncertain. Another small group was uncertain about the ages where different work was appropriate, including whether children helping mothers in their businesses or jobs can be categorized as child labor. These tended to be older women, which makes sense as the labor laws in Paraguay have changed in the past decade. A final group was defensive: They often suggested that their children wanted to work and they, as mothers, had no control over them. These results suggested to us that we could make changes in our messaging to be clearer about age-appropriate tasks and child labor.
Maximizing the Benefits of this Assessment Approach
One of the benefits of this assessment approach was the positive reaction of the women. We designed the assessment in a way that created greater affinity and trust in the digital advisor: To that end, Fundación Capital created characters for these advisors, to make them more relatable. The digital advisor characters that delivered the gender norms messaging were called Luci and ConHéctor. To make the research appear seamless to the respondents, we also created a character named Yoli to play the role of the “interviewer,” describing her as a friend of Luci. One of our colleagues provided the voice of Yoli, recording the questions sent to the women in the survey using a very localized language and a friendly tone. We also created a visual persona for Yoli, below.
Click here to listen to Yoli introducing herself to respondents over WhatsApp
From the respondent’s point of view, it made sense that first Luci was sending them materials as an advisor and then Luci’s friend, Yoli, was asking them what they thought. At the end of the survey, many respondents said they were pleased to be consulted.
This study was done with both women who received messages from the digital advisor and those who did not, as we wanted to understand some of the early effects of the digital advisor initiative on gender attitudes. Preliminary insights show that women who received the digital advisor messages tend to have more progressive views around gender roles, child labor and the use of contraceptives. However, in the case of gender-based violence we saw that despite a general abhorrence of gender violence, many women are unsure of what to do when they witness an act of violence. Likewise, there is still a significant degree of victim-blaming.
We believe there is potential in these methods, as they can not only pick up greater nuance but can also be done at a low cost and as a remote alternative to face-to-face data collection during COVID-19.
Next, we will be working on implementing the same method in Colombia, where Fundación Capital is executing another digital advisor program. We’re looking forward to continuing to develop this approach, and to utilizing the sometimes messy — but always illuminating — responses it provides.
Daryl Collins is an Independent Consultant and Senior Advisor, Carolina de Miranda is the Director of the Social & Livelihood Promotion Department at Fundación Capital, and Laura Morínigo is a Project Manager at Fundación Capital.
Photo courtesy of Caribb.
- Impact Assessment, Technology