Evaluation that Creates Value for Participants: A Client-Centric Approach
Let’s avoid ‘extractive’ methods of data collection
Do market-based approaches to development work? Are disadvantaged populations in fact better off as a result of their interactions with the constellation of social enterprises, impact investors and philanthropists that have arisen in the past decade?
As NextBillion readers know, a reasonable way to try to answer this question is to conduct an impact evaluation, in which researchers survey people about the changes that they have experienced as a result of a particular intervention. Yet the very methods of evaluation threaten to re-entrench the power dynamic that a market-based approach to development seeks to mitigate in the first place.
Too often, the process of collecting data to evaluate impact, regardless of the intent, feels extractive to the research participants. Extractive industries are those that obtain natural resources from the earth without provision for the potential negative consequences of extraction. Similarly, some evaluations extract data from disadvantaged communities without providing any benefit in return. Such evaluations reinforce real and perceived imbalances in power and opportunity between the people doing the research and the people being studied.
A mango farmer Root Capital surveyed in Burkina Faso in 2011 expressed it best:
“Here you come to ask us the same silly questions that you go sell to aid sponsors. Now when the aid comes you keep it for yourself. I don’t want to answer any question. Go take the answers for the ones we provided last year [to a different surveyor unknown to Root Capital]. … You’re all crooks of the same family. You’ll ask me my name, my family size, the kind of goods I have, and so on and so on. I am tired of all this and I am not answering a question, nor will anyone else in this family.”
This quote comes from the first study that Root Capital commissioned to understand the impact of our lending. We hired a Burkinabé anthropologist to conduct qualitative interviews with farmers. His report included his perspective on our impact on farmer livelihoods. But what stuck with us was his account of his personal interactions with those farmers while conducting the research. Indeed, we heard similar responses from other farmers in this community and in other communities where we have conducted research.
With each subsequent impact study, we have sought to increase not only the quality of research about impact but also the value created for our clients. Our working paper on “A Client-Centric Approach: Impact Evaluation That Creates Value for Participants” describes the general principles, practical tips and case examples that we have compiled from more than 20 impact studies conducted over the past five years.
Our intent is not to coin a new term or invent a new impact framework or methodology. Rather, it is to honor the rights of the business leaders and small-scale producers who participate in our impact evaluations, and find creative ways for evaluations to deliver more value to them. It is to ensure that the data collection, analysis and reporting process benefits those being researched, rather than simply extracting information for the benefit of third parties. By doing so, we hope to significantly increase the value of the research, notably to participants, without proportionately increasing the cost.
The rub is that these observations about the power dynamics inherent in evaluation are not new. Indeed, they are decades old, and spurred the development of evaluation methods such as participatory rural appraisal (PRA), rapid rural appraisal (RRA) and participatory analysis for community action (PACA). These techniques are taught in universities, albeit only to a small number of students. The American Evaluation Association has an interest group in Collaborative, Participatory & Empowerment Evaluation, for example.
And yet, much evaluation done today still feels extractive to participants. I have told the above anecdote of the mango farmer several times to international development and monitoring and evaluation professionals. Occasionally there is uncomfortable silence but more frequently heads nod sagely around the room. This is a scenario many evaluators have experienced firsthand or heard about from their colleagues. A couple of years ago, Howard White, then-executive director of the International Initiative for Impact Evaluation, asked in a blog, “Is impact evaluation of any use to ‘project beneficiaries’?” You can probably guess the answer, but his post is well worth reading nonetheless.
The challenge is not to merely agree in principle. Few if any disagree with the idea of making evaluation more valuable to participants. The challenge is to get better at translating our principles into practice.
This is how I hope the lean research movement can help those of us engaged in research and evaluation among disadvantaged populations: first, to raise awareness of the challenge and advocate for making evaluation valuable for participants; and second, to lower the barrier between intention and action by providing practical, field-tested methods for doing so. Through the Lean Research Hub we look forward to learning from and sharing with a growing community of practice that shares these goals.
Images courtesy of Root Capital.