NextBillion - An Initiative of the William Davidson Institute at the University of Michigan

Development organizations and donors sometimes measure impact by examining indicators such as poverty levels within the communities where they work, and asking themselves whether the project they ran had a positive effect on people’s lives. In other words, was the target population lifted — if only slightly — out of poverty?

But most of the time, answering that question requires the use of a household survey, which collects responses within generalized, predetermined categories, designed by outside technical experts. While this type of quantitative assessment has a place in the field, it maintains the traditional balance of power in development practice, because the outcomes being measured are determined and evaluated by outsiders, rather than the communities they serve.

But what if the script were flipped? What if the power to measure success was controlled by the project beneficiaries — or “clients” as we say at iDE — and not the organization that delivered the project? Instead of measuring a project’s impact based on its progress in reducing poverty, what if “prosperity” and all that it means within a community were a determining metric of project outcomes? And what if the target population itself made the decision about whether those positive outcomes had actually been met?

If we are to embrace the concept of locally-led development, it is not enough for international organizations to only listen to what clients or beneficiaries say they want from us. Instead, we must also measure the outcomes of those projects, according to what the clients themselves say is important.

The Need for Impact Measurement Based on Local Context

My organization, iDE, also known as International Development Enterprises, is a pioneer in the practice of market-based development. We believe in designing projects according to local context, rather than taking a cookie-cutter approach. Since 1982, we have increased incomes and improved the livelihoods of 41.9 million people by empowering local entrepreneurs and rural people to create market-based solutions to challenging social problems. And like many development organizations, we have long determined our impact by measuring income increases among the micro, small and medium-sized enterprises we worked with, as that is considered a key indicator of progress out of poverty. But these businesses have produced a multiplier effect at the household and community level that goes far beyond income alone.

That’s why, as we continue to evolve and innovate, we are expanding our data collection, moving beyond income alone, and extending our measurement approach to take in prosperity as well. To do this, we are first leveraging a range of globally validated tools that measure several impact areas, while reducing the data collection burden on the clients we work with. Second, we are centering our clients’ voices by using participatory, qualitative measurement approaches such as “Most Significant Change” to set our goals and measure our impact.

Developed by Dr. Rick Davies and Dr. Jess Dart, the Most Significant Change technique focuses on collecting stories from project participants, as opposed to examining predefined numerical indicators based on surveys. In these stories, project clients are asked to describe the significant changes they have experienced by participating in the project. Importantly, selection groups made up of a different set of project stakeholders then select the stories that most resonate with that community’s experience.

First applied in Bangladesh, Most Significant Change provides a simple way to collect and understand a large amount of qualitative information that delivers a rich picture of what is happening in a community — rather than a simplified picture where social and economic developments are reduced to numbers alone.

Putting the ‘Most Significant Change’ Method into Practice

iDE recently conducted a global review that adapted the Most Significant Change method to find out, in their own words, what the people and households we work with really want from our efforts. Conducted across nine low-income and emerging countries, the goal was to understand how to consistently make our work impactful, by gaining insights into how our clients defined prosperity and integrating those insights into our global and local impact strategies.

We asked people questions such as: “What change would you like to see in your life in the next five years?” “Why are those changes important to you?” “Would those changes make you feel more prosperous?” and “Are there any changes you would like to see in your community?”

All together, 249 stories (66% from women) were translated and reviewed. In them, people emphasized a desire to improve community infrastructure like transportation and access to electricity, to improve the business environment with fair markets and technology, and to become climate resilient through sustainable agricultural practices and preparedness for shocks. Among the people who told us their stories, 80% mentioned the role of agriculture, while 78% mentioned needs around finance, and 59% talked about gender equity and social inclusion. We saw consistent themes around what people wanted to use money for, such as sending their children to school and investing in their businesses, and also around visions for their community, such as improved healthcare.

As part of the evaluation process, iDE then gathered together community representatives — not just our staff, or outside experts or other stakeholders — to form the committees that selected the “most significant” stories, elevating those that aligned with their community values. This was the critical “participatory” element of the work, ensuring our approach was truly led locally. Armed with the knowledge about what our clients really want, we are now integrating their insights into our global measurement strategy.

Moving Past the Limitations of Traditional Impact Measurement

This type of approach also seems to be gaining traction among other development sector players, as the limitations of an exclusive focus on poverty in impact measurement have become clearer. For instance, much like iDE’s clients have voiced a desire for improved infrastructure, healthcare, education and social inclusion, a 2023 McKinsey Global Institute report proposed the need for a complementary benchmark to gauge progress beyond the $2.15 per day extreme poverty line. As McKinsey describes it: “We frame the empowerment line as the level at which people can meet their essential needs and realize more of their potential. It has a global floor of $12 per day in purchasing power parity (versus the extreme poverty line of $2.15 PPP).”

Like our clients, iDE believes that delivering and measuring empowerment and prosperity is a significant step forward. We encourage other organizations to experiment with participatory, locally-led approaches — and to use them not only to help elevate and center client voices in their strategy development, but also to guide their project design and measurement methodology. We also call on donors to fund projects that deploy this innovative approach, as the Canadian government recently did with our work. To achieve outcomes that are relevant to local contexts, we need to stop predetermining development outcomes, and let the people who matter actually participate in measuring them.

Henok Begashaw is director of inclusive monitoring, evaluation, research and learning at iDE (International Development Enterprises).

Photo credit: Michael Balinga/CIFOR