Pablo Antón Díaz

The First Commandment of Social Impact Generation: We need a better understanding of the nuances of creating social impact

The first commandment for companies and organizations that aim to generate social impact is “Thou shalt adopt mechanisms to measure it.” Every line is the perfect length if you don’t measure it, and this same rule applies to impact investing. Narratives on a handful of beneficiaries can be useful and serve as great instruments for attracting new investors, but the only sure way of knowing whether your efforts are indeed relevant in the lives of people is through systematic measurement.

However, despite its importance and increased relevance, this is still not a common practice in the sector. Instead, many impact reports focus narrowly on counting the total number of units distributed or the number of beneficiaries supported. This practice is insufficient and defeats the purpose of what a true “impact measurement” should be: a source of valuable insights about the benefits being transferred, their magnitude and their potential to expand.

Reports including data on the number of people reached and featuring heart-warming stories about a few cases where the beneficiaries’ lives were improved as a result of the organization’s intervention can certainly serve marketing purposes. However, they can also be misleading in overstating the organization’s true impact, since there is no way to distinguish whether the experiences of the people highlighted in these stories are shared by the majority of the targeted group — or if they are merely outliers whose outcomes surpassed the rest, either by pure luck or other unrelated factors.

But how can we find out if the rest of the beneficiaries are really better off because of us? Individual results may vary and every person’s experience is unique and inimitable. But is it at least possible to determine whether results were positive for a significant portion of those targeted? Understanding any outcomes that are attributable to our intervention is paramount, not merely for supervision purposes or as a marketing/funding tool, but also for conducting useful analyses, which then can be used to improve the effectiveness of our impact and our operating efficiency.

To accomplish this, we need to truly make an effort to understand the end beneficiaries and the different ways they interact with the product/service provided, instead of merely trying to count them (a modest first step).

For social enterprises, monitoring and evaluation methods can be classified into two opposing extremes. On one hand, we have management information systems with standardized metrics that serve as the source for the kinds of reports mentioned above. These instruments are typically limited in nature and rely almost exclusively on self-reported data from clients. As a result, in many cases, this methodology is barely suitable to accurately count the number of beneficiaries reached, let alone to make any claims about the benefits that were (supposedly) transferred to them.

On the other hand, we have impact evaluations and their trendier siblings RCTs (randomized control trials), which are generally considered the most credible means of accurately identifying the impact of any social intervention. During the last decade, RCTs have enjoyed broad acceptance as tools for ascertaining what works and what doesn’t in the development field, and for disproving myths and common preconceived notions. Their main drawback, however, is that they can be quite expensive and take several years to implement (and that’s even when everything goes according to plan).

The ultimate goal should be for impact measurement practices to serve as a middle ground between these two poles: a sort of sweet spot where we could rely on readily available information compiled on a rolling basis by portfolio companies, but still be able to say something insightful about the benefits we are transferring to people. Nevertheless, it seems that very few efforts are currently bridging this gap. To understand why, we need to take a closer look at the rationale behind the data that is being requested from investee companies, which focus mainly on tracking the use of funds. “How many lives touched?” reads the entry on the survey. Yet from an analytic point of view, responding to this question offers very few insights.

In an effort to find the middle ground, I traveled recently to Ecuador with the purpose of expanding the level of granularity in the performance reports we were getting from a financial institution (FI) in our portfolio that works with vulnerable populations in rural areas. After meeting with their accounting department and reviewing their internal databases, I was able to make an initial assessment on their reporting capabilities. I discovered a vast array of useful client-level data, which was unknown to us until that point, and could serve as complementary to better understand the effect of the FI’s operations. The FI fostered a very close relationship with its clients and therefore had more information on them and their economic activities than peer institutions, which turned out to be very useful in enriching our analysis at the outcome level. Additionally, because of this close connection, the FI could also easily ask clients for feedback after they repaid their loans, which could allow us to start compiling two data points for each borrower on relevant indicators such as income level and saving amounts. For their infrastructure projects, we were also able to obtain social data on the number of people who benefitted from each project, and in some cases, even data that hinted towards increases in education skills and job placement rates. We also were able to find some efficiency disparities between projects when we compared the different costs among projects that were for very similar uses and scales.

We might never use this outcome-level data to run a full-fledged impact evaluation, and will therefore never be able to really ascertain the causal link between the FI’s work and these observed results. Nevertheless, the fact that we were able to get this data in such a cost-effective way will nonetheless help make our assessment come full circle and improve our ability to evaluate this business’s social impact.

In a sector dominated by investment bankers and corporate finance specialists, it is important that we strive to promote a better understanding of the nuances and inner workings of creating social impact, and reject the view that it’s merely a ready-to-wear additive to usual business, even in those instances where the targeting was done correctly. After all, the harsh reality is that social impact is not created merely by reaching the poor, and counting the number of poor people reached is not enough to prove it.

We need to do a better job of comprehending the complex trade-offs between social and financial returns. We need to make sure that when our products/services reach these vulnerable populations, they do more good than harm in the long run. We also need to be certain that the benefits are sustainable. Finally, we need better proof of what works and what doesn’t, and we need to learn from our mistakes. This is still a nascent industry, and we have a long way to go.

In order to encourage discussion around this and other topics related to impact investing and the business and development potential of base of the pyramid markets, the IDB is holding the BASE III Forum on June 29 – July 1 in Mexico City at the Centro Banamex.

Pablo Antón Díaz is a Development Effectiveness Consultant at the Inter-American Development Bank’s (IDB) Opportunities for the Majority Initiative (OMJ).

Impact Assessment
impact measurement