NextBillion - An Initiative of the William Davidson Institute at the University of Michigan

There is no shortage of evaluation frameworks/impact assessment tools out there for the BoP space – the Best Available Charitable Option (BACO), the Progress Out of Poverty Index and the Triple Bottom Line approach; and no shortage of discussion around metrics, with several posts on Nextbillion recently (here, here, here and, most recently, here). The quest to measure the social and environmental benefits of enterprises serving the base of the pyramid continues in 2010.

Last year, I asked the question ’Why should we evaluate social enterprises?’ In this post, I ask the question ’What should we try to measure?’

Some of these discussions have focused on the issue of outputs vs. outcomes, with strong arguments on both sides. Before presenting my own views, I would like to clarify the distinction between these often misunderstood concepts.

Outputs are the products and services that the enterprise delivers to its customers. Outputs are the direct result of resources and activities (such as number of products sold).

Outcomes are the changes in the socio-economic conditions of the customers and the changes in the environment that are a result of activities and outputs (such as a specific, measurable, improvement in health). Outcomes may be short term, intermediate or long term.

Outputs are certainly easier to measure, and easier to attribute to activities of an enterprise. But are they good enough? Should we strive to start measuring outcomes? Can we ever sufficiently attribute a social outcome to the activities of a social enterprise?

This discussion reminds me of my first class in Research Methods, when my professor started the class by saying ’You can’t prove anything with statistics’, perhaps the most important thing I’ve learned in research. The fact is, in the social sciences, you can never really ’prove’ anything. Even using the ’gold standard’ of evaluation design – randomized control trials (PDF link), it is simply impossible to completely attribute an outcome to a particular set of activities or outputs.

Digging a little deeper into the question of causal inference (PDF link) or how to prove that a particular activity and/or output X causes an outcome Y, we need to confirm 3 conditions.

• 1. Temporal Order: We must confirm that X occurs before Y
• 2. Observed Covariance: As X changes, so does Y. For example, when we sell more pesticide treated bed nets in a region, and we observe a lower incidence of malaria in the same region.
• 3. No Rival Explanations: There are no other potential explanations for the change in Y, other than X. Assuming the first two conditions are met, if we can be absolutely sure that nothing else could have caused the change in Y, we can be sure that X caused Y.

Condition 3 is where most theories break down, since it is impossible to completely rule out all rival explanations for a particular outcome.

The important thing to remember is that enterprises usually have control over the outputs they produce, through their activities (such as the number of products sold, or the number of employees hired). However, outcomes are usually also affected by a number of external factors, over which the enterprise may have little or no control (such as the weather, macroeconomic conditions, usage patterns or the political situation).

And this is the root of the problem of establishing causality. It is easy enough to count the number of products a company produces, but how do we assess the social or environmental outcomes of the product? There are potentially dozens of rival explanations for an outcome like reduced incidence of malaria in a region, so how can we conclusively attribute it to the sale of the bed nets?

We can’t definitively prove it. Development economics has been struggling with the issue of causality for decades and no one has a foolproof solution. However, there is a vast amount of literature that tries to address the issue in program evaluation through logic models, research design and statistical methods to rule out and control for rival explanations. By controlling for external variables, and carefully mapping the various factors that affect short term, intermediate term, and long term outcomes, we can begin to get a fairly good idea of how a particular activity or product contributes to an outcome. There are no certainties in this field; it’s all about ’getting to maybe’. Shadish, Cook and Campbell’s book ’Experimental and Quasi-Experimental Designs for Generalized Causal Inference’ is probably the most comprehensive compilation of these methods, but other excellent books to learn more are ’Outcome Based Evaluation’ and ’Utilization-Focused Evaluation’.

Let’s remember that this is not the first time people are struggling with issues like this. Many traditional non-profit programs and social enterprises ultimately aim to deliver similar social and environmental outcomes. An enterprise selling fuel efficient cookstoves wants to achieve the same social and environmental outcomes as a non-profit program distributing them for free. Frameworks, methodologies and indicators have already been developed to evaluate these outcomes. We need to study what has been done, see what we can learn, and adapt it for our purposes. The methods are not perfect, but they do exist, and are constantly improving.

A common refrain we hear is ’what gets measured, gets done’ and organizations often want to just start measuring something. However, if you measure the wrong thing, you’re likely to do the wrong thing. The BoP field is still at a nascent stage, and of course many enterprises and intermediary organizations lack the capacity to track long term outcomes. But as Kevin Starr of Mulago (PDF Link) asked in this debate, “Why would you fund something that you don’t think causes outcomes?”. Ultimately, we’re trying to achieve outcomes, so let’s think about measuring them.