A Shared Destination: Why we need open source impact metrics
Editor’s note: This is the third in a series of three articles offering new methods in generating, comprehending and acting upon impact metrics. Check out part one and part two.
The definition of social impact dramatically differs based on the role we are referencing, e.g. impact investor, funder, program agency or practitioner. Though there is a plethora of standards, tools and frameworks, most organizations do not have the necessary resources or skills to navigate the maze of Theory of Change (TOC), Logic Model, Social Return on Investment (SROI), outcome management tools and standard metrics.
Despite all of these options, many program managers are not clear about:
- How to demonstrate “impact evidence” that fits well into a larger impact narrative
- How to evaluate a program with a higher level of confidence
Frameworks like SROI and TOC offer important strategy tools, yet organizations constantly find themselves stitching together many tools and frameworks to have a more complete and accurate understanding of impact. As a result, many measurement and evaluation (M&E) practitioners are increasingly turning to randomized control trial program evaluation. But even this approach has many pitfalls, such as accuracy of a counterfactual, time and costs – among other factors.
The objective of this post is to describe some of the best practices in effectively building a social impact evidence system.
Build upon collective impact where possible
An article from The Stanford Social Innovation Review makes a compelling case for a collective impact and identifies five major factors in the success of such a program. In my previous article, I applied this concept to the HomeKeeper case. Now that we have established that there is a need for collective impact and we reviewed a few similar examples, let’s summarize the key principles of this concept.
The collective system may have many players. However, similar players with comparable activities or goals must have unified metrics. Programmatically, this means that every initiative or product with similar goals should inherit one or more metrics clusters. Having established standard metrics allows a system owner to slice and dice a database according to well-defined criteria. For example, under this concept, a microfinance program not only would be able to apply the same standardized metrics to all of its partners, but also compare its results based on region, country, organization type, client type, customer model, beneficiary type, etc.
Build, test and share your metrics
When global and community organizations have spent thousands of hours refining their metrics, they should be in the position to share them openly for the common benefit. That way, other organizations won’t waste their limited resources reinventing what is already out there.
With that premise, the developers of impact measurement systems should provide an open repository of metrics, and allow other users to customize them to their own context and needs. Such systems also would allow users to easily frame goals and build standard-based reports accordingly. Only with shared best practices and collaboration among the organizations will the social sector be able to produce a higher impact.
Related:
BUILDING EFFICIENT SOCIAL SECTOR INITIATIVES THROUGH NEW GENERATION IMPACT MEASUREMENT
BUILDING A COLLABORATIVE SOCIAL IMPACT MEASUREMENT OPERATING SYSTEM
Goals also can be shared among social sector organizations. For example, the 17 Sustainable Development Goals are mainly thought to be adopted at the global and, in some cases, national level. These goals can be rolled down by agencies to be implemented at the regional and local level, defining more granular metrics and indicators.
The challenge with the current impact measurement tools is that most of them only focus in one framework at the time (e.g. TOC, SROI, Logic Model, etc). Currently, organizations engage practitioners who can help them figure out the proper way to use them and combine them, in order to have a truly complete picture of the impact. For now, this is a necessary pain point, but it could also be sorted out with the adequate technology that allows the user to apply those frameworks automatically without special training or resources. That would let the organizations focus on the strategic improvements.
Build Gradually
An organization won’t be ready to start measuring impact until it has clearly defined a TOC and a sustainable business model. The next step would be to define core metrics, and the program/beneficiary data collection process. We strongly recommend migration away from MS-Excel to a new generation tool like the ones described in our previous article. MS-Excel has many issues, including lack of tracking, data inconsistency and inaccuracy, referential data and many others.
Building impact evidence for an individual organization or program requires significant effort. Building aggregated impact is even more difficult. At every layer of a system, a lot of context is lost before it gets finally reported. Most current frameworks and tools are designed to measure individual partner performance and are based on a quantitative approach.
However, anyone who has worked in this sector knows that reporting only quantitative data provides a limited vision of the results. Field leaders and program managers who are typically closer to the program evaluation process are more aware of the challenges in analyzing results. They understand the importance of aspects such as data quality, accuracy and the ability to audit it.
We need a system with a proper qualitative evaluation that can be flexible for each program and allows program leaders to assign a specific weight to the evaluation criteria depending on their relevance.
Conclusion
Building a strong impact evidence system should become a priority for every social sector organization, if they want to remain innovative and capable of fulfilling their stakeholders’ expectations.
In the past decade, we have witnessed a major shift in how funders view their role in philanthropy. Many early adopters are already adjusting to this new approach that requires a readiness from all social sector organizations.
Both funder and beneficiary organizations should understand this trend and get ready to be a vibrant player in the impact ecosystem. The alternative is obsolescence.
- Categories
- Impact Assessment, Technology
