NextBillion - An Initiative of the William Davidson Institute at the University of Michigan

As the initial lockdowns of the COVID-19 crisis begin to loosen up, it is clear that we are facing a much different, longer-term reality that includes some level of social distancing and a protracted economic crisis. Within this context, social impact organizations are scrambling to adjust their internal operations, continue fundraising and adapt their programming. The resulting changes will affect these organizations on practically every level – and impact measurement and management (IMM) is no exception.

IMM is sometimes seen as a “nice to have” in the social impact sector, and therefore may be seen as something that can be reduced or minimized during the crisis. But in fact, based on our experience at the Aspen Network of Development Entrepreneurs (ANDE), this would be a mistake. In working with nearly 300 organizations to support small and growing businesses in emerging markets, we’ve seen that IMM is as important as ever in the time of COVID-19 – but it will require some changes to be done well in the “new abnormal.”

The importance of IMM during the pandemic

If you work in a social impact-focused business or organization, you are undoubtedly grappling with a huge amount of uncertainty, and your needs – and those of the people you serve – have likely changed significantly over the past few months. A well-designed IMM system that focuses on fast feedback loops and decision-oriented data is critical for providing insights on how your programs and services might need to be adapted to address these changing needs.

At ANDE, for example, we leveraged the systems we have in place for our standard collection of impact data for our members to do a rapid assessment of their initial actions and projected challenges in light of the crisis. This gave us critical data that allowed us to develop a suite of activities to respond to these challenges. This response has included a COVID-19 resource page, a set of webinars and virtual events for members to share their strategies in adapting to the crisis, and an advocacy and coordination effort among funders to highlight the dire financial needs of entrepreneur support organizations in emerging markets, and funding strategies to address these needs. Because we were able to quickly use existing systems to gather data, we are moving forward with relevant and needed programming for our members, while exercising ANDE’s leadership in the small and growing business space.

Adapting IMM to the COVID-19 crisis

Even organizations that are continuing their investment in robust IMM systems during the crisis will likely need to change their practices. The reality of social distancing, the presence of a new set of needs on the ground, and a significantly increased level of uncertainty across all sectors means that the IMM systems that were so well-designed for 2019 may need significant overhauls to be relevant now. Here are a few changes organizations can consider, and resources to help them make these adjustments:

Converting to remote data collection

Even as lockdowns begin to ease, it is likely that continued social distancing precautions will mean that face-to-face data collection is impossible for some time. This means that data collection will need to be done remotely, via either online or mobile phone-based tools. Luckily, there is a wide range of tools and guidance available to help ensure that both quantitative and qualitative data can still be collected at a safe distance:

• J-PAL recently published a guide on best practices for conducting phone surveys – which they are continuing to update as new tips and tools become available.
• GeoPoll is one of the most well-established platforms for conducting mobile surveys, and they offer a free guide to conducting research in emerging markets.
• 60 Decibels, which spun out of Acumen’s Lean Data Initiative and pioneered a number of practices in rapid mobile assessments, has a Remote Survey Toolkit that helps organizations develop high-quality mobile data collection systems.
• CartONG is a French NGO specializing in humanitarian data analysis. They have generated a guide to remote assessment that includes links to further technical guides, and suggested vendors or technology platforms.

Adjusting metrics and data points for what matters now

The impact of COVID-19 is already being felt across all segments and aspects of society, and the ripple effects will continue to spread in unpredictable ways for years to come. While many of your existing metrics may still be applicable, many organizations may need to consider substituting or adding new metrics related specifically to the effects of the COVID-19 pandemic into their IMM systems. There are a number of emerging research efforts and toolkits which provide a strong starting point for new metrics and data points that will help you continually monitor pandemic-related impacts, and improve your programming accordingly:

• Innovation for Poverty Action (IPA) has launched their RECOVR effort to undertake and coordinate research on the wide range of effects of COVID-19 in the developing world. As part of this, IPA developed the RECOVR survey instrument to “facilitate comparisons, document real-time trends of policy concern, and inform decision-makers about the communities that are hardest-hit by the economic toll of the pandemic.” They have made this survey instrument publicly available so other organizations can collect similar data in their own contexts – and have also created a hub of other survey instruments that capture different aspects of the impact of the crisis.
• The Center for Global Development has published a survey in English, Chinese and Spanish to assess the pandemic’s impacts on small and growing businesses, capturing a range of responses including business projections and financial health. While originally used to evaluate Chinese businesses, this instrument can be adapted to other contexts.
• SocialSuite, a technology platform designed to help organizations conduct effective IMM, has developed a free COVID-19 Social Impact Assessment to help monitor how the crisis is affecting an organization’s people and initiatives. This underscores the importance of not only considering the impact on those you are helping, but also on your own employees, whose well-being is critical to carrying out effective programs.
• Mobenzi, a company based in South Africa offering a suite of data collection platforms, has developed a set of tools to assess the socio-economic impact of COVID-19 through a variety of metrics.

Maintaining flexibility and adaptability

The COVID-19 pandemic has created a highly uncertain world—the only certainty is that things will remain extremely unpredictable for the foreseeable future. Now more than ever, the principles of adaptive management are important to consider when developing or adjusting IMM systems. Traditional logframe approaches that assume a predictable pattern of cause and effect are likely to be inadequate, and may miss important unintended consequences and causal drivers during such a complex time. Luckily, IMM practitioners have been developing ways to build complexity and adaptability into measurement systems for years, so it is a good time to acquaint yourself with how these approaches can be incorporated into your own IMM practice:

• The Impact Management Project’s Five Dimensions of Impact framework is a helpful starting point in ensuring that your IMM is set up to generate a robust understanding of all of the different ways your program may be creating change during the crisis.
• FSG’s report on Evaluating Complexity provides guidance on how to take a more iterative and adaptive approach to measuring change, to avoid becoming locked into a set of metrics and assumptions that may become obsolete as conditions change.
• USAID’s Collaborating, Learning and Adapting toolkit provides a broad set of resources to tackle all of the challenges of designing and implementing an adaptive IMM system.

The prospect of redesigning IMM approaches and methods in the context of a global pandemic may seem daunting. But there are practical steps organizations can take, and significant resources they can leverage, to ensure they are able to make data-driven decisions and maintain their impact during the crisis. The pandemic’s unpredictability only reinforces how necessary quality information and insights are in responding to the wide variety of challenges it is raising. We hope that this guidance can provide a useful starting point for organizations moving forward.

ANDE is collating IMM-related resources that can help organizations respond to the COVID-19 pandemic. If you have a helpful resource to contribute, please visit our Impact Measurement and Management in the Context of Coronavirus page. 

Matthew Guttentag is the Research and Impact Director and Mallory St. Claire is the Impact Analyst at the Aspen Network of Development Entrepreneurs (ANDE).

Photo courtesy of Myriams-Fotos.