The State of Assessment: Moving the conversation forward to accelerate value creation (Part 2 of 2)
This is the second in a two-part series on the assessment debate. The first post focused on whether organizations should assess impact and explored the evolution of assessment to better understand why this debate continues. In today’s post, we discuss the current state of assessment and share our stance on the debate. We wrap up with our top five core activities to accelerate value creation through assessment.
The current state of assessment pushes us to extract more value from assessment, such as learning agendas for the broader development community. A framework called Metrics 3.0, released in 2014, calls for integrating impact metrics with operational metrics as well as carrying out evaluations that contribute to collective learning agendas for the broader “ecosystem” to create the most value from impact assessment.
Going back to the d.light example cited in yesterday’s article, it is clear that the question “Do d.light’s solar units improve the welfare of selected households?” fits in the collective learning agenda, as other organizations providing and investing in solar units would benefit from this knowledge. However, the answer to the question “Does d.light’s business model improve customers’ welfare?” provides specific information to help d.light’s operations and decision-making. This example also illustrates how the assessment’s goals help influence the indicators and methodology used. The Abdul Latif Jameel Poverty Action Lab and Innovations for Poverty Action value randomized control trials (RCTs) because they contribute to a broader learning agenda and influence policy. Proponents of RCTs have been quick to clarify that RCTs should only be used when a knowledge gap – which could occur either at the organizational or the ecosystem level – exists and when it is possible to do one well.
Organizations are now more educated and strategic in what they assess and measure. In the past, partly due to the confusion created by this debate, organizations collected too much or too little data, resulting in much unused data. Many organizations simply gathered data at donors’ request or feared that if they didn’t, investors would question their investment; but they struggled to make ordered sense of it. At the same time, organizations feared they’d overestimate their impact or base their estimates on false assumptions.
Today, many groups – including ours – develop thoughtful organization-wide data design processes with a clear rationale that minimizes wasted resources and maximizes data collection value. The increased focus on using impact data for learning and improvement reminds us that there are ways to create value through other types of assessments beyond impact evaluations. Although the term “impact” is largely used when discussing assessment – since it is the end goal of the work of social-mission driven organizations – many other indicators are also useful to help such organizations assess their performance and learn on their way to achieving that impact. The CART principles provide helpful guidance in deciding what to assess and how to create the right-sized data collection system. CART dictates indicators to be credible, actionable, responsible on resources and transportable to other contexts.
Many professionals now accept all assessment types (impact, process, performance) and methods (qualitative and quantitative) and use a range of them to incorporate multiple perspectives and types of indicators. At this point, the conversation largely revolves around effective and efficient assessment and integrating it into operations. There is also a movement toward tools that allow for quicker and less resource-intensive assessment such as rapid feedback loops, lean data, social performance and impact proxies.
WDI’s Performance Measurement Initiative is part of two consortiums that were recently awarded funding from USAID’s U.S. Global Development Lab to test and implement innovative approaches to measuring development impact. (Full disclosure: WDI is the parent organization of NextBillion). The consortiums seek to provide measurement solutions that depart from traditional approaches that can be overly rigid and often are slow to deliver results. The goal is to find new and practical ways to assess hard-to-measure results within complex systems in which development occurs. Through this work, the consortiums hope to enable quicker learning and adaptive management from data, as well as evidence-based decision-making and policy creation.
Before we end, we want to briefly explore ideas that leaders in the assessment field have shared to help different types of organizations make these data decisions. Unitus Seed Fund, a seed-stage venture firm operating in 10 Indian cities, suggests selecting indicators based on key questions associated with the organization’s stage of growth. Another idea presented by Alnoor Ebrahim and V. Kasturi Rangan, both faculty members at Harvard Business School, suggests organizations focus what they assess within a logframe on the complexity of their theory of change and operational strategy. For instance, if the goal of an ambulance service is to provide timely and quality transport for patients to a hospital, then the company should assess indicators related to its services rather than an outcome such as improved health years later.
The path forward: Assessment for value creation
Socially minded organizations must find transparent ways for assessment – especially to access investment and engage in nontraditional partnerships. Referring back to the distinction made in yesterday’s article, we believe assessment creates value for businesses and nonprofits alike for reasons such as learning at the organization and community level, benchmarking, identifying causality and enabling course correction. Deciding what (indicators) and how (methods and tools) to assess depends on the most important question(s) an organization wants to answer. The newly released Sustainable Development Goals (SDGs), with 17 goals and nearly 170 targets for these goals, places a focus on measuring results. The SDGs were developed in consensus with the broader development community and will shape development through the next 15 years. These goals also focus on whether projects target correct low-income populations.
So, let’s agree to end the assessment debate and advance to a conversation about how to extract the most value. As Jed Emerson states, “In the end, I hate the whole metrics debate. … It is repetitive, mind-numbing and distracting from the critical task of fighting the forces presently destroying our societies and planet.” Emerson instead pushes us to continue on the journey of documenting impact in order to learn – which aligns nicely with the Metrics 3.0 framing.
Five core activities to accelerate value creation
To accelerate this learning:
1. Businesses should contribute to ecosystem-level learning agendas and all organizations should strengthen their capabilities to develop strategic data collection systems.
2. Expand the use of technology. Advancement in technology has enhanced our ability to collect data, such as via mobile phones, as well as share and visualize data in new geospatial ways.
3. Share data across organizations. In our push to collect data especially from low-income areas, we should not inundate people with too many questions, resulting in survey fatigue. Many questions asked by different organizations are likely to overlap; thus, we call for data sharing.
4. Share results back with respondents as well as provide them with information in exchange for their survey responses. Mobile phones provide an opportunity for organizations to be less extractive in their data collection and to begin conversations with their local stakeholders.
5. Measure the long-term impacts, i.e. three to five years after a solution is implemented. Traditionally, many projects have focused on their short-term outcomes due to the prohibitive costs of longer-term data collection.
We look forward to continuing this conversation with you through the comments section below.
Acknowledgements: The authors would like to thank the following people for their time and helpful comments to strengthen this article, a version of which originally appeared in Thinkers Magazine: Chiropriya Dasgupta (director, Strategic Initiatives at Enterprise for a Sustainable World), CJ Fonzi (senior project manager, Dalberg Global Development Advisors), Daniele Giovannucci (president, Committee on Sustainability Assessment), and Thane Kreiner (executive director, Miller Center for Social Entrepreneurship, and the Howard and Alida Charney University Professor at Santa Clara University).
Heather Esper is the Program Manager of Impact Assessment at the William Davidson Institute at the University of Michigan (WDI), which is the parent organization of NextBillion.
Yaquta Kanchwala Fatehi is a Senior Research Associate for the Performance Measurement Initiative.