NextBillion - An Initiative of the William Davidson Institute at the University of Michigan

Monitoring and Evaluation (M&E) has moved into the mainstream of the development industry, as seen by foundations like the Inter-American Foundation and Conservation, Food & Health requiring the incorporation of rigorous M&E structures in their grants. While the jargon can get messy – from “results-driven framework” and “impact assessments” to “evidence-based approaches” – the basic idea remains: getting NGOs to start measuring their programs in deeper ways.

While I welcome the spike of interest in evaluation, it has largely focused on wide-eyed Westerners writing up some logical framework from their desk in Washington, D.C., impressing donors with said logical frameworks, and flying to the project site once a year to “evaluate and assess.” As if you can put agricultural transformation, participatory research and leadership development into a brief report by someone who landed in the community two days ago. I’ve always been intrigued by organizations changing the game and investigating creative ways to get M&E into the hands of beneficiaries. To achieve truly sustainable, locally-led development, we must include program participants not only in project design, but also in project analysis.

Despite pesky administrative necessities that rob the time and resources of small startup nonprofits, Semilla Nueva (where I serve as director of Monitoring, Evaluations and Grants) has done our best to invest heavily in M&E, particularly in participatory evaluation methods. A few noteworthy tools are our Malnutrition Study, which trains local mothers to measure kids’ height and weight changes, and our Experimental Parcel Analysis worksheet, which uses images to represent investments and earnings so that even illiterate farmers can make a basic return on investment calculation of the new technologies they are testing. Our tools, such as the Experimental Parcel Analysis worksheet, are being picked up by other regional NGOs as well as the Guatemalan national government, promoting more sound M&E throughout the country. I am proud to say we have become one of those innovative organizations I have always admired.

In 2013 I co-designed the first SN Community Survey along with a few other gringos on the team and a University of Florida graduate student originally from Guatemala. This team of external evaluators spent the summer coasting between communities on chicken buses, cruising among rural households on bicycle, surveying more than 150 farmers in the region, and when they had the chance, catching a nap in hammocks with fresh mangos in hand. The survey gathered data on everything from demographics and economic realities of the communities, to perceptions of our farmer-to-farmer model and agricultural development in general. To analyze our impact, we surveyed farmers in SN communities and farmers in “control” communities outside SN influence.

We learned a lot that year. We learned of failing government fertilizer handout programs, of coyote middlemen keeping corn prices low and farmers vulnerable, of internal community conflict and why cooperative models fail. We also learned a lot about evaluation strategy – farmer confusion over questions designed by gringos, the realistic time requirements of open-ended questions leading to rants from frustrated farmers without other outlets to voice their opinions, and the inefficiencies of outsiders implementing surveys with locals.

In 2014 we repeated the Community Survey. I tweaked it a bit, trying to focus on the most important data for internal evaluation. Last year’s survey gathered a lot of similar information on demographics, technology use and knowledge transfer. We learned about perceptions of community cooperation, of sustainability and climate change and, most importantly, about what they’ve learned so far through SN training sessions. However, we really wanted to amplify the voice of the farmer.

And, as we looked to expand to 15 new communities starting this year, we could take it a step further. With more staff, more resources and more time to dedicate to M&E, we wanted to do what we never got a chance to do in our current communities: A baseline study that provides for better impact analysis down the road. And then, a lightbulb went on; here was our opportunity to really engage the bottom-up M&E innovation I’ve been waiting for. We hired new field technicians who will oversee agricultural programs in the 15 new communities directly from existing communities. All local farmers, these leaders have participated in Community Surveys for the past two years and now they need an activity to introduce themselves to the community. Boom. The surveyed become the surveyors, and M&E becomes localized.

Above: SN’s food security coordinator, Jennifer Brito, conducting part of our 2014 Malnutrition Study.

During their week-long orientation to the job, I spent a day training these farmers on evaluation, impact assessment and survey execution. Here were three men, lacking high school degrees but armed with a combined 60 years of farming and leadership, eager to impress their boss. When I asked their impression of a “survey,” they recognized it as that time of year when university students come in and ask them a bunch of questions. When I asked about why measuring impact matters, they linked it to donors’ needs. Everyone in the room identified these surveys as a common occurrence in their communities, something carried out by outsiders and the results ultimately affecting outsiders as well.

But when told that they would be the ones asking the questions this time, eyebrows raised and some rustled in their chairs. So we talked about the objectives of a survey, about the importance of establishing a baseline by which to measure impact. “But no one knows what no-till is in these communities yet, why would we ask them?” farmer Jose posited. I explained how doing the same survey every year can demonstrate progress. Perhaps in year one we learn they have experimented with it, and by year two maybe we will find out they have implemented no-till on all of their land. We talked about why impact matters to the community and not just to donors. “We are testing to see if the knowledge is transferring, if farmers are learning from the training,” farmer Cirilo said. “It could be that we give trainings, but the farmers don’t learn, and we need to know why. Wow, that sounds like a really good idea.” Boom. The evaluated become the evaluators, and M&E becomes localized.

I spent the day role playing with our new technicians, myself acting out the local campesino (smallholder farmer) and giving complex, nuanced responses while the new field technicians learned the role of surveyor, translating those multifaceted answers into multiple-choice answers. By the end of the day, they were no longer playing a role. I saw their confidence develop throughout the day in their words and gestures.

With the development of these tools and training tactics, we want to provide practices that can replace traditional models of aid focused solely on outputs – number of schools built, medicine provided, trainings given – with a more measured approach that focuses on outcomes. How has community behavior changed? How is knowledge being transferred? What are we learning about what’s working?

I left that day glowing with excitement and pride. This is development working, putting the tools for sustainable development – from project design to evaluation – into the hands of the beneficiaries themselves. The extranjera (outsider) shares her expertise and the campesino gains empowerment. And slowly but surely the project is becoming truly locally-led. While we struggle to define and measure qualitative impacts like empowerment and leadership, I can offer a humble eyewitness testimony to successful development that day.

Kristin Lacy is director of Monitoring, Evaluations and Grants at Semilla Nueva.