Download this article as pdf.

Operating in four countries in eastern Africa, Farm Africa is a charity working to reduce poverty permanently by unleashing African farmers’ abilities to grow their incomes and manage their natural resources sustainably. We develop innovative models that help farmers to not only boost yields, but also gain access to markets and add value to their produce. Our aim is to develop successful models that can be replicated at scale to deliver sustainable change for whole regions.

Farm Africa works in complex contexts and understanding impact is critical to our success. If we don’t know what change we have brought about and why, we cannot learn what works best in which situations or persuade others to adopt our approaches to benefit more people than we can reach alone. For us, impact is essentially about understanding what difference we make – what return do our investments generate?

Challenges in measuring ROI

The first challenge of measuring return on investment (ROI) in social investment is of course defining it. For most of our work, the main ‘investment’ comes from project donors. However, the communities we work with also invest their time and resources, and most projects are also partly subsidised by general funding. It is important to understand the extent of these additional investments to ensure we know the true cost of achieving our results.

Defining the ‘return’ side of the equation is more complex. Farm Africa’s primary interest is the benefits of our work for rural communities. If people do not benefit enough from the activities we introduce, they will not maintain them and we simply cannot expect to deliver lasting change. Yet putting a value on these benefits and understanding what they mean for small-scale farmers is challenging for a variety of reasons.

Firstly, economic return (cash income) for smallholders will always be an important aspect of our work and is usually necessary to ensure sustainability. But it is not sufficient. To really know what difference we are making, we also need to understand the social and environmental impacts of our work. Farm Africa is committed to improving the livelihoods of both current and future generations. This means integrating economic and environmental sustainability in all our work. It is easy to commit to this but valuing and comparing different dimensions of impact in practice is not straightforward.

For example, if a project was found to boost household incomes of smallholder farmers but did so in a way that degraded soils, damaging the prospects of future generations, we would not consider this successful and would take immediate action to stop the negative effects. Although true impact cannot be known until after a project has closed, it is important that we are mindful of potential impacts, both positive and negative, during project lifetimes. Responsive, and responsible, management requires regular collection and application of feedback and evidence from the communities we work in.

Then there is the question of the appropriate period over which to assess returns. The lifetime of our projects is usually insufficient to see full impact. Indeed we design projects specifically to bring about lasting change. But what is the right period? By the end of the project we can assess intermediate changes and estimate the likely longer-term impacts, but these cannot truly be known until sometime after completion.

Of course, the longer we wait to measure impacts, the more external factors will have influenced people’s outcomes and the more difficult it is to define the contribution of our work. Added to this, when a project closes, so does its funding stream, making it difficult to find funders willing to support measurement after the project is complete. Yet, failing to measure longer-term impacts can lead to short-term ‘sticking plaster’ approaches being prioritised over sustainable change, as they tend to cost less per head at the outset.

Our approach

So, should charities try to continually measure all short-, medium- and long-term economic, social and environmental impacts in all their work? Farm Africa believes not. We face trade-offs when deciding how much of our limited resources to allocate to measurement. On the one hand, we want to dedicate enough that we are confident we will know what is working and where change is needed to deliver the best results. On the other hand, we must be careful not to divert resources that would be better used to deliver more or better services to our target communities.

Spending on measurement has an opportunity cost and we aim to spend where the ‘marginal benefit’ is highest. Rather than stifling innovation, we believe that appropriate measurement should be a key feature of innovation – after all, there is little point being creative if you are unable to demonstrate to yourself or others that your innovation works. We encourage our project teams to develop measurement systems that are consistent with our organisational approach but commensurate with their scale and needs. This ‘fit-for-purpose’ approach has led us to combine universal minimum standards that ensure we can track success across the board, with targeted use of general funds to go deeper to address priority knowledge gaps.

For example, last year we tested the use of mobile technology as an alternative to face-to-face demonstrations to train farmers in optimal sesame cultivation. As we had been working in the area for some time we already had good evidence that, if farmers’ knowledge about appropriate agronomic techniques is improved, then their yields and subsequently income also tends to improve. To evaluate the technology, we focused on the unproven link in the chain – could the use of training modules on tablet computers increase knowledge as effectively as field-based training does? We found that the mobile training course did in fact increase knowledge to a similar level and did so at around one third of the cost. As a result we are now further exploring the use of mobile technology on a larger scale.

Similarly, we identified a gap in our knowledge of the wider, less tangible benefits of our natural resource management work. For example, in Chilimo in Ethiopia, where we launched our first participatory forest management project in the 1990s, satellite imagery clearly demonstrates that deforestation was not only halted but forest conditions actually improved throughout the project. Similar results have been found across our forestry work, yet the socio-economic impacts of shifting forest management from state-control to joint management with communities have been more challenging to value. Rather than attempt to measure these less tangible outcomes in every project, we decided to conduct a ‘social return on investment’ study focusing on one or two longer running forestry projects. This will ensure sufficient depth of evidence to generate more credible conclusions that can inform how we measure, and ultimately deliver, similar work across the region in the future.

These two examples of sesame cultivation in Tanzania and forestry in Ethiopia show how different Farm Africa’s projects can be, and illustrate the necessity of tackling impact measurement on a case-by- case basis. While there are similarities in some of the impacts of these two projects in terms of increased yields and household incomes, other impacts, such as reduced carbon emissions associated with reforestation in Ethiopia are not found across all our projects and therefore require a bespoke approach to measurement.

This article first appeared in Philanthropy Impact Magazine Issue 11 Download this article as pdf.