Impact measurement should be designed at a sector level and funders should be prepared to pay for it. These are two of the conclusions drawn by New Philanthropy Capital (NPC) in its latest report: Measuring Together: Impact measurement in the Youth Justice Sector.

Impact measurement is exciting interest among funders and charities in the wake of government cuts and the rise in a culture of payment by results – but issues around metrics, definitions, attribution, benchmarks and the best ways to measure and capture the value of interventions confound the development of impact measurement and charities’ engagement with it.

In the first in a series of reports exploring impact measurement at sector level—in this case looking at youth justice – NPC offers ways for charities to overcome these difficulties by creating a common framework that could be used to help them and funders within the space understand their impact and that might also allow for meaningful comparisons between services.

NPC’s Tris Lumley says: “The report is not a magic bullet for measuring impact, but it offers a process to help charities agree what are the most appropriate outcomes for the sector, and how to measure them.”

The main message behind the report is that impact measurement—designing a framework of what to measure and how—is better done across a sector.

NPC believes that by looking at measurement in a sector as a whole, drawing on existing research and knowledge, the main challenges can be identified and solutions proposed. It would also encourage cooperation between charities and funders working within the same field, and avoid charities working in isolation to develop their own frameworks, says NPC.

“Expecting every charity to create its own measurement tools and frameworks is ridiculous. It would be like asking every GP to measure their own impact. As the report shows a co-ordinated approach could overcome the barriers that exist including a feeling of isolation among many charities and the lack of a structure around what and how to measure,” says Lumley.

The framework suggests the outcomes that should be agreed and how they could be defined, captured and attributed, while highlighting some of the complexities.

NPC’s recommendation to funders is that they should invest in charities that measure their results, pay for evaluation, and fund projects that are designed to improve measurement, both for individual charities and for the sector as a whole.

It also suggests the government should make it easier for charities to access re-offending data, publish performance data on individual prisons (as announced recently by the Ministry of Justice), and publish quality data on the costs of different youth justice services. Charities getting started with measuring their imapct should begin by collecting basic data from the young people they work with, and analysing what they find says the report. They should also work closely with other charities to coordinate their measurement efforts, and work with statutory agencies to maximise their chances of accessing the best data.

Lumley says the creation of cost effective, robust and meaningful outcome measurement tools that work across a sector may still be years away, even if charities work together and share their findings. “However, if charities are expected to measure on an individual basis it could take decades,” he says.

The barriers that exist are financial, cultural and systemic explains Lumley.

“The real key is to get buy in from leaders. We need to get the chief executives of charities and funding leaders to demand outcome measurement and to make sure it is funded, as well as to share what they develop.  There are systemic issues that also still need to be addressed such as where to find analysts – the people with the right skills are in short supply - and how to access data, which is difficult in the youth justice sector.

“However, some organisations are making great strides and I am optimistic that outcome measurement will eventually be done consistently well and by the majority of charities,” says Lumley, who points to the successful cultural change in financial reporting that was kick started by the Charities Act 1993. “Now nearly every charity in England and Wales that is required to produces a set of audited accounts for the Charity Commission as a matter of course. So cultures can be changed.

“Some things are harder to measure than others. But that is not a reason not to do it. I have yet to come across a field where outcomes really cannot be measured. Measuring re-offending  might be easier than trying to understand how the trauma of asylum seekers fleeing conflict and persecution has been addressed. But if you don’t measure impact you just can’t manage your work properly—in the worst case you may not realise your intervention could actually making beneficiaries worse off,” says Lumley.  

Impact reports however can be misleading and Lumley says they should be read with care: “Charities whose results look more impressive compared to another may have ‘cherry-picked’ the groups they work with. Though it is not a legal requirement for charities to share information on the methodology used to measure their impact, methodology and assumptions needs to be considered in understanding the real impact of a project. The best impact reports don’t just make claims about results—they show their working too.”

Lumley rejects the idea that ‘cherry-picking’ could in fact be a sensible approach for charities—ensuring that they succeed as easily as possible.

“For charities, often the greatest impact will come from making sure you find those who need your help most, not those that who are easiest to help. A charity helping people back into employment may seem to get the best results if it’s getting lots of people into jobs, but if those people are completely work-ready, and it’s avoiding working with those who have been unemployed for years, it’s probably not creating the greatest possible impact. To improve people’s lives, and society, as much as possible, it might have to deliberately pick the high-hanging fruit.

“That’s why impact reports need to be transparent about methodology.” he explains. “They tell us what the problem is you’re trying to tackle, how badly it affects people, and how much they benefit through your work.”

Measuring Together: Impact measurement in the Youth Justice Sector can be downloaded from the NPC website.