We love evaluation

'It takes 75 editorials to pass a law'

Robert L. Bartley, late Editor of the Wall Street Journal opinion page

'It’s becoming clear that a rapidly changing technology landscape requires human rights organizations to adapt and become more nimble. We would argue that funders also need to adapt their approaches. Many are still asking for outdated ways of measuring impact — with an overemphasis on numbers, for instance, or in holding grantees accountable to rigid and predictive multi-year outcomes (a potential pitfall of Logic Frameworks)'

'Who are we, if not measured by our impact on others?'

Carl Sagan, Astronomer & Author

'Evaluation' is one of those buzz kill words. It sounds pedantic, reductive, time consuming, difficult, dangerous. As filmmakers and storytellers, many of us have a deep-rooted suspicion that if applied to our creative work, it just might kill it - or at least miss the point.

When you consider the sheer amount of energy, passion and time that has to go into making the film, devising and executing a brilliant distribution strategy, let alone delivering the campaign… It’s easy to see why some of us might have a mental barrier with the concept of evaluating something as complex and multifaceted as impact.

But whether we like it or not, the business of evaluation, of metrics and measurement has arrived in documentary film. This is provoking anxiety not only about our individual practice but raising implications for the whole field.

Do we have the resources or the competency to evaluate? What if funders start to demand the wrong kind of evaluation? Will evaluation in the end stifle creativity, ultimately determining and limiting which kind of films get made?

We need to keep our nerve. Because it’s only bad evaluation that analyses creativity to death. It’s only bad evaluation that tries to break everything down into numbers. And it’s only bad evaluation that sucks up time and money that could be better spent elsewhere.

Illustration by Mike McQuade

Illustration by Mike McQuade, from Stanford Social Innovation Review

'To solve today’s complex social problems, foundations need to shift from the prevailing model of strategic philanthropy that attempts to predict outcomes to an emergent model that better fits the realities of creating social change in a complex world'

'The two questions we should be asking are ‘how big is the boulder, and how far did it move?'

Ricken Patel, Avaaz

'Not everything that can be counted counts, and not everything that counts can be counted'

Albert Einstein, Theoretical Physicist, Architect of the Theory of Relativity

This is about learning to love great evaluation: deploying techniques that can capture the properties and power of your documentary film campaign, and allowing you to communicate the extraordinary impact your film is having in the word. Because who will tell your story if you don’t?

Great evaluation will help you to secure new funders and to maintain existing funders. In a changing media landscape this is critical to the sustainability of our field.

Great evaluation will also impress and comfort all the other kinds of partners we want to work with, from grass-root organisers to leading campaign organisations. It can help you find collaborators and mobilize your audiences. You can show what a film project is going to deliver that their army of expert campaigners, lobbyists, and researchers cannot.

Great evaluation can be learnt from – both within a project, to improve it as we go; and across impact filmmaking as a whole field, so we can learn from each other and all do it better.

Fundamentally, great evaluation is great because it makes more change happen faster.

In this chapter, we’ll dig into great evaluation in more detail exploring what makes evaluation great, what makes it complex, mapping out a plan with the help of a worksheet, and reviewing a multitude of tools to help you with execution. We’ll draw on a few case studies to illustrate what we mean. You should already have read most of these, but if you haven’t, head to the Librarynow.

Then we’ll get into helping you draw up your own Impact Evaluation Plan, with the help of a worksheet, basically a simple list of the essentials you want to keep track of to help you maximise your impact. We’ll start by helping you refine the goals from the Strategic Plan you made in chapter two, then identify some Indicators, or data points, that you’ll be looking to gather, using our Indicator Matrix to find what will work for you. There’s also some practical help on designing surveys and an overview of tools available to film teams to help you make the most of your evaluation work.

And once you’re done, we reckon you might love evaluation just as much as we do.

The solution is to be proactive. Make a plan based on the goals of your campaign, and share it with funders and partners - create it with them. Ask what matters to them, why they funded the project, and consider their goals as well as your own - co-create a model that helps the film team gain insight on the work while it delivers key outcomes to the stakeholders. Done right, evaluation can be revealing whilst delivering strategy, powerful in demonstrating effect.

Made byDoc Society Made possible by: Ford Foundation - Just Film Bertha Foundation Sundance Institute Knight Foundation