Embracing complexity

'It is generally more appropriate to explore how media have contributed to impact, rather than to make causal claims about impact… Generally speaking, causation cannot be established between content and impact, particularly with larger scale goals such as changed policies and conditions. Rather, look for a correlation among the reach of your media, degree of engagement, and evidence of impact to infer that a contribution has been made to the change'

Steven LaFrance, CEO, Learning for Action

'It is important that any evaluation methodology distinguish between ’attribution’ and ‘contribution’. The vast majority of social issue documentaries and their engagement campaigns are entering into a community (however small or large) of activists, leaders, organizations and coalitions that have laid groundwork long before the films and campaigns were conceived and they will be there for many years continuing to build the movement. This critical work must be acknowledged - just another reason an element of ‘story’ is so important in this evaluation work - so that this relationship can be explained'

The Fledgling Fund

'Finding target audiences is absolutely crucial, but one key aspect to any campaign that uses global ‘broadcast’ media platforms (including Twitter and YouTube, for instance) is that you must brace yourself for exposure to the broadest possible audience. You have to imagine how our media content may be consumed and used by people who fundamentally disagree with the presumptions behind your work and the implicit value system in your messaging'

Johanna Blakley, USC Annenberg Norman Lear Center

Great evaluation recognises impact is complex and this complexity has three big consequences. First, we need to be proportionate about the role of film and of the team in the story of change. You shouldn’t expect to or indeed try to prove direct causation of the big, end-of-chain change in the world. The most you will ever do, indeed the most anyone will ever do, is contribute.

Second, that part of complexity is unintended consequences. However good and noble your intentions, there is a chance that what you do might cause harm. What you can and should do with evaluation is be alert to this, devote time to identifying possible risk areas, and gather data on those as well – for example by regularly surveying the attitudes of the subject community to the film as well as those you’re trying to affect. Keep asking the question; is this impact campaign advancing the goals of the movement, and are people on the ground are getting what they need?

Finally, the complexity of impact documentary means you will never be able to design for everything. What you can do, though, is understand that impact is inherently unpredictable, so be attentive to your campaign and the work of the movement you’re contributing to, and... brace yourself. If you gather the right data, that’s exactly what evaluation will help you do.

Pro tips

Do the ‘So What?’ test.

Once you’ve decided what data you want to gather, and indeed throughout the process, keep asking yourself the ‘So What?’ question. Say you’re doing a regular survey, and the number of people who agree with a certain statement has changed. Do you know what you could or would do differently as a result? Is there anything? If not, should you really be asking the question at all, or is it just a waste of time and money?

Avoid the ‘good news’ trap.

Once the work is over, you’ll naturally want to make the best of it, so in assessing responses, you’ll only look for good news and perhaps skirt over the unexpected and unwelcome turn of events. That’s human nature. But it’s not great evaluation.

Be patient – but ready to move!

The nature of complexity is that two scenarios are possible. The first, and more likely, is that it may well take years for the full impact of your project to be ready for a final evaluation report. It’s more than likely that you won’t be able to report anything truly meaningful for two to three years or more. The second, though, is that it all kicks off with virtually no notice. If the stars align, you could find yourself very busy very quickly. If you’re gathering data, you’ll stand a chance of spotting the opportunity early.

Give credit where it is due.

Thank your funders and partners. They’ve likely been in the trenches for years. You did this together. So pay your respects. It will mean they are more likely to share information and data which could be essential to tracking the impact of your project. They are also more likely to value the evaluation and share it widely if they are properly credited.

Be proud of joining the choir.

Impact often continues after you’ve gone through evaluation. The life cycle of a film and impact campaign can be open and continuous -- long after the case studies are published. Telling the stories of change that come to light when a film embeds in the culture has its own impact in the larger effort to place artists in the web of social change. Your film has tremendous value, but it’s not likely to be the solo voice or the final word.

Case study: Granito vs A Small Act

In many ways, these two projects could not be more similar – even in name they bear a strong resemblance. 'Granito' translates as ‘tiny grain of sand’, relating to the idea that every individual has their little bit to add to the achievement of social justice.

In others, they are starkly different. The Granito project began back in 1982 with the making of When The Mountains Tremble, exposing the fact that the Guatemalan government was responsible for the murder of Maya civilians. But only in 2013, after 30 years, could the team really point to concrete impact – with the guilty verdict against General Rios Montt on 10 May 2013.

By contrast, the concrete impact of A Small Act started at its first screening at Sundance in 2011. The team had at that point barely considered the idea of an impact campaign, struggling even to get the film finished. Halfway through the screening, a member of the audience stood up and pledged $5,000 to the Hilde Back Education Fund, the scholarship fund set up by Chris Mburu to give Kenyan children the opportunity to get an education just as he had received from Hilde Back. Another audience member followed suit, and by the end of Sundance, $90,000 had been pledged. By the end of 2013, the team estimate that $2.4m dollars will have been pledged to the fund, an uplift of well over 5000%.

So because of the complexity of driving change with film, one project found things moving very fast despite little planning, and had to run to keep up – while another had to stick with it for 30 years. For our current purposes, though, the key point is that this resulted in a very different role for evaluation in the two projects.

For the Granito team, evaluation has been about finding more and more ways to build the story, and with it the Maya community’s belief in their ability to achieve justice. This evaluation, in the form of constant and committed presence in Guatemala, and continuous dialogue with the community, has been the inspiration behind the creation of the second film, and the companion radio, web and participatory documentary projects. It has been a slow but continual process.

For the team behind A Small Act, evaluation has been a rather more intense experience. From the moment that audience member stood up at Sundance, they were forced to respond fast, with opportunities coming at them from all sides and daring to be wasted – and they also had to make sure they stayed true to their own ambitions for the film. After all the team’s key ambition wasn’t necessarily to generate funds for the Hilde Back Education Fund, but to reframe development narratives, highlighting through the role played by Chris Mburu in the untold story that developing countries often help themselves.

In order to stay true to this, the team had to shape the opportunities that were coming to them, carefully framing their ask to inspire people to do a small act of their own relevant to them, whatever that might be. Donating to the fund remained a default option, and one many took, but this approach inspired countless other acts. All the team could do was keep their ears open, and try to keep inspiring broad action – small acts everywhere rather than donations to a single fund.

The stories they collected and told were a continual support to them in doing this, and in fact, as the money flows start to calm, they would argue that a major part of the impact is only just beginning – in the last two years, the film has started to screen in communities in East Africa, and inspire small acts there.

With that in mind, it’s possible that in 30 years’ time, we might look back and think that the stories of A Small Act and Granito were pretty similar after all. Either way, both are examples of great evaluation here and now.

Made byDoc Society Made possible by: Ford Foundation - Just Film Bertha Foundation Sundance Institute Knight Foundation