Embrace complexity
Context. Context. Context.
Great assessment and evaluation recognises that impact is complex and accounts for context. Different issues, different kinds of impact, and different kinds of films all intersect in different ways and this complexity has three big consequences.
First, we need to be proportionate about the role of film, and of the team, in the story of change. You shouldn’t expect to - or indeed try to - prove direct causation of the big, end-of-chain change in the world. Remember that your film is likely a part of a movement, but unlikely to be the cause of it. The most you will ever do, indeed the most anyone will ever do, is contribute. And this is important to untangle.
The Invisible War, as we saw in Chapter 2, helped contribute to culture change around sexual abuse in the military. Looking back, we can see it was one piece in a constellation of media, activities, leadership and intersecting efforts that led to the #MeToo movement. Similarly, Bully contributed to a tremendous amount of traction on the issue of bullying in schools. CNN even did a special, “The Bully Effect” to explore the impact the film had had. And yet even so, how can we isolate the impact of Bully from the constellation of other actors, such as the It Gets Better Project, that focused on elevating popular attention to the issue at about the same time.
The Invisible War team were able to make clear claims about the policy momentum the film built because there are statements from policymakers who attribute their positions and actions to the film. Similarly, The Bully Project signed up over 140 mayors across the United States to learn about and take local action to improve school climates and reduce bullying, and years later is now working with the American Academy of Pediatrics to train pediatricians to screen for and advocate for patients impacted by bullying. It too will be able to make the claim that the film has had a direct impact on the will and capacity of the pediatricians they’ve trained to support bullied youth. So, be proportionate. Be accurate and give credit where it is due.
Second, part of complexity is unintended consequences. However good and noble your intentions, there is a chance that what you do might cause harm. What you can and should do with evaluation is be alert to this, devote time to identifying possible risk areas, and gather data on those as well. For example, you might regularly survey the response of the subject community to the film as well as the responses of those whom you’re trying to affect. Keep asking the question: is this impact campaign advancing the goals of the movement, and are people on the ground getting what they need? In this way, impact and strategy go hand-in-hand.
This brings us to the final point. The complexity of impact documentary means you will never be able to design for everything. What you can do, though, is understand that impact is inherently unpredictable, so be attentive to your campaign and the work of the movement you’re contributing to, and... brace yourself.
Pro tips
Do the ‘So What?’ test
Once you’ve decided what data you want to gather, and indeed throughout the process, keep asking yourself the ‘So What?’ question. Say you’re doing a regular survey, and the number of people who agree with a certain statement has changed. Do you know what you could or would do differently as a result? Is there anything? If not, should you really be asking the question at all, or is it just a waste of time and money?
Avoid the ‘good news’ trap
Once the work is over, you’ll naturally want to make the best of it, so in assessing responses, you’ll only look for good news and perhaps skirt over the unexpected and unwelcome turn of events. That’s human nature. But it’s not great a way to measure impact.
Be patient – but ready to move!
The nature of complexity is that two scenarios are possible. The first, and more likely, is that it may well take years for the full impact of your project to be ready for a final evaluation report. It’s more than likely that you won’t be able to report anything truly meaningful for two to three years or more. The second, though, is that it all kicks off with virtually no notice. If the stars align, you could find yourself very busy very quickly. If you’re gathering data, you’ll stand a chance of spotting the opportunity early.
Give credit where it is due
Thank your funders and partners. They’ve likely been in the trenches for years. You did this together. So pay your respects. It will mean they are more likely to share information and data which could be essential to tracking the impact of your project. They are also more likely to value the evaluation and share it widely if they are properly credited.
Be proud of joining the choir
Impact often continues after you’ve finished reporting on the impact of your work. The life cycle of a film and impact campaign can be open and continuous - long after the case studies have been published. Telling the stories of change that come to light when a film embeds in the culture has its own impact in the larger effort to place artists in the web of social change. Your film has tremendous value, but it’s not likely to be the solo voice or the final word.
Case study: Granito vs A Small Act
In many ways, these two projects could not be more similar – even in name, they bear a strong resemblance. 'Granito' translates as ‘tiny grain of sand’, relating to the idea that every individual has their little bit to add to the achievement of social justice. But the two films progressed at dramatically different paces.
The Granito project began back in 1982 with the making of When The Mountains Tremble, exposing the fact that the Guatemalan government was responsible for the murder of Maya civilians. But only in 2013, after 30 years, could the team really point to concrete impact – with the guilty verdict against General Rios Montt on 10 May 2013.
By contrast, the A Small Act team had at that point barely considered the idea of an impact campaign, struggling even to get the film finished, by the time of its first screening at Sundance in 2011. And yet, halfway through the screening, a member of the audience stood up and pledged $5,000 to the Hilde Back Education Fund, the scholarship fund set up by Chris Mburu who is the main protagonist in the film to give Kenyan children the opportunity to get an education just as he had received from Hilde Back. Another audience member followed suit, and by the end of Sundance, $90,000 had been pledged. By the end of 2013, the team estimate that $2.4m dollars had been pledged to the fund, an uplift of well over 5000%.
One project found things moving very fast despite little planning, and had to run to keep up – while another had to stick with it for 30 years. As a reminder, that’s because different issues, different kinds of impact, and different kinds of films all intersect in different ways. In the case of these two films, this also resulted in a very different role for evaluation on the two projects. One demanded a more dynamic and adaptive evaluation approach/plan - one that could identify, track and learn from unintended outcomes - while the other did not.
For the Granito team, evaluation has been primarily retrospective and about finding more and more ways to build the story, and with it the Maya community’s belief in their ability to achieve justice. This evaluation, in the form of a constant and committed presence in Guatemala, and continuous dialogue with the community, has been the inspiration behind the creation of their second and third films, as well as the companion radio, web and participatory documentary projects. It has been a slow but continual process.
The team behind A Small Act were forced to be responsive, with opportunities coming at them from all sides, but they also had to make sure they stayed true to their own ambitions for the film. After all, the team’s key ambition wasn’t necessarily to generate funds for the Hilde Back Education Fund, but to reframe development narratives to illustrate that developing countries often help themselves.
