Take an Evaluative Approach

Film teams that take an evaluative approach to their projects gather information and feedback as they go so they can learn and improve their approach. So, the film or components of the project, or objectives, or design will necessarily shift as they proceed. In this way, evaluation is not necessarily simply about tracking outcomes - it’s about learning. 

The Water Warriors team took an evaluative approach to their campaign. To track impact, they systematically collected responses to two key questions from each of their screening partners: 

  1. Why did you show this film? [tracking intent/objectives]
  2. What about it was useful? [film strengths and outcomes]

They used the responses to track the intent (objectives) of screening hosts and understand what strengths they saw in the film. They also used the exhibit to test their assumptions and gather feedback on their messaging with various communities. 

This iterative process allowed them to learn and make sure they were on the right track. They could ensure communities felt they were being represented the way they wanted to be - a participatory approach that was important to the team. It was through this process, for example, that they learned that the communities they captured on film wanted the intensity of being in a direct action blockade to come through. 

Through the process, the project evolved, eventually becoming an effective way to provide a platform for local groups and allowing them to broaden the reach of their work to new residents with whom the organisers had not yet connected.

We’ll discuss evaluative approaches more in Chapter 6. But for now we leave you with this: it may be important to distinguish the differential impacts of various parts of a campaign so you can appropriately make changes as you proceed to maximise your project’s impact potential. 

Tracking Confusion

Because some shorts’ impact campaigns tend to be transmedia, the question of how to account for a short’s unique contribution to change can also become more challenging. In other words, when a short is connected to a larger campaign with other media and story-based components, what are you measuring the impact of: the short, or the campaign?

Take Chasing Coral as an example. A shorter, 4-minute version of the feature length film that was hosted on Upworthy was viewed nearly 900k times. Released in the days leading up to COP23, it offered a burst of energy around climate negotiations taking place and an urgent reminder of what's at stake. In addition, it featured the full, time-lapse footage revealed toward the end of the feature- length film, making this powerful visual evidence of climate change available for organisations and everyday citizens to use as an awareness-raising tool. These were distinct contributions from the role that the feature length played in other moments of the campaign and film’s life. Screening hosts also shared short films before the film as a way to connect the film's global story to local issues – advocacy organisations shared locally produced shorts touching on their campaign priorities and businesses shared short films showcasing their commitment to sustainability. In settings like these, the shorter films played a role in helping audiences make a connection. They worked in concert with Chasing Coral. But their contributions were distinct.

Being attentive to the different ways that different project components contribute to change matters less if you don’t plan to make changes and/or you simply want to understand the overarching response to your project. So, be clear what it is you’re evaluating and why, and build your evaluation plan around your needs.

Measurement Mismatch

So often, because a short is so easily distributed online, the data for online/digital reach becomes easy to collect. But this data - while important - is sometimes a mismatch for the primary impact dynamic of a project. In other words, in some projects for which the primary impact dynamic is to change minds and raise awareness, measuring reach, digital engagement and interaction can be valuable indicators of impact. But if the primary impact dynamic is to build communities and foster new relationships, then the quality and sustainability of interaction and engagement will become more important. If the primary impact dynamic is policy change, then reach only matters in so far as you can show the connection between your audience and the pressure they placed on policymakers by way of the short. 

The film Virunga was an exposé that shed new light on the activities of Soco International, a British oil company that was putting the Virunga National Park and local communities at risk through its oil exploration. Because the team aimed to use the story to build public understanding about the importance of the park as a source of peace and prosperity for the region and the existential threats it faced, wide viewership was key. Therefore, those numbers mattered a great deal in that context. But the team also produced a series of shorts, referred to as Virunga Yetu. This series had a different focus: to engage local villagers and increase their desire to protect or support the park, and to shift their value for sustainable development over extraction. As such, online views matter little in this context where the target audience does not have access - and even if it did, what mattered most was the quality of the reaction.

Humbling Impact

It’s also just as valuable to know the limitations of what you can say about impact. Take as another example the films that IFProductions created on sexual violence in the Congo. 

Despite the fact that the team was able to reach 2 million people in the Congo (or more, they stopped tracking the numbers) with their film Fighting the Silence through their traveling Mobile Cinema project, the filmmakers were reluctant to say too much about the impact of their work on the issue. That’s because the local organisations that use their films (and that received special training on how to moderate conversations, using guides developed to help them) intentionally engaged audiences multiple times with various pieces. 

“Besides our films and workshops, there has been so much activity in the Congo around this specific issue from different organisations that it is impossible to trace which activities did what,” says Ilse van Velzen. “What we did see however is the ‘power of film’ and what impact it leaves. It starts dialogues and breaks through taboos. People who would normally not talk about their own experience were able to do so by referring to the people they saw in the film, rather than themselves.”

Impact, she stresses, is also about ensuring they do no harm and that the process from beginning to end is done with accountability and safety in mind. So, that meant engaging with local communities in the filmmaking process (as described earlier in this chapter in 5.3). And it meant making sure to follow up with organisations, social workers and aid workers linked to the screenings, to have safety nets in place for audiences after screenings to provide trauma and other support as needed.

Having said that, Ilse stresses the vital importance of conducting a proper evaluation. “Sometimes it’s hard to get evaluation into budgets, but you have to fight for it. Because that’s where you can truly understand what happened, and in so doing, your work and the field at large can learn and grow.”

Made byDoc Society Made possible by: Ford Foundation - Just Film Bertha Foundation Sundance Institute Knight Foundation Dox Box DOCSP Ambulante MX