Putting It All Together

So here we are!

 

By now you’re hopefully seeing the value and possibilities that open up when you track the impact of your efforts. We’ve talked about how best to do that and using the learnings to strengthen your efforts. We’ve addressed the importance of being honest about what you want to know and your capacity as a team, and designing your evaluation plan from that place. We’ve also looked at each impact dynamic and the various tools and indicators to help you track what’s happening for each. In this section we offer a couple of examples of what it can look like when you pull it all together.

Take Your Place: An in-house evaluative approach

A Place at the Table, by Lori Silverbush and Kristi Jacobson, is a film about the epidemic of concurrent hunger and obesity in communities across the U.S. To track the impact of their efforts around the film, Active Voice Lab received a grant to explore various assessment methods, co-designed by an in-house evaluator and their How Do We Know/Learn? advisors. Importantly, they were able to collect baseline information - essential for determining what, precisely, changed as a result of using a documentary. Additionally, having an evaluator in-house allowed the impact strategy and assessment to inform one another from the start (remember the feedback loop above?) 

The in-house evaluator was able to be present for important strategy and implementation discussions, and as a result was able to identify in real-time how new understandings about the context in which community partners were working was influencing shifts in program strategy - and, in turn, what implications these shifts had for the evaluation. In other words, Active Voice’s program and evaluation teams, working closely together, were able to course-correct along the way. 

For example, the initial design included engaging organic food advocates, but early inquiries revealed a significant gap between their priorities and those of the hunger/obesity advocates.

In conversations with partners, the impact team came to understand that while anti-hunger and obesity advocates faced similar institutional obstacles, they rarely worked together. 

So, they decided to use the film to pull stakeholders together for community-based discussion and planning that could be tailored to the different needs that each organisation and community faced. The resulting strategic plan included a series of braintrusts (small cross-sector planning meetings) and screenings with local organisations in 24 cities across the country. The aim was to use these activities to ignite a common, system-based analysis and spur action around solutions.

The evaluation plan called for data collection and three distinct points in the campaign over the course of a year: 1-2 weeks before a screening to gather baseline data, 2-3 weeks after the screening, and 6-7 months after the screening. The data included surveys from audience members and community hosts at each point in time (as well as application data from host organisations), surveys of community partners at two points in time (pre and post-screening), as well as interviews with community hosts. In addition, data was collected from individuals who received information about the film but did not actually see it; these evaluation participants served as a comparison group for the audience level outcomes analysis. Doing so allowed them to track progress and point to a few distinct outcomes. The data showed that their efforts led to:

  • Increased awareness of the direct relationship between hunger and obesity among 4 out of 5 audience members.
  • Attitudinal shifts among 40% of audiences who said they were more supportive of federal government responsibility to address hunger and obesity
  • Increased discussion about the issues continued among 91% of audience members in the months after seeing the film
  • Increased coordination and collaboration among local organisations that continued in 62% of communities up to 7 months after the screenings took place
  • New activities in 77% of communities after the screening, including new programs.

Check out this infographic, which lays it all out. As you can see, the team had a clear vision of impact from the start, and an evaluation plan that informed their strategy and allowed them to track exactly what they needed to know and only what they needed to know, allowing them to make statements about the impact of the campaign that was backed by data.

Hunting Ground Australia: Targeted impact tracking

From the start, the impact team for Kirby Dick and Amy Ziering’s The Hunting Ground knew they had limited capacity for evaluating their efforts. So, they had to be smart about how they used their resources. 

When the film arrived in Australia, after a successful run in the U.S., most university vice-chancellors contended that sexual violence was not a problem on their campus in the way that it was in the U.S., even though the impact team’s early research and discussion with experts in gendered violence indicated that many of the issues raised in the film were also relevant in the Australian context. So, t​he Australian impact team set out to use the film as a catalyst for the sector, to build pressure for a baseline survey on the prevalence of campus sexual violence,​ and to learn more about the Australian context​. 

The team produced screening support materials and how-to host guides; developed specific “working with media” fact sheets for student representatives, survivor-victims and journalists writing about sexual assault; and adapted the US education materials and Action Toolkit for the Australian context. Their strategy also included the development of the Australian Human Rights Centre’s good practice for universities, On Safe Ground (2017), the commissioning of a sex and ethics education module for use in university contexts, running a campus screening program, and forming the relationships, coordination, and leadership needed to address campus sexual violence. These activities were in and of themselves impressive and promising. 

The team also worked with the Australian Human Rights Commission, National Union of Students, and Universities Australia to set up a first-ever baseline survey of campus sexual assault and sexual harassment, undertaken by students nationally across all campuses, resulting in the Change The Course (2017) report.

To measure the impact of their efforts, they knew they needed to track the following indicators:

  • Student reactions to the screenings and resulting activities (from a desire to take action to increased understanding​ of the issue​) by way of monitoring questions and exit surveys
  • University requests for screenings for staff and students (indicating a new willingness to have the conversation and listen to student responses about their experiences on campus)
  • University and college uptake of Sex, Safety and Respect training programs
  • Implementation of new policies as recommended by the findings of the nationwide survey (as outlined in the Australian Human Rights Commission’s Change the Course report and the Australian Human Rights Centre’s On Safe Ground report).

Altogether, t​he team hosted 70 campus screenings in 33 of Australia’s 39 universities where they tracked these indicators.​​ They also worked with student leaders in conferences and national meetings where the film was used to influence the behaviour of their peers and campus culture. Through straight-forward but consistent event observation and communications with university partners, they were able to show that these events​​ led to:

  • Increased student and university leader interest in issues raised in the film, including:
  • University policies and procedures
  • Available data on incidents of assault
  • Available support for survivors and preventative measures in place
  • Understanding consent
  • Independent reviews of student experiences of sexual assault in some university colleges
  • The introduction of new sexual assault policies on campuses
  • A massive increase in the number of university residences implementing or improving sexual violence prevention training, and utilising a sexual assault service in delivery of that training

Their theory of change suggested that mobilising student interest and engaging institutional leadership would lead to new initiatives, practices, and campus cultures. Their progress on campus policy changes to date suggests they may be right!

It’s worth remembering that the campaign included a national broadcast on ABC2, community screenings, conference screenings and more. So, the impact of the team’s efforts went much further than the above.

Read the yearly progress reports and university case studies from The Hunting Ground Australia Project here:

Geek Out: Ideas for further reading

And look out for the evaluation report that will wrap up the three years of activities undertaken by the impact team, and show where the baton has been taken up by two community-based organisations who have partnered with the film over the campaign.

There are many more impact reports in the Library to peruse. But don’t take our word for it… why not take a look for yourself?

{% trans "Geek Out: Ideas for further reading


Made byDoc Society Made possible by: Ford Foundation - Just Film Bertha Foundation Sundance Institute Knight Foundation