Evaluation toolbox

Hopefully by now you have an idea of what you want to measure or evaluate, and now you need to figure out how to get it done. In this section you’ll find an ample list of tools that you can apply to reach your chosen indicators, along with a deep dive into a time-honoured tool that remains ubiquitous yet often underestimated... the survey.

If you are tracking impact to meet the requirements of funders and partners, often a rich longitudinal narrative containing anecdotal evidence may be enough. This is great, but we’d also love to share some other techniques borrowed from many disciplines – including newsrooms, marketing, advertising, the arts and social science - which can give a much more profound insight into the power of your film, and provide evidence that you are reaching your impact goals.

Remember - too often, evaluation and impact assessments are driven by telling the funder what they want to know and what they want to hear. But ideally your interest in tracking impact is driven by curiosity and a desire to learn as you go.

To work out what might be right for your project, we’ve drafted a toolbox of techniques broken down by the four Impact Dynamics: Changing Minds, Changing Behaviours, Building Communities and Changing Structures.

Some of these tools and techniques are free and easy to set up. Some are expensive and require expert skills. But don’t be deterred. Even if you are cash strapped, you may still be able to secure pro bono support from a PR/marketing firm. You might be able to persuade a polling company to slip a couple of questions on for free at the end of another job. You may be able to collaborate with a PhD student or social sciences department at a university. You’d be surprised how many people might be willing to help.

This toolbox is a work in progress. Indicators and techniques can be applied to multiple dynamics but, as with all of this guide, we think this will give you a good start. If you have more ideas please email us so we can slot them in.

Evaluation of changing minds:

tracking a shift in public awareness or attitudes, at the individual level or amongst the general public, created by a films impact campaign.

Indicator Tool/Technique
Accumulate direct feedback post screenings of the film which illustrate the impact on the audience Through unsolicited letters, emails, phone calls or conversations with the filmmaker
Capture Audience understanding, reaction quotes, comment cards with simple entry/exit surveys at your film screenings.
Conduct entry/exit audience survey at a physical screening: assess levels of viewer knowledge, attitudes and behaviours on an issue before and after watching your film. Be sure to collect emails and survey them again in 12 months to learn what has stuck.
If you have your audiences email addresses, you can always conduct an online questionaire using a service such as Survey Monkey which is free: https://www.surveymonkey.com/. Qualtrics is an example of a pay for service: http://www.qualtrics.com/
You could capture audience feedback during screenings of the film which demonstrate impact of the project Check out Ovee which connects audiences anytime to watch together and interact with content which is used by ITVS https://ovee.itvs.org/pages/about
Try out Harvis http://www.afourthact.com/harvis/ a mobile web application that empowers individuals to share their perspectives by capturing their real-time emotional responses to recorded or live-media.
You could commission a Qualitative survey; providing deeper analysis of the impact of the film on the audience compared to those who have not seen it. Propensity Score Matching: Assess levels of viewer knowledge, attitudes and behaviours on an issue. Examine and compare people who did see your film with those who did not. Or with those who saw a different film on a similar topic." %}
You may be able to demonstrate that the film and film campaign reached a greater % of target audience reached compared to alternative strategies Partner orgs might be able to share target audiences reached by comparitive projects or comparitive investments - inlcuding publihing research or paying for advertisements.
{% trans "You could carry out Quantative surveying among the general population Commission a survey of public opinion via indepedendent agency such as Neilson or Gallop around the issue, before films release and repeat after a period for comparison.
Another more cost effective technique which can give you one data point is to commission a question through an Omnibus Poll. Some pollsters such as ComRes will offer a charity rate for appropriate projects.
If you can't afford an agency, try out Google Surveys https://www.google.com/insights/consumersurveys/home
Commission an SMS poll. http://www.smspoll.net/
Tracking the volume of press articles linking film to issues raised (not just film reviews but analysis and discussion) could provide evidence of a new trend Hire a media agency for press evaluation providing Advertising Value Equivalency (AVE) and PR value - a proxy to the number of people likely to have seen an advert or read an article on an issue. This is a standardised tool but not seen as very accurate.
{% trans "Run Google Alerts for press coverage or Google Trends to show how an issue trends over time. Correlating against release period of film and major campaign initiatives
Try PageOneX which tracks, code, and visualize major news stories based on the proportion of newspaper front pages that they take up. http://pageonex.com/
Online servies are available which offer global media monitoring and analytics such as Meltwater http://www.meltwater.com/products/meltwater-news/. Or Cision http://www.cision.com/uk/
Track the creation of new media content triggered by the film and impact campaign Commission of extended news reports, specially commissioned programming to sit around the film
Capture evidence of frame change within the mass media associated with the film and impact campaign Try Sentiment Analysis. Available tools include http://www.rankspeed.com/ a search tool that does a sentiment analysis on the blogosphere / twittersphere. A service also provided by http://www.trackur.com/
Consider tools such as Media Cloud: http://mediacloud.org/ examine how the network public sphere reflect public debate and discourse about key issues; examine a film in relation to the discourse around a key issue. Other examples are Story Pilot by Harmony Institute and ConText (please see further in the chapter) by University of Illinois
Monitor the update of issue, frames, language or characters by thought leaders, celebrities, government officials, academics etc. Research news archive for evidence
Review the evidence of change in standing of leading advocates in the media space or creation of new advocates (including subjects of the film or filmmakers themselves) Research news archive.
A rise in online traffic to your films website or campaign website may provide basic evidence of growing engagement Use Google Analytics for detailed stats about visitors to your campaign website.
Analysis of the the habits of those visitors to your films website or campaign website may show engaged users. For example the numbers of new visitors ( demonstrating a growing audience) / traffic sources or referals (which might indicate whether you are reaching reaching target audiences) / pageviews (which could be an indicator of levels of engagement) Have a look at 'action' based analytics like Mixpanel or KISSMetrics
The growth of your email list as well as the open rate of newsletters over the duration of the film impact campaign may provide evidence of growing engagement Track within your email programe such as MailChimp http://mailchimp.com/, or you could use Campaign Monitor https://www.campaignmonitor.com/ or Assemble http://assemble.me/
The activity over your films social media accounts may also show further evidence of engaged users. For example evidence of engaged posts ie: sharing / commenting, not just liking. Try out Facebook insights. Here is a great starter guide from Mashable http://mashable.com/2010/09/03/facebook-insights-guide/
For managing multiple social media accounts try Hootsuite https://hootsuite.com/ or Quantcast https://www.quantcast.com/ - providing worldwide audience and demograophic data for websites, videos, widgets, blogs etc
https://www.hashtracking.com/ can provdie real and historic hashtag intelligence
Try out free tool sharedcount.com to get a sample of the conversation
Content analysis of user-generated content on social media may demonstrate attitude shifts in the public associated with the release of the film and impact campaign. Try http://www.rankspeed.com/ a search tool that does a sentiment analysis on the blogosphere / twittersphere or http://www.trackur.com/
{% trans "Or a service like Crimson Hexagon which can track conversations and measure sentiment on a variety of social networks http://www.crimsonhexagon.com/.
You may be able to capture evidence of new terms entering the vernacular as a consequence of your film and impact campaign Try Sentiment Analysis of the blogosphere / twittersphere. Available tools include http://www.rankspeed.com/ and http://www.trackur.com/

Evaluation of changing behaviours:

Demonstrating how a film campaign has actively mobilised people to do different not just think different, whether that’s to buy or boycott, donate or volunteer.

Indicator Tool/Technique
A rise in the numbers of signatures on a pledge or petition associated with the film campaign Partners may be able to share data. Or else it might be captured on Avaaz.org or 360.org. Note that Google Analytics offers extensive documentation on how to track various metrics, with easy-to-implement page code and relatively easy event tracking, which can be especially useful for action KPIs such as form completions or donations.
A surge in letter writing associated with the messages of the film campaign Partners may be able to share data or may be reported in the press
An increase in fundraising or donations for a given partner organisation Partners may be able to share data. Also try Google analytics (see above)
An Increase in membership for a given partner organisation Partners may be able to share data.
An Increase in volunteering for a given partner organisation Partners may be able to share data.
Attendance at live event or gathering for protest associated with the film campaign Partners may be able to share data. Plus evidence from news orgs.
Increase in requests for information or for referrals to services associated with the film campaign Partners may be able to share data. Note: This can be an affordable way to gather compelling data, but it requires up-front preparation to identify and agree on the right metrics and to establish baseline performance, as well as commitment from partners to track and provide the information.
A lawsuit is filed which is associated with the films campaign News analysis. Court reports. Evidence from partners
Change in purchasing behaviour which is associated with the films campaign Sales analytics or Survey. This could be national or targeted ie: Street intercept survey of target community
Change in voting behaviour which is associated with the films campaign Review public records or conduct a survey. This could be national or targeted ie: Street intercept survey of target community
Change in social behaviour which is associated with the films campaign Review public records or Survey. This could be national or targeted ie: Street intercept survey of target community. Or field observation of target community over time
Target audiences reports change in behaviour Commission Qualitative surveying. Survey targeted focus groups and repeat with same sample after 12 months to track longitudinal behaviour change
General population reports change in behaviour Quantative surveying in general population. Commission a survey of pulic behaviour via indepedendent agency such as Neilson or Gallop around the issue, before films release and repeat after a period for comparison. Could also be verified in Public Records.
If you can't afford an agency, try out Google Surveys https://www.google.com/insights/consumersurveys/home
Individual narratives of transformation biographical / longitudinal study of how people's lives are transformed by participating in the film production / outreach process.

Evaluation of building communities:

Demonstrating how a film campaign has provided the focal point around which communities can organise.

Indicator Tool/Technique
The film is being used by grassroots organisations for community screenings or influencer screenings Record the views for clips, trailer and/or full film screened live or on websites and on social media.
Evidence of material from the film being integrated directly into campaigns and campaign materials
Evidence of a rise in membership of grassroots organisations and a strengthened base of support triggered by the film and impact campaign Partners may be able to share data. Measured through enrollment and membership rosters
Evidence of a rise in volunteering in the community triggered by the film and impact campaign Partners may be able to share data. Measured through enrollment and membership rosters
An increase in calls or referrals to community services triggered by the film and impact campaign Partners may be able to share data.
Evidence of rise in attendence at organised community meetings or protests due to the film or film campaign Partners may be able to share data. Record from news organisations
Legal actio filed by community groups due to film and impact campaign News analysis. Lawsuit analysis.
Surge in fundraising or donations to community organisations associated with the filmor the films issues Partners may be able to share data.
Evidence of leadership development in the community as triggered by the film: meaning those who can facilitate policy change are identified, engaged, and commit to change. Demonstrated through biographical evidence. Testimony from campaign partners. Interviews and press.
Evidence of new collaborations between community or grassroots organisations inspired by the film and film campaign, uniting around common policy goals and outcomes Testimony from campaign partners. Interviews, review of minutes. Press releases.
Formation of cross disciplinary / cross organisation committees. ie; strengthened coalitions within community organisers Number and frequency of meetings and information sharing opportunities. Interviews, progress reports, meeting minutes.
Formal agreements for sharing information and responsibilities. Partner organizations collaborate/advocate on common policy goals and outcomes. Evidenced through MOU / contract. Meeting minutes. Press releases.
Try Issue Crawler, the network visualisation tool. http://issuecrawler.net, is used by NGOs and other researchers to answer questions about specific networks and effective networking more generally
Evidence of a shift in organisation and strengthened organisational capacity triggered by the film and impact campaign Advocacy Capacity Assesment Tool / Self Assesment Spider Diagram
Awards granted to orgs/ individuals featured prominently in the film and campaign Partners to share details. Press releases or news reporting.

Evaluation of changing structures:

Evidence that legal & political, corporations and major institutions are affected by the film impact campaign.

Indicator Tool/Technique
Screenings in local government / national legislatures / transnational instituions such as UN or EU Public record and press releases will confirm screening events and where possible also capture list of attendees
Use of the film by campaigners or advocates to lobby elected officials Make a forensic record of all infleuncer screenings from one on one meetings to government committees and party political conferences. Note who attended, who spoke out and track where this leads.
Elected official refers to the film or film campaign or quotes from the film on the record Review Public Record and press to capture quotes by politicans / lawmakers
Evidence of the creation of new political advocates due to the film and impact campaign Review Public Record and press to capture quotes by politicans / lawmakers
Evidence that the film has influenced Policy development / adoption / amendment / implementation / enforcement Review public record to show when film or film campaign sited in cited in the amendment or formation of new legislation. Check record of legistative sessions and oversight committees. Tools such are available for tracking political process such as http://www.opencongress.org/
Film / campaign mentioned during court proceedings Press or Public Record
Reversal of legal action associated with the film and impact campaign Press or Public Record
Call for hearing or an investigation is triggered due to the film and impact campaign Press or Public Record
Fine or penalty is levied Press or Public Record
Firing/Resignation triggered Press or Public Record
Criminal charges are brought Press or Public Record
Screenings at major national or international conferences Press or Public Record and where possible note key audience members
Use of the film by campaigners to lobby chief executive or board members of corporations or major institutions Make a forensic record of all your infleuncer screenings. Note who attended, who spoke out and track where this leads.
Corporate or institutional leader refers to the film and campaign Review Public Record and press to capture quotes
Press announcements. Quotes in speeches and correspondence
Changes in internal/external marketing campaigns, CSR priorities, new purchasing or sales strategy Press announcements. Annual Review. Budget monitoring. Institution stops or starts releasing data.
Copycat behaviour on the High Street or amongst competitive instiutions due to the film and impact campaign Press announcements.
Changes to corporate bottom line due to the film and campaign Financial news analysis
Corporate partnerships terminated due to the film and impact campaign Press announcements
Adoption of film and associated impact campaign tools into major institutions Reporting from film team / outreach partners - number of programs adopting tools, reach of those programs

How to create a good survey:

It may be that you want to understand how audiences are relating to your film: what they learned, how it shifted their perceptions or attitudes, what it inspired. If that is the case, a survey or poll will be a powerful tool for you. Surveys make it easy to track shifts in attitudes, awareness or behaviours across a range of audiences and over time.

The primary question for your team is: what is the information you are looking for and want? Are you curious about what lessons varied audiences take away from a film? And if so, why do you want to know this? Do you plan to edit the film further? Or are you using the feedback to refine your impact strategy? Do you want to know if the film has the capacity to move people towards specific actions? If so, great. Be sure to account for different event designs or facilitators, since you are not just measuring the impact of the film but also the effectiveness of the event itself.

There are other watch-outs too. A long, badly designed survey can confuse more than enlighten, and even irritate your audience. They are also time consuming for the film team, so make sure you need one. If all you want to know is that your film is landing in the way you expected, you may simply need to survey a pilot group. After that, you may not need one. Better to focus your data collection on more important things like action, intent to get involved, etc. Or it may mean you should refine your data collection plan. For example, perhaps you can shorten the survey so that audiences can focus on the questions you are still curious about. Just remember to note the change in your reporting. 

There are also more subtle challenges, which are connected with good practice. Here’s our 10 step guide to developing a great survey.

1. Define the audience available for survey

To state the obvious, screening The Invisible War on Capitol Hill is very different to screening Weapon of War in a remote community in the Congo, and different again to the audience who sees your film in theatres or downloads it from Netflix. But too often, survey writers of all shades forget to think about who it is they’re asking questions of. Write down each audience – then follow the rest of the process for each audience separately. Don’t be tempted to try to combine them right until the end, because you’ll most likely miss something.

2. Review your impact goals and understand which are relevant to that audience

This is where you need your Impact Evaluation Plan again. Go back to it, and filter it for goals that are relevant to the audience.

3. Brainstorm a list of questions

Starting from each of your impact goals, generate a first draft list of questions for that audience. What would you love to know? Don’t try to make these questions perfect at this point – we’re coming to that...

4. Do the ‘So What?’ test

Now you’ve got a first list of questions, it’s time to do the ‘So What?’ test. You’ve got a list of what you’d like to know - now what do you need to know? How will the answers help you? Try to keep it short. Ideally, your survey wont take anyone longer than 10 minutes to complete.

Remember, every additional question you ask beyond the first few will get less and less thought and attention in response; and it creates more work for you to do in analysing the data. Quantity and quality are in direct competition when it comes to surveys, so whilst you shouldn’t give up on the information you genuinely do need and could respond to, if you can get it down to one question, do.

5. What’s the baseline?

It’s all very well saying that people are likely to do, say, or feel something after watching your film. But unless you know whether they were already likely to do it before watching, you can’t claim any impact on that. That’s why baselines matter. There are a few ways to achieve this:

  • You could try contacting researchers working in the specialist area of your film to see what data they have already collected before the release of the movie. And see if there is an opportunity to repeat the original study post impact campaign.
  • You can try to get some idea of a comparison with a broader group of which your audience is a subset. So-called “omnibus” (“for everyone”) studies enable this by getting questions out to a group (usually 1000 people) that is representative of the nation as a whole, usually for under $1000 per question. If the audience at your film say something different to your baseline group, you can often use that to point to a difference you’ve made.

Remember that unless you’ve enlisted professional evaluators to help you, this methodology can be very tricky because there are so many variables to account for in your comparisons.

  • More powerfully, you can do pre and post-screening questions – asking the same question before and after, so the only variable in between is the experience of your film. Of course, even that’s not perfect, because they may have read about the film in order to have decided to see it, so your impact may actually have begun before the screening. But it’s a good one to consider.
  • Even better, if you can get contact details, you can go back to people after a significant period, and find out if they actually acted on intentions formed at the screening and if they are still doing so. This takes out the ‘heat of the moment’ factor, and is a real test of impact at this level. The problem, of course, is that asking people for their details has its own problems – you have to be very sure of where you stand on data protection rules, and asking for data is another question that will take up their attention... so this is a good thing to do, but is best done sparingly.

If you can’t capture a baseline, another tactic is to ask audiences to self-assess. In other words, you can ask them how the film shifted their perceptions, feelings, or understanding. It too is imperfect, but perfectly reasonable if you acknowledge your methodology in your reporting.

6. Check for two-in-one

What often happens when you refine a survey down like this is that you become tempted to bring two questions into one. The alarm words to look out for are ‘and’ and ‘or’.

If The End of the Line survey (see case study above) had asked “How likely are you to support political action on Bluefin tuna fishing and seek to buy sustainable fish yourself?”, they might have blurred their information and never have recognised the opportunity they had. These are two questions, not one.

7. Don’t lead the witness

You need to be very careful to check the framing of your questions. In the first draft, you will almost definitely be leading your respondents to the answer you want – it’s human nature. 

Back to The End of the Line - if they’d asked “Would you be more likely to buy sustainable fish as a result of watching the film?”, the right answer is clearly “yes” – and people like giving the right answer. Challenge yourself to make it possible to give the answer you don’t want.

8. Close the questions

Now you need to make sure that your questions can be answered simply, ideally by checking a box. The bad version of The End of the Line question this time would be “What would you be most likely to do differently?”, with an open space for responses to be filled in. This is all very well in some ways, but unless you’re asking a lot of people you may not get what you’re looking for (the vast majority of open survey questions are left blank). And, even if you get a response, analysing that data is going to be a real headache for you. 

The more you can provide options, yes/no questions, and checkboxes (e.g. select one, select your top 3, select all that apply) that include “other (please describe)” and “don’t know/not applicable” kinds of options, the more likely you are to get answers, and answers you can use.

All this said, while carefully crafted closed-ended should be the majority of the survey items, it’s also ideal to include a maximum of one or two open-ended questions to understand the audience members' experience qualitatively as well.

9. Think about the best medium

At this point, the actual questions should be pretty much written. Now you need to think about how to administer the survey – should it be paper-based? SMS? Online?

Some evaluators recommend electronic data collection whenever possible to help with streamlining the flow of data, since paper surveys can be easily lost or misplaced. But it all depends on your audience and digital access. Informing change has just put together a full inventory of tools/platforms for online data collection, available here: www.informingchange.com/cat-resources/survey-platforms

If you can get it down to a single question, or three at most, SMS might well be a good option. Data gathered this way is easily analysed, and people are often very willing to text an answer quickly - you may even be able to follow up and gain permission to use their contact details on an ongoing basis. There are a wide variety of platforms that offer this functionality for free, with the most popular in the United States being Poll Everywhere. Even the popular Mailchimp platform offers a free “text to join email list” function. And with mobile phones now widely used across the world, there are ways of doing this almost everywhere if you do your research. Again, just be conscious of your audience and both cost and access to mobile and other technologies.

Online survey tools like Survey Monkey and Survey Gizmo are also widely available (and can be mobile-optimised for access by smartphone). These platforms also often have the benefit of making data collection, analysis, data visualisation and even reporting relatively painless. The challenge, of course, is that only people sufficiently motivated to go to the site to register their views will fill in your survey, which can create a major bias in your data towards people who care. If you can afford it, offering some sort of incentive (a downloadable voucher or similar) can help overcome this. 

Sometimes though, you can’t actually beat an old-fashioned paper-based survey. The difficulty is analysis, since you don’t want to spend hours going through responses, but there are relatively low-cost hard and software solutions (for example, autodata.com) that can take the pain out of this.

10. Pilot test it – check the language

Whatever you do, always, always, always test it before you go live. You will have done something that makes it not quite work. Whatever questions and whatever medium you decide upon, get 10 friends to try it out first, and ideally try it out at a rough cut screening too before you commit to it fully. The pain and cost of knowing the data that’s coming in won’t be usable, or of trying to change something when it’s up and running, is seriously worth avoiding.

Need more help?

The nature of our work is that there are always people willing to help, and there are plenty of people who design this kind of thing for a living – whether in marketing or research agencies, or in the social sciences departments of universities, including the Media Impact Project at the University of Southern California Annenberg Norman Lear Center. Evaluation firms include Harmony Labs, Impact Architects, Informing Change, and Learning for Action. Often, getting their help (particularly the latter) can add credibility to what you do as well – so ask around your friends, partners, funders and supporters, and get help.

Geek Out: Ideas for further reading

Made byDoc Society Made possible by: Ford Foundation - Just Film Bertha Foundation Sundance Institute Knight Foundation Dox Box DOCSP Ambulante MX