1. Introduction

 

Use this guidance as a starting point for evaluating online works or hybrid projects. This guidance was developed during the Covid-19 pandemic in collaboration with National Portfolio Organisations (NPOs), Arts Council England (ACE) and expert peers.

 

2. Online works question bank

Background

In 2020 we developed an online works evaluation template to support Impact & Insight Toolkit (Toolkit) users evaluate their online works effectively. The template was available to Toolkit users until April 2023. Based on the online works evaluation template, we developed a concise set of online works question bank questions. The online work question bank provides the opportunity to receive feedback on the work being evaluated as well as indicating the respondent’s intention to experience similar works in the future.

While developing the online work evaluation template and, subsequently, the online works question bank, our desk research indicated there were multiple factors that are important across most online works:

• Maintaining engagement with pre-existing audiences
• Promoting wellbeing amongst pre-existing audiences (battling loneliness and isolation)
• Maintaining a high ‘quality’ experience
• Fundraising via requests for donations e.g., ‘pay what you can’ or lesser ticket prices
• Expanding reach to establish new audiences

Each question included within the question bank ‘speaks’ to at least one of these factors. The questions have been carefully designed to collect data that will be of value to both the individual organisation and the sector overall, whilst still allowing space for customisation.

 

The questions

Question subject	Question text	Instruction text	Time to answer (s)	Question Type	Answer Options
Connection with Others Online	I felt connected with others experiencing this work online	How much do you agree with this statement?	10	Dropdown Question	• Strongly agree • Agree • Neutral • Disagree • Strongly disagree
Device Used	On what sort of device did you experience this work?	Please choose an option from the dropdown menu	10	Dropdown Question	• Desktop computer • Laptop computer • Tablet • Mobile phone • Smart TV • Device connected to television
Finding the Work	Was it straightforward for you to find this online work?	Please choose from the options below	5	Yes/No Question	• Yes • No
Meeting Need	I felt that this digital experience met my need for cultural activity	How much do you agree with this statement?	10	Dropdown Question	• Strongly agree • Agree • Neutral • Disagree • Strongly disagree
Online Frequency	How frequently do you experience cultural works digitally?	Please choose an option from the dropdown menu	10	Dropdown Question	• Every day • Every week • Every 2 - 3 weeks • Every month • Every 2 - 3 months • Every 4 - 6 months • Once or twice a year • This was my first time

 

Accessing the question bank

In the Culture Counts platform, navigate to the ‘Design’ page of your survey, click on ‘Question Bank’, and then select ‘Online Works’. Scroll through the list of questions and select the ones you want to add to your survey.

 

Using the question bank

When selecting your questions from the question bank, ask yourself ‘What other quantifiable factors will help me understand the impact of this work?’

Each question you ask should help you to understand:

1. Whether your intentions for the work you’re evaluating were met
2. What impact your work has had on its participants

There is no ‘one size fits all’ approach and we would encourage you to consider and select the questions that you feel will genuinely support you in your evaluation. Don’t ask questions for the sake of it!

Consider including a combination of dimension questions and question bank questions. A well-considered combination will not only provide you with more context and understanding for interpreting your results, but also provide some variety for your respondents, avoiding survey fatigue.

 

3. Evaluation design

Survey length

When evaluating online works, it is important to keep your surveys concise.

Although there isn’t currently a body of research with well-defined answers for the effects of survey length for online works specifically, an attendee experiencing an online work may be less committed than when attending an in-person work – it’s much easier to open a web browser on a computer than to attend a venue in person – and the amount of effort they invest in answering a survey correlates strongly with their commitment.

When designing a survey for members of the public, whether for online or in-person work, we suggest aiming for a completion time of no more than 3 minutes. If the work being evaluated is much shorter, which is possible
for online works, we would recommend reducing the survey length accordingly.

For example, for a work of 10-15 minutes long, we recommend reducing the survey to 1 minute only.

We suggest aiming for a completion time of 6 minutes for peer reviews, with some flexibility. There is evidence to suggest that the drop-off rate increases after 8 minutes.

 

Further questions

Depending on the specifics of the work you are evaluating, there are other questions you may wish to consider.

If you are running work online via a means which allows for live commentary, (YouTube Premier, IGTV, Facebook Live etc) we suggest using a custom slider or dropdown to ask for their agreement level to the following
statement:

Question: The ‘live’ comments made me feel part of an audience community experiencing this work.

If you anticipate the reach of the work will be reasonably small, for example an online workshop with a maximum of 30 participants, we suggest asking more free text style questions, as they allow for richer insight. This is not
recommended for works with larger reach, as conducting analysis on qualitative feedback can be resource-heavy. For example, you could use a non-directive free text question such as:

Question: Is there anything else you’d like to share about your experience?

If your work is targeting people that regularly engage with arts and culture, you could ask a question about the respondent’s expectations. We suggest using a dropdown question, such as the one below, to learn how the public perceives the concept of online work and how that perception influences their responses to the quality-focused dimension questions.

Question: Did the work meet your expectations?

You may want to understand how much effort the attendee put into experiencing the work, and establish what, if any, effect this has. For instance, if you’re surveying a piece of theatre which was pre-recorded and now being shown via digital means, you might want to know the extent to which a similar atmosphere was recreated at home: did the attendee close the curtains, turn off their phone and turn up the volume? If so, what impact does this have on their experience? It might even be that they organised with friends or family to watch simultaneously, and then had a video-conversation afterwards to discuss it. A multiple-choice question, such as the one below could provide much insight.

Question: How did you recreate the atmosphere of the theatre at home?

The questions shown above are, of course, only a sample of further questions you could ask your respondents. There are many other topics you may wish to consider, for instance:

• Providing the opportunity to join a mailing list
• Gauging appetite to engage with similar works in the near future
• Asking a Net Promoter Score question, enabling you to gain some understanding of the further reach of your work

Through a combination of question bank questions and carefully selected additional questions, the insight you could gain from your survey could be invaluable as you develop your future programme.

 

4. Survey delivery

 

When setting up your survey, it’s important to consider how it will be delivered, to achieve as many responses as possible. Generally, with online work, you won’t be able to conduct face-to-face interviews, you also might not have access to the email addresses of attendees. However, depending on the nature of your online work, there may be opportunities to:

• Encourage those that are experiencing the work to complete the survey via live commentary (consider IGTV, YouTube live etc.)
• Use social media to publicise the work and the subsequent survey
• If it’s a live stream, encourage those that are delivering the work to publicise the survey
• If the work is pre-recorded, have a survey’s QR code and URL included at various points encouraging attendees to respond throughout the work; for example, if a theatre production is being streamed, a QR code and URL could be displayed at the interval point and/or before the final credits. The presentation of a QR code would make your survey more accessible to those that are experiencing the work on one device and able to scan the QR code on their smartphone.

It is important to acknowledge that there is likely to be a lower response rate to surveys for online works, in comparison to in-person. This is partly due to the lower level of commitment required to experience the work. However, you can still gain important insights. All responses to your survey will contribute to your understanding of how your online works are perceived by those that experience them, and the impact your organisation is having as a whole.

The team at Counting What Counts is ready to support you in your use of the question bank. We can advise on question choice, delivery methods and which respondent categories you may wish to use. Please do not hesitate to get in touch with any questions.