As a user of the Impact & Insight Toolkit, you will have access to an evaluation template entitled, ‘Evaluation for Online Works Template’. This template has been designed in collaboration with arts and cultural organisations as well as funding bodies in order to establish a survey to capture insight into the ever-increasing use of digital media to experience arts and cultural works. More information on its development can be found here. This document explains how a Toolkit user might choose to develop and alter the existing template to suit their individual needs.
This evaluation template has been designed to support Toolkit users evaluate their online works effectively. It provides the opportunity to receive feedback on the work being evaluated as well as indicating the respondent’s intention to experience similar works in the future.
Desk research indicated that there were multiple factors that are important across most online works:
- Maintaining engagement with pre-existing audiences
- Promoting wellbeing amongst pre-existing audiences (battling loneliness and isolation)
- Maintaining a high ‘quality’ experience
- Fundraising via requests for donations e.g. ‘pay what you can’ or lesser ticket prices
- Expanding reach to establish new audiences
Each question included within the template ‘speaks’ to at least one of these factors stated above. The question schedule has been carefully designed to collect data that will be of value to both the individual organisation and the sector overall, whilst still allowing space for customisation.
3. Dimension usage
If you are a National Portfolio Organisation you can use the ‘Evaluation for Online Works Template’ to meet your funding requirements. When you select the template you will then be able to choose between the Core or Flexible option.
A Core evaluation for NPOs must include all ACE Core dimensions as listed below.
When you tick the Core option the system will automatically populate your surveys with these dimensions.
Two of the selected dimensions, Rigour and Relevance, are also part of the core set for CPPs.
These core dimensions have been selected to provide continuity across both a NPOs and CPPs evaluations, regardless of the financial year and funding requirements. This will provide you with comparable data points which will enable you to understand the difference between work that is experienced online vs in-person.
A Flexible evaluation, on the other hand, allows you to choose any 4 dimensions from the Core Cultural dimensions or Participatory dimensions categories. Please see here for a list of the dimensions available in each of these categories.
We suggest that whichever dimensions you choose, they speak to the aims of the work and your organisation’s mission.
4. Survey length
As always, we want you to obtain maximum insight without overwhelming respondents with too many questions and placing too high a demand on their time.
The importance of keeping a survey concise increases when surveying work experienced online. The reason for this is that the attendee’s commitment to experiencing the work is likely to be less than when attending an in-person work – it’s much easier to open a web browser on a computer and go to a website than to attend a venue in person – and the amount of effort they might invest in answering a survey correlates strongly with their commitment. However, there is not yet a body of research with well-defined answers for the effects of survey length for online works specifically. Therefore, we are primarily recommending similar lengths for live works as outlined below.
When designing a survey for members of the public, we usually suggest aiming for a completion time of 3 minutes. If the work being evaluated is much shorter, which is possible for online works, we would recommend reducing the survey length accordingly. For example, for a work of 10-15 minutes long, we recommend reducing the survey to 1 minute only.
We would suggest aiming for a completion time of 6 minutes for peer reviews, with some flexibility. There is evidence to suggest that the drop-off rate increases after 8 minutes.
The Evaluation for Online Works Template is designed to be all-encompassing – something that can be used if you have no other data or if you’re uncertain how to gather further data. It is therefore on the longer side of what we would suggest but it provides a solid grounding, enabling you to capture key data as you look to learn more about the impact your online work is having.
There are two options to reduce survey length:
- Remove some questions which are less relevant for your evaluation activity
- For options on making the survey shorter please see the ‘Suggested questions alterations’ section below.
- Split the questions across two different surveys and distribute both
- Please get in contact for assistance in doing this.
5. Suggested question alterations
At 3 minutes long, you might feel that the public survey within the template is too lengthy, especially if you are confident in gathering additional data to supplement your findings via the Toolkit. Therefore, we have recommendations for tailoring your survey to make it suitable for your organisation and work.
If you are confident in using the analytics generated by the platform on which you’re presenting your work (e.g. YouTube analytics), we recommend removing these questions as this information can typically be gained from the platform:
- Multiple choice: On what sort of device did you experience this work?
- Dropdown: How long did you engage with this work for?
- Short-text: Why did you not stay for the full duration of the work?
- Yes/no: Do you live in the UK?
- Short-text: Which country do you live in?
- Demographic: What is your postcode?
- Demographic: What is your age?
The table below shows all the questions included in the survey template, along with the factor the question relates to and an approximate time to answer – please note that the example below is for a Flexible evaluation i.e., 4 dimensions from the Core Cultural dimensions category. We recommend prioritising the questions which relate to maintaining a high-quality experience, and otherwise removing questions which relate to factors you are less interested in.
|Question wording||Related Factor||Second Related Factor||Time to answer (s)|
|Captivation: it was absorbing and held my attention||Maintaining a high ‘quality’ experience||5-10|
|Rigour: it was well thought through and put together||Maintaining a high ‘quality’ experience||5-10|
|Relevance: it had something to say about the world in which we live||Maintaining a high ‘quality’ experience||5-10|
|Challenge: It was thought provoking||Maintaining a high ‘quality’ experience||5-10|
|Please write three words to describe your experience of this work||Maintaining a high ‘quality’ experience||15|
|How would you rate your experience overall?||Maintaining a high ‘quality’ experience||5|
|I felt connected with others experiencing this work. How much do you agree with this statement?||Promoting wellbeing amongst pre-existing audiences (battling loneliness and isolation)||Maintaining a high ‘quality’ experience||10|
|I felt that this digital experience met my need for cultural activity. How much do you agree with this statement?||Maintaining engagement with pre-existing audiences||10|
|Have you experienced this organisation’s work previously?||Maintaining engagement with pre-existing audiences||Expanding reach to establish new audiences||5|
|How did you find out about this work?||Expanding reach to establish new audiences||10|
|Why did you choose to experience this work?||Maintaining a high ‘quality’ experience||10|
|On what sort of device did you experience this work?||Maintaining a high ‘quality’ experience||Expanding reach to establish new audiences||10|
|Was is straightforward for you to find this work?||Maintaining a high ‘quality’ experience||Expanding reach to establish new audiences||5-10|
|Which one of these statements best describes how you experienced this work?||Expanding reach to establish new audiences||Maintaining engagement with pre-existing audiences||5-10|
|How long did you engage with this work for?||Maintaining a high ‘quality’ experience||10|
|Either during or afterwards, did you share anything about this work on social media? (Did you tweet about this event; share it on Facebook; tag the organisation on Instagram etc.?)||Expanding reach to establish new audiences||Maintaining a high ‘quality’ experience||5|
|Suppose we charged a fee to access a work like this. What would you be willing to pay?||Fundraising via requests for donations e.g. ‘pay what you can’ or lesser ticket prices||Maintaining a high ‘quality’ experience||10|
|How frequently do you experience cultural works digitally?||Expanding reach to establish new audiences||10|
|Whilst social distancing remains in place, would you seek out works like this again?||Maintaining a high ‘quality’ experience||Maintaining engagement with pre-existing audiences||10-15|
|Do you live in the UK?||Maintaining engagement with pre-existing audiences||Expanding reach to establish new audiences||5|
|What is your age?||Maintaining engagement with pre-existing audiences||Expanding reach to establish new audiences||5|
You can remove any of the above questions by clicking on its corresponding dustbin icon in the ‘Design’ page.
Dependent on your organisation’s current position and the specifics of the work you are evaluating, there are many other questions you may wish to consider.
If you are running work online via a means which allows for live commentary, (YouTube Premier, IGTV, Facebook Live etc) it would be appropriate to use a custom slider or dropdown in to ask for the agreement level to the following statement:
The ‘live’ comments made me feel part of an audience community experiencing this work.
If it is anticipated that the reach of the work will be reasonably small, say it’s an online workshop with a maximum of 30 participants, there would be a case to incorporate more free-text style questions, as this allows for richer insight. This is not recommended when there is a larger reach as conducting analysis on qualitative feedback can be resource-heavy. An example of a non-directive free-text question can be seen below:
Is there anything else you’d like to share about your experience?
If a piece of online work is targeting those that regularly engage with arts and culture, a question about the respondent’s expectations could be asked. A survey could contain a question such as the one below, presented as a dropdown, to provide a useful point of reference to learn how the public perceives the concept of online work and how that perception influences their responses to the quality-focused dimension questions.
Did the work meet your expectations?
It might be that you’re interested to learn how much effort has gone into the attendee’s experience of the work in order to establish what, if any, effect this has. For instance, if you’re surveying a piece of theatre which was pre-recorded and now being shown via digital means, you might be interested in knowing the extent to which a similar atmosphere was recreated at home: did the attendee close the curtains, turn off their phone and turn up the volume? If so, what impact does this have on their experience? It might even be that they organised with friends or family to watch simultaneously, and then have a video-conversation afterwards to discuss it. A multiple-choice question, such as the one below could provide much insight.
How did you recreate the atmosphere of the theatre at home?
The questions shown above are, of course, only a sample of additional questions you may wish to ask of your respondents. There are many other topics you may wish to consider, for instance:
- Providing the opportunity to join a mailing list
- Gauging appetite to engage with similar works in the near future
- Asking a Net Promoter Score question, enabling you to gain some understanding of the further reach of your work
Through a combination of the questions included within the template and carefully selected additional questions, the insight you could gain from this survey could be invaluable as you develop your future programme.
6. Survey delivery
When setting up your survey, it’s important to consider how it will be delivered in order to achieve a maximum number response. When a work is delivered online, the opportunity for follow up face to face interviews is not there. Similarly, it may be that you do not have access to the email addresses of those that have experienced the work. Dependent on how your work is delivered, there may be opportunities to:
- Encourage those that are experiencing the work to complete the survey via live commentary (consider IGTV, YouTube live etc.)
- Use social media to publicise the work and the subsequent survey
- If it’s a live stream, encourage those that are delivering the work to publicise the survey
- If the work is pre-recorded, have a survey’s QR code and URL included at various points encouraging attendees to respond throughout the work; for example, if a theatre production is being streamed, a QR code and URL could be displayed at the interval point and/or before the final credits. The presentation of a QR code would make your survey more accessible to those that are experiencing the work on one device and able to scan the QR code on their smartphone.
It is important to acknowledge that there is likely to be a lower response rate to surveys for online works, in comparison to in-person. This is partly due to the lower level of commitment required to experience the work, as mentioned in the Survey Length section of this guide. However, this is not to say that you won’t achieve great insight. All responses to your survey will contribute to your understanding of how your online works are perceived by those that experience them, and the impact your organisation is having as a whole.
The team at Counting What Counts is ready to support you in your use of the evaluation template, tailoring it to your needs. We can advise on question choice, delivery methods and which respondent categories you may wish to use.
Please do not hesitate to get in touch with any questions at all: email@example.com