As a user of the Impact & Insight Toolkit, there will be a new evaluation template in your dashboard entitled, ‘Evaluation for Online Works Template’. This template has been designed in collaboration with arts and cultural organisations as well as funding bodies in order to establish a survey to capture insight into the ever-increasing use of digital media to experience arts and cultural works. More information on its development can be found here. This document explains how a Toolkit user might choose to develop and alter the existing template to suit their individual needs.
The new evaluation template has been designed to support Toolkit users evaluate their online works effectively. It provides the opportunity to receive feedback on the work being evaluated as well as indicating the respondent’s intention to experience similar works in the future.
Desk research indicated that there were multiple factors that are important across most online works:
Each question included within the template ‘speaks’ to at least one of these factors stated above. The question schedule has been carefully designed to collect data that will be of value to both the individual organisation and the sector overall, whilst still allowing space for customisation.
If you are a National Portfolio Organisation you can use the ‘Evaluation for Online Works Template’ to meet your funding requirements. When you select the template you will then be able to choose between the Core or Flexible option.
A Core evaluation for NPOs must include all ACE Core dimensions as listed below.
When you tick the Core option the system will automatically populate your surveys with these dimensions.
Two of the selected core dimensions, Rigour and Relevance, are also part of the core set for CPPs.
These have been selected to provide continuity across both an NPOs and CPPs evaluations, regardless of the financial year and funding requirements. This will provide you with comparable data points which will enable you to understand the difference between work that is experienced online vs in-person.
A Flexible evaluation allows you to choose any 4 dimensions from the Core Cultural dimensions or Participatory dimensions categories. Please see here for a list of the dimensions available in each of these categories.
We suggest that whichever dimensions you choose, they speak to the aims of the work and your organisation’s mission.
As always, we want you to obtain maximum insight without overwhelming respondents with too many questions and placing too high a demand on their time.
The importance of keeping a survey concise increases when surveying work experienced online. The reason for this is that the attendee’s commitment to experiencing the work is likely to be less than when attending an in-person work – it’s much easier to open a web browser on a computer and go to a website than to attend a venue in person – and the amount of effort they might invest in answering a survey correlates strongly with their commitment. However, there is not yet a body of research with well-defined answers for the effects of survey length for online works specifically. Therefore, we are primarily recommending similar lengths for live works as outlined below.
When designing a survey for members of the public, we usually suggest aiming for a completion time of 3 minutes. If the work being evaluated is much shorter, which is possible for online works, we would recommend reducing the survey length accordingly. For example, for a work of 10-15 minutes long, we recommend reducing the survey to 1 minute only.
We would suggest aiming for a completion time of 6 minutes for peer reviews, with some flexibility. There is evidence to suggest that the drop-off rate increases after 8 minutes.
The Evaluation for Online Works Template is designed to be all-encompassing – something that can be used if you have no other data or if you’re uncertain how to gather further data. It is therefore on the longer side of what we would suggest but it provides a solid grounding, enabling you to capture key data as you look to learn more about the impact your online work is having.
There are two options to reduce survey length:
At 3 minutes long, you might feel that the public survey within the template is too lengthy, especially if you are confident in gathering additional data to supplement your findings via the Toolkit. Therefore, we have recommendations for tailoring your survey to make it suitable for your organisation and work.
If you are confident in using the analytics generated by the platform on which you’re presenting your work (e.g. YouTube analytics), we recommend removing these questions as this information can typically be gained from the platform:
The table below shows all the questions included in the survey template, along with the factor the question relates to and an approximate time to answer – please note that this is the Flexible option i.e., 4 dimensions from the Core Cultural dimensions category. We recommend prioritising the questions which relate to maintaining a high-quality experience, and otherwise removing questions which relate to factors you are less interested in.
You can remove any of the above questions by clicking on its corresponding dustbin icon in the ‘Design’ page.
Dependent on your organisation’s current position and the specifics of the work you are evaluating, there are many other questions you may wish to consider.
If you are running work online via a means which allows for live commentary, (YouTube Premier, IGTV, Facebook Live etc) it would be appropriate to use a custom slider or dropdown in to ask for the agreement level to the following statement:
The ‘live’ comments made me feel part of an audience community experiencing this work.
If it is anticipated that the reach of the work will be reasonably small, say it’s an online workshop with a maximum of 30 participants, there would be a case to incorporate more free-text style questions, as this allows for richer insight. This is not recommended when there is a larger reach as conducting analysis on qualitative feedback can be resource-heavy. An example of a non-directive free-text question can be seen below:
Is there anything else you’d like to share about your experience?
If a piece of online work is targeting those that regularly engage with arts and culture, a question about the respondent’s expectations could be asked. A survey could contain a question such as the one below, presented as a dropdown, to provide a useful point of reference to learn how the public perceives the concept of online work and how that perception influences their responses to the quality-focused dimension questions.
Did the work meet your expectations?
It might be that you’re interested to learn how much effort has gone into the attendee’s experience of the work in order to establish what, if any, effect this has. For instance, if you’re surveying a piece of theatre which was pre-recorded and now being shown via digital means, you might be interested in knowing the extent to which a similar atmosphere was recreated at home: did the attendee close the curtains, turn off their phone and turn up the volume? If so, what impact does this have on their experience? It might even be that they organised with friends or family to watch simultaneously, and then have a video-conversation afterwards to discuss it. A multiple-choice question, such as the one below could provide much insight.
How did you recreate the atmosphere of the theatre at home?
The questions shown above are, of course, only a sample of additional questions you may wish to ask of your respondents. There are many other topics you may wish to consider, for instance:
Through a combination of the questions included within the template and carefully selected additional questions, the insight you could gain from this survey could be invaluable as you develop your future programme.
When setting up your survey, it’s important to consider how it will be delivered in order to achieve a maximum number response. When a work is delivered online, the opportunity for follow up face to face interviews is not there. Similarly, it may be that you do not have access to the email addresses of those that have experienced the work. Dependent on how your work is delivered, there may be opportunities to:
It is important to acknowledge that there is likely to be a lower response rate to surveys for online works, in comparison to in-person. This is partly due to the lower level of commitment required to experience the work, as mentioned in the Survey Length section of this guide. However, this is not to say that you won’t achieve great insight. All responses to your survey will contribute to your understanding of how your online works are perceived by those that experience them, and the impact your organisation is having as a whole.
The team at Counting What Counts is ready to support you in your use of the evaluation template, tailoring it to your needs. We can advise on question choice, delivery methods and which respondent categories you may wish to use.
Please do not hesitate to get in touch with any questions at all: firstname.lastname@example.org
Powered by BetterDocs