Developing the ‘Evaluation for Online Works Template’ to suit your Organisation and Works

As a user of the Impact & Insight Toolkit, there will be a new evaluation template in your dashboard entitled, ‘Evaluation for Online Works Template’.  This template has been designed in collaboration with arts and cultural organisations as well as funding bodies in order to establish a survey to capture insight into the ever-increasing use of digital media to experience arts and cultural works.  More information on its development can be found here.  This document explains how a Toolkit user might choose to develop and alter the existing template to suit their individual needs.

View/download a PDF of this page here

    1. The Template’s Purpose
    2. Dimension Usage
    3. Survey Length
    4. Suggested Question Alterations (Removals and/or Additions)
    5. Survey Delivery

 Sergio Rota ‘Big Sphere of Journals’ via Unsplash

The Template’s Purpose

The new evaluation template has been designed to support Toolkit users evaluate their online works effectively.  It provides the opportunity to receive feedback on the work being evaluated as well as indicating the respondent’s intention to experience similar works in the future.

Desk research indicated that there were multiple factors that are important across most online works:

  • Maintaining engagement with pre-existing audiences
  • Promoting wellbeing amongst pre-existing audiences (battling loneliness and isolation)
  • Maintaining a high ‘quality’ experience
  • Fundraising via requests for donations e.g. ‘pay what you can’ or lesser ticket prices
  • Expanding reach to establish new audiences

Each question included within the template ‘speaks’ to at least one of these factors stated above.  The question schedule has been carefully designed to collect data that will be of value to both the individual organisation and the sector overall, whilst still allowing space for customisation.

 

Dimension Usage

The template includes four of the core cultural experience dimensions.  These have been selected to provide continuity across both an NPO’s and CPP’s evaluations, regardless of the financial year and funding requirements. This will provide you with comparable data points which will enable you to understand the difference between work that is experienced online vs in-person.  The four featured in the evaluation template can be swapped for any of the other core cultural dimensions as you see fit.  As a reminder, the core cultural experience outcomes for NPOs are below, with those included in the template italicised:

  • Concept
  • Captivation
  • Distinctiveness
  • Challenge
  • Rigour
  • Relevance

Two of the selected core dimensions, Rigour and Relevance, are also part of the core set for CPPs.

We suggest that whichever core cultural experience dimensions you choose, they speak to the aims of the work and your organisation’s mission.

 

Survey Length

As always, we want you to obtain maximum insight without overwhelming respondents with too many questions and placing too high a demand on their time.

The importance of keeping a survey concise increases when surveying work experienced online.  The reason for this is that the attendee’s commitment to experiencing the work is likely to be less than when attending an in-person work – it’s much easier to open a web browser on a computer and go to a website than to attend a venue in person – and the amount of effort they might invest in answering a survey correlates strongly with their commitment.  However, there is not yet a body of research with well-defined answers for the effects of survey length for online works specifically.  Therefore, we are primarily recommending similar lengths for live works as outlined below.

When designing a survey for members of the public, we usually suggest aiming for a completion time of 3 minutes.  If the work being evaluated is much shorter, which is possible for online works, we would recommend reducing the survey length accordingly.  For example, for a work of 10-15 minutes long, we recommend reducing the survey to 1 minute only.

We would suggest aiming for a completion time of 6 minutes for peer reviews, with some flexibility. There is evidence to suggest that the drop-off rate increases after 8 minutes.

The new template is designed to be all-encompassing – something that can be used if you have no other data or if you’re uncertain how to gather further data.  It is therefore on the longer side of what we would suggest but it provides a solid grounding, enabling you to capture key data as you look to learn more about the impact your online work is having.

The public survey included within the template has been tested; its completion takes approximately 3 minutes.

There are two options to reduce survey length:

  1. Remove some questions which are less relevant for your evaluation activity
    • For options on making the survey shorter please see the Suggested Alterations section below.
  2. Split the questions across two different surveys and distribute both

 

Suggested Question Alterations

Removals

At 3 minutes long, you might feel that the public survey within the template is too lengthy, especially if you are confident in gathering additional data to supplement your findings via the Toolkit.  Therefore, we have recommendations for tailoring your survey to make it suitable for your organisation and work.

 

Recommendation 1

If you are confident in using the analytics generated by the platform on which you’re presenting your work (e.g. YouTube analytics), we recommend removing these questions as this information can typically be gained from the platform:

  • Multiple choice: On what sort of device did you experience this work?
  • Dropdown: How long did you engage with this work for?
    • Short-text: Why did you not stay for the full duration of the work?
  • Yes/no: Do you live in the UK?
    • Short-text: Which country do you live in?
    • Demographic: What is your postcode?
  • Demographic: What is your age?

 

Recommendation 2

The table below shows all the questions included in the survey, along with the factor the question relates to and an approximate time to answer. We recommend prioritising the questions which relate to maintaining a high-quality experience, and otherwise removing questions which relate to factors you are less interested in.

Question wordingRelated FactorSecond Related FactorTime to answer (s)
Captivation: it was absorbing and held my attentionMaintaining a high ‘quality’ experience5-10
Rigour: it was well thought through and put togetherMaintaining a high ‘quality’ experience5-10
Relevance: it had something to say about the world in which we liveMaintaining a high ‘quality’ experience5-10
Challenge: It was thought provokingMaintaining a high ‘quality’ experience5-10
Please write three words to describe your experience of this workMaintaining a high ‘quality’ experience15
How would you rate your experience overall?Maintaining a high ‘quality’ experience5
I felt connected with others experiencing this work. How much do you agree with this statement?Promoting wellbeing amongst pre-existing audiences (battling loneliness and isolation)Maintaining a high ‘quality’ experience10
I felt that this digital experience met my need for cultural activity. How much do you agree with this statement?Maintaining engagement with pre-existing audiences10
Have you experienced this organisation’s work previously?Maintaining engagement with pre-existing audiencesExpanding reach to establish new audiences5
How did you find out about this work?Expanding reach to establish new audiences10
Why did you choose to experience this work?Maintaining a high ‘quality’ experience10
On what sort of device did you experience this work?Maintaining a high ‘quality’ experienceExpanding reach to establish new audiences10
Was is straightforward for you to find this work?Maintaining a high ‘quality’ experienceExpanding reach to establish new audiences5-10
Which one of these statements best describes how you experienced this work?Expanding reach to establish new audiencesMaintaining engagement with pre-existing audiences5-10
How long did you engage with this work for?Maintaining a high ‘quality’ experience10
Either during or afterwards, did you share anything about this work on social media? (Did you tweet about this event; share it on Facebook; tag the organisation on Instagram etc.?)Expanding reach to establish new audiencesMaintaining a high ‘quality’ experience5
Suppose we charged a fee to access a work like this. What would you be willing to pay?Fundraising via requests for donations e.g. ‘pay what you can’ or lesser ticket pricesMaintaining a high ‘quality’ experience10
How frequently do you experience cultural works digitally?Expanding reach to establish new audiences10
Whilst social distancing remains in place, would you seek out works like this again?Maintaining a high ‘quality’ experienceMaintaining engagement with pre-existing audiences10-15
Do you live in the UK?Maintaining engagement with pre-existing audiencesExpanding reach to establish new audiences5
What is your age?Maintaining engagement with pre-existing audiencesExpanding reach to establish new audiences5

Having copied the survey into your own evaluation, you can remove any question by clicking on its corresponding dustbin icon in the ‘Design’ page.

 

Additions

Dependent on your organisation’s current position and the specifics of the work you are evaluating, there are many other questions you may wish to consider.

If you are running work online via a means which allows for live commentary, (YouTube Premier, IGTV, Facebook Live etc) it would be appropriate to use a custom slider or dropdown (see here for guidance on how to create these) to ask for the agreement level to the following statement:

The ‘live’ comments made me feel part of an audience community experiencing this work.

If it is anticipated that the reach of the work will be reasonably small, say it’s an online workshop with a maximum of 30 participants, there would be a case to incorporate more free-text style questions, as this allows for richer insight.  This is not recommended when there is a larger reach as conducting analysis on qualitative feedback can be resource-heavy.  An example of a non-directive free-text question can be seen below:

Is there anything else you’d like to share about your experience?

If a piece of online work is targeting those that regularly engage with arts and culture, a question about the respondent’s expectations could be asked.  A survey could contain a question such as the one below, presented as a dropdown, to provide a useful point of reference to learn how the public perceives the concept of online work and how that perception influences their responses to the quality-focused dimension questions.

Did the work meet your expectations?  

It might be that you’re interested to learn how much effort has gone into the attendee’s experience of the work in order to establish what, if any, effect this has.  For instance, if you’re surveying a piece of theatre which was pre-recorded and now being shown via digital means, you might be interested in knowing the extent to which a similar atmosphere was recreated at home: did the attendee close the curtains, turn off their phone and turn up the volume?  If so, what impact does this have on their experience?  It might even be that they organised with friends or family to watch simultaneously, and then have a video-conversation afterwards to discuss it.  A multiple-choice question, such as the one below could provide much insight.

How did you recreate the atmosphere of the theatre at home?

The questions shown above are, of course, only a sample of additional questions you may wish to ask of your respondents.  There are many other topics you may wish to consider, for instance:

  • Providing the opportunity to join a mailing list
  • Gauging appetite to engage with similar works in the near future
  • Asking a Net Promoter Score question, enabling you to gain some understanding of the further reach of your work

Through a combination of the questions included within the template and carefully selected additional questions, the insight you could gain from this survey could be invaluable as you develop your future programme.

 

Survey Delivery

When setting up your survey, it’s important to consider how it will be delivered in order to achieve a maximum number response.  When a work is delivered online, the opportunity for follow up face to face interviews is not there.  Similarly, it may be that you do not have access to the email addresses of those that have experienced the work.  Dependent on how your work is delivered, there may be opportunities to:

  • Encourage those that are experiencing the work to complete the survey via live commentary (consider IGTV, YouTube live etc.)
  • Use social media to publicise the work and the subsequent survey
  • If it’s a live stream, encourage those that are delivering the work to publicise the survey
  • If the work is pre-recorded, have a survey’s QR code and URL included at various points encouraging attendees to respond throughout the work; for example, if a theatre production is being streamed, a QR code and URL could be displayed at the interval point and/or before the final credits. The presentation of a QR code would make your survey more accessible to those that are experiencing the work on one device and able to scan the QR code on their smartphone.

It is important to acknowledge that there is likely to be a lower response rate to surveys for online works, in comparison to in-person.  This is partly due to the less commitment required to experience the work, as mentioned in the Survey Length section of this guide.  However, this is not to say that you won’t achieve great insight.  All responses to your survey will contribute to your understanding of how your online works are perceived by those that experience them, and the impact your organisation is having as a whole.

 

The team at Counting What Counts is ready to support you in your use of the evaluation template, tailoring it to your needs.  We can advise on question choice, delivery methods and which respondent categories you may wish to use.

 

Please do not hesitate to get in touch with any questions at all:

support@countingwhatcounts.co.uk

 

Page updated: 09/07/2020

 

X
X