1. Introduction

 

Use this guidance as a starting point for evaluating Outdoor Arts (OA). If outdoor or non-venue-based events are a crucial part of your programme, it’s important you choose an appropriate way to evaluate them, in order to understand the impact of your work. This guidance was developed in collaboration with National Portfolio Organisations (NPOs), Arts Council England (ACE) and expert peers.

 

2. Outdoor Arts question bank

Background

The development of this question bank began with a deep-dive review of previous evaluation research for OA and identifying a series of relevant questions. In March 2021, we used these questions to kickstart a discussion with NPOs, ACE and external peers, to hone the questions and finalise a question bank. We would particularly like to thank Without Walls for their significant contribution to this consultation process.

 

The questions

Question subject	Question text	Instruction text	Time to answer (s)	Question Type	Answer Options
Effect of Weather	Do you think that the weather affected your experience?	Please choose an option from the dropdown menu	10	Dropdown Question	• Yes - the weather had a positive effect on my experience • Yes - the weather had a negative effect on my experience • No - the weather had no effect on my experience • I'm not sure
Environmentally Friendly	Were you aware of the organisers encouraging attendees to be friendly towards the environment?	For instance, were there signs asking you not to drop litter? Or recycling points? Or was the use of single-use plastic discouraged?	5	Dropdown Question	• Yes • No • I'm not sure
Future Likelihood OA	How likely are you to go and experience other work outdoors following this?	Please choose an option from the dropdown menu	5	Dropdown Question	• Very likely • Likely • I don't know • Unlikely • Very unlikely
Indoor Arts Attendance	Do you consider yourself to be a keen indoor arts and culture attendee?	Think about whether you enjoy a trip to the theatre, exploring museums, visiting exhibitions, attending concerts, book readings…	5	Dropdown Question	• Yes • No • I'm not sure
Motivation for OA Attendance	Why did you attend this outdoor arts event?	Select from the list below	10	Multiple choice Question	• I was excited at the idea of being at an in-person, live event • To support the arts/cultural organisation • To enjoy the relaxed atmosphere • To be part of a community enjoying the event • To see companies/performers I admire • To connect with nature and the outdoors • To be part of a crowd • To be at an event that could be enjoyed by the whole family • To see this space brought to life with cultural activity • It wasn't going to cost me lots of money • It seemed a more COVID-safe idea than being indoors • To experience this work specifically
Outdoor Setting Contribution	In what way(s) did the outdoor setting add to your experience?	Imagine that you had experienced the same work indoors and then select from the list below	10	Multiple choice Question	• I felt more of a connection to the environment • I felt it was more accessible and welcoming to all • I felt it was a safer way to experience culture • I felt more of a connection to this community • I felt that being outside enhanced the art • I don't think that the outdoor setting added to the experience • I'm not sure
Payment for OA	Have you ever paid to attend an outdoor arts event?	Please choose from the options below	5	Yes/No Question	• Yes • No
Prior OA Attendance	Have you attended an outdoor arts event before?	Think about whether you have ever attended a festival, open-air concerts, open-air theatre, firework displays, drive-in cinema, carnivals…	5	Yes/No Question	• Yes • No

 

Accessing the question bank

In the Culture Counts platform, navigate to the ‘Design’ page of your survey, click on ‘Question bank’ and then select ‘Outdoor Arts’. Scroll through the list of questions and select the ones you want to add to your survey.

 

Using the question bank

When selecting your questions from the question bank, ask yourself “What other quantifiable factors will help me understand the impact of this work?”

Each question you ask should help you to understand:

1. Whether your intentions for the work you’re evaluating were met
2. What impact your work has had on its participants

There is no ‘one size fits all’ approach and we would encourage you to consider and select the questions that you feel will genuinely support you in your evaluation. Don’t ask questions for the sake of it!

Consider including a combination dimensions and question bank questions. A well-considered combination will not only provide you with more context and understanding for interpreting your results, but also provide some variety for your respondents, avoiding survey fatigue.

 

3. Evaluation design

Dimension selection

We recommend considering dimensions from Community Outcomes and Placemaking categories in our Dimensions Framework, as possible additions to your OA evaluation. An example of a dimension from these outcome areas is “I feel a sense of community here”. You may find something like this useful to capture, as the often hyper-local nature of OA events lends them to being experiences that can foster a sense of community and belonging.

There are also dimensions from the Qualities category you could consider; for instance, “I would come to something like this again”. A full list of dimensions is available and we suggest approaching the idea of adding additional dimensions with the question ‘What are my intentions for this piece of Outdoor Art?’.

 

Further questions

The OA question bank research and consultation process brought up several additional OA specific topics. Although these haven’t been included in the Question bank, they may be relevant for your OA work:

 

· Time of day

Will the time of day affect how someone experiences your work? It’s very possible it will! For example, if you are evaluating an all-day festival, the time of day a respondent attended the festival will present variables, such as: the weather, which specific events are taking place, how dark it is, how hungry someone might be…

If you think time of day may affect an attendee’s experience, we suggest asking:

Question: During which times were you at Festival X?

Why do we suggest asking for a time of visit, instead of what the weather was like or what specific works they encountered? Asking the time of day allows you to cross-reference their answer with information you already have access to, such as the weather, programme etc. It allows you to gain additional insight from your data, without having to add multiple questions, reducing survey fatigue.

 

· Safe space

‘Safe space’ has been included as there are safety considerations specific to outdoor works, such as: adequate lighting, uneven ground etc.

However, ‘safe space’ is a term that can mean many different things. Therefore, if you choose to include a question about feeling safe in a space, we encourage you to make the ‘type’ of safety explicit, for example:

Question 1: Did you feel that this was a safe space to express yourself, without judgement?

Question 2: Did you feel safe with the people around you in this space?

Question 3: Did the organisers set out and enforce adequate instruction to make you feel safe?

All three questions, should use ‘survey logic’ to provide respondents with a free text question to express how they felt unsafe, if they answer ‘no’.

 

· Length of stay influencers

Many outdoor works are ‘drop in’ events. It can be difficult to know what prompted someone to stay at the event for certain periods of time.

For instance, if it is a sunny day, an attendee might feel inclined to stay longer at an OA event; if an attendee felt there were too many people and it was crowded, they might be inclined to leave.

There will always be external factors you have no control over that may cause someone to leave earlier than expected (e.g. work commitments or child tiredness); however, there are other things that you may be able to influence for future occasions (e.g. crowd control or having covered areas in case of rain).

You will not know what can be done to encourage someone to stay longer at your event unless you ask why they chose to leave early. Therefore, if you are running a ‘drop in’ OA event, the following question schedule could be considered:

Question: Did you stay at this event for the length of time you expected to?

Answer options: Yes – I stuck to my expectations; No – I stayed longer than I expected to; No – I left after a shorter period of time than I expected; I didn’t have any expectations for how long I would stay

If they select ‘No – I stayed longer than I expected to’ use ‘survey logic’ to ask:

Question: Why did you stay longer than expected?

Answer options: I enjoyed myself more than I expected to; It was lovely to be outside in this weather; My external commitments were postponed or cancelled; There was a great vibe.

If they select ‘No – I left after a shorter period of time than I expected’ use ‘survey logic’ to ask:

Question: Why did you stay less time than expected?

Answer options: I didn’t enjoy myself as much as I expected to; It was unpleasant being outside in this weather; My external commitments (work/care/appointments etc.) were brought forwards…

 

· Makeup of party

It is often difficult to capture makeup of the parties attending ‘drop in’ events, especially if they are unticketed. Asking questions around party makeup can assist you in learning who your work is drawing in: families, groups of friends, solo attendees…

You would then be able to tailor or adapt your programming or marketing accordingly.

Surveys are most often completed by adults and, if ticketed, those that purchased the tickets. In order to delve into this a little further, one could ask the following series of questions:

Question: Did you visit this event with anyone else?

Answer options: Yes; No

If they say ‘yes’, use ‘survey logic’ to ask:

Question: What word best describes your relationship with those you attended with?

Answer options: Friend(s); Mixed age family; Adult family; Co-workers; Partner(s)

This could be explored further with number style questions, such as:

Question 1: How many adults (aged 18+) did you attend with?

Question 2: How many children (aged 0-17) did you attend with?

 

· Economic impact

We know that measuring an OA event’s economic impact is incredibly important. However, we also know that, in order to conduct a thorough and effective assessment, a different evaluation strategy and approach is required. That is why we have not included economic impact in the OA question bank. That said, we would like to point you in the direction of All About Audiences’ ‘Measuring the economic impact of a project or activity’ practical guide to kick-start your thinking. A thorough economic impact assessment can support future applications to deliver further works in outdoor spaces.

 

4. Survey delivery

 

Collecting survey responses can be more complex at outdoor and unticketed events, which is why we’d like to share some of our key tips and recommended survey distribution methods:

 

Interviewers

Benefits:

• Opportunity to gather more qualitative feedback from visitors
• Develop relationships with the community
• Can be used with a live internet connection or offline

Try placing interviewers on site to collect survey responses from visitors; this is a chance for your organisation to engage in conversation with visitors and to observe their body language and behaviours. We always suggest carrying a notebook to jot down any additional notes from your conversations. This qualitative data will complement the data collected through your surveys, providing deeper insight. Having real conversations with visitors can also help them to feel more connected to your organisation. For more advice on engaging interviewers, read our guidance for interviewers.

 

QR codes (quick response codes)

Benefits:

• Collect multiple survey responses at once
• Less resource-heavy
• Easy access for the visitor

QR codes are barcodes that, when scanned by a smartphone camera, take you to a specific website – in this case, use a survey’s QR code to direct visitors to your survey. You can find your survey’s QR code on the ‘Summary’ page – it has been autogenerated for you.

QR codes are a less resource-heavy option which allows your audience to use their own devices, and enables you to collect an unlimited number of responses simultaneously. Of course, you will only be able to capture responses from a particular demographic who own smartphones; however, it could be a good idea to use QR codes alongside other survey delivery methods. For example, you could have QR codes displayed around the site as well as interviewers collecting survey responses on tablets.

 

Paper surveys

Benefits:

• Doesn’t require an internet connection

Culture Counts is a digital platform, however, in certain instances we understand that this is not appropriate. Please see our paper survey guidance for more detail about whether or not paper surveys are appropriate for your event and how to arrange for a paper survey to be set up.

 

Postcards & social media

Benefits:

• Reach attendees after the event

Share your survey with audiences by displaying the QR code or URL on a printed postcard or in a social media post. Postcard advertisements can be relatively low-budget and can be used to display the online URL for the survey, or the QR code. For example, you could hand out a postcard as people are leaving the event, for visitors to complete the survey later. If you’re distributing the survey on social media, include the survey in a post specific to the event, rather than on your bio, to help make sure that you’re addressing the survey to people who actually attended the event.

You could of course use this postcard to advertise any other elements of your organisation including upcoming works or experiences, or links to social media.

 

Use an online survey later

Benefits:

• Reach attendees after the event

Outdoor and unticketed events can be a great place to encourage people to sign up to your organisation’s mailing list. If someone chooses to register, also ask if they would be happy to receive a one-off survey about the event. Once you’re back at your desk, you will be able to send these people your standard Online public survey. This way not only are you fostering engagement with the public, but you’re also maximising on your potential for collecting survey data.

The team at Counting What Counts is ready to support you in evaluating OA. We can advise on question choice, delivery methods and which respondent categories you may wish to use. Please do not hesitate to get in touch with any questions.