1. Introduction

 

This guide explains why sample size is important and gives some tips on how to achieve a good sample that’s representative of your audience or participants.

 

2. Useful definitions

 

To better understand the process of sampling and why it matters, there are a few useful concepts and definitions to remember.

 

Population and sample

The population is the group of people that we want to learn about.

The sample is the people from that population who complete a survey.

For our purposes, the population is typically everyone who experienced a specific cultural project. We then send surveys to these people to collect a sample from that population.

 

Representativeness

This refers to how well the sample represents the wider population. For example, if we know the population includes a mix of young and old people, then the sample should also include a mix of young and old.

We want a representative sample as it means that estimates that we make using the sample are applicable to the population.

 

Sample estimates

When we look at the data from the sample and do something like calculate an average, what we are doing is using the sample to make an estimate for everyone in the population – even the people we didn’t survey.

For example, if we want to know what proportion of people have attended works from this organisation before, the sample (the people who filled in a survey) will give us an estimate of that proportion for the whole population (everyone who attended).

 

Sample size & margin of error

The margin of error tells us how precise our estimate is based on the sample that we collected. A smaller margin of error means the estimate is more precise.

Sample size matters as:

1. A larger sample gives us a smaller margin of error
2. A smaller margin of error means that the estimates are more precise
3. More precise estimates gives us more confidence in the patterns that we detect in the data

 

3. Sample size goals

 

A good general rule of thumb that you will see repeated on statistics websites is to aim to sample 10% of the population, with a minimum sample size of 100.

We recommend following this general guide where possible. However, we appreciate that for many organisations using the Toolkit, whilst their work might impact upon large numbers of people over the course of a year, it might be difficult to achieve these numbers for individual projects.

For smaller projects, we recommend collecting as many surveys as you can; see our best practice surveying section below for tips on how to do this. This table also gives indicative sample size targets to achieve specific margins of error for the dimensions questions:

Total number of audience members/visitors	Sample required for 5% margin of error	Sample required for 8% margin of error	Sample required for 10% margin of error
1000	141	61	40
500	123	57	39
250	98	52	37
100	62	40	29
50	40	28	23

 

If your organisation is primarily focussed on participatory projects with a smaller numbers of attendees, please use the participatory evaluation template and our analysis of your data will take this into account.

 

4. Best practice surveying

 

There are a range of actions one can take to increase the representativeness of a sample and the likelihood for people to respond.

We recommend that everyone should implement at least the basic steps below to give themselves a good chance of achieving a representative sample. The advanced tips require more resources to implement but will give the best chances of getting a larger sample.

 

Basic techniques

 

For all surveys – whether face-to-face or online

• Limit the length of the survey to 20 questions
• Tell the survey respondent approximately how long the survey will take – be truthful!
• Explain the survey’s purpose and why they are being asked to participate

For surveys sent via email

• Send the surveys within 24 hours of the work taking place
• Use a clear subject line for the email
• Customise the email to include organisational branding
• Send the survey to everyone whose email address you have access to
• Set and provide a deadline for when the survey will close. This might mean you miss out on the odd late response, but it prevents people completing a survey weeks after their experience, when their memory may have faded and the accuracy of their response may be poor.

 

 

Example email

Subject: Share your feedback on Our Cultural Event

To: my.email@organisation.com

 

Thank you for attending our event ABC.

We invite you to share your feedback on the production and your experience.

Your feedback will provide valuable input about the artistic quality of our work & assist us to improve our work in the future.

The survey will take approximately 3 minutes to complete.

If you have any questions about this survey, please contact our Data Insights Manager at [email protected].

Complete the survey

 

Advanced techniques

 

For all surveys – whether face-to-face or online:

• Offer incentives to those completing surveys, such as inclusion in a prize draw
• Use more than one sampling approach e.g., via email & face-to-face
• Use A/B testing to experiment and improve survey introduction wording, branding and survey timing
• Incorporate the surveying within the experience, rather than an afterwards activity

For online surveys

• Include survey links in social media content
• Include survey QR codes in marketing materials in venues

For face-to-face surveys:

• Ensure that the interviewers are representative of your audience; people are more receptive to requests from people that they can relate to
• Ensure that the interviewers are approachable and personable