1. Band 2 and 3 NPOs

 

Requirements

 

1. A band 2 or 3 NPO must complete four evaluations, where:

• At least two adhere to the Core evaluation type
• The remaining two adhere to either the Core or the Flexible evaluation type

The Flexible evaluation type is new and allows the NPO to choose their own dimensions for that evaluation. A comprehensive description of what both the Core and Flexible evaluation types must include is given below.

 

2. Within a month of completing each of the four evaluations, the NPO must then:

• Create an Insights Report and submit the data to Arts Council England (ACE) – this process occurs entirely within the Culture Counts platform.
• Share the Insights Report with their Relationship Manager via email – this requires the NPO to save the created Insights Report and send it to their Relationship Manager manually via email.

 

3. Band 2 and 3 NPOs must also create a second report type – the Annual Summary Report. This is an annual, combined report of the NPOʼs four submitted evaluations. It shows comparisons between the different evaluated works and provides opportunities for reflection. By 31st May, the NPO must:

• Create an Annual Summary Report – this occurs within the Culture Counts platform.
• Share the Annual Summary Report with their Relationship Manager and ACE via email – this requires the NPO to save the created Annual Summary Report and send it to their Relationship Manager manually via email. The NPO should also share the Annual Summary Report with the Arts Council Reporting team manually via email. The email address is: [email protected]

NPOs are encouraged to include additional questions within their surveys, and there are suggestions made within the platform to make this simpler for specific types of work (e.g. Online Works or Participatory). However, this is not mandatory.

Beyond the minimum requirement of four mandatory evaluations, all NPOs are welcome to undertake as many additional evaluations as they like. There is no requirement to share these evaluations with ACE and they can be designed in any way that NPOs choose.

 

Band 2 & 3 peer review

In addition to including at least one peer review in each evaluation, it is a mandatory requirement for all band 2 or 3 NPOs to register at least one peer reviewer from their organisation to the Peer Matching Resource, available via the Toolkit. This peer reviewer may be invited by other NPOs to provide a review.

It is expected that each NPO will provide four peer reviews over the course of each evaluation year. In some instances, this may not be possible. If the NPO is uncertain whether they will be able to provide four peer reviews per year, they should discuss this with their Relationship Manager.

 

2. Band 1 NPOs and Sector Support Organisations (SSOs)

Requirements

There is no requirement for band 1 NPOs or SSOs to use the Impact & Insight Toolkit but, if they do choose to use it, we expect them to follow the standards set out here for at least one Core or one Flexible evaluation each year. Findings from this evaluation should then be shared with their Relationship Manager, using the agreed reporting template.

If you are a Band 1 or SSO and are finding it difficult to secure a Peer Reviewer, please contact Counting What Counts to discuss your options.

 

3. Core and Flexible evaluation types

There are two types of evaluation which may contribute to NPO mandatory requirements, these are Core and Flexible. For both evaluation types, survey data resulting from specific questions must be collected from four respondent groups. These requirements are shown in the tables below.

 

Core evaluation type

Respondent Group (See Appendix A)	Self Prior	Self Post	Peer Post	Public
Required Questions	All dimensions in the ACE Core Dimensions - self and peer reviewers category*	All dimensions in the ACE Core Dimensions - self and peer reviewers category*	All dimensions in the ACE Core Dimensions - self and peer reviewers category*	• All dimensions in the ACE Core Dimensions - public respondents category* • Standard demographics**
Required Responses	At least 1 survey response	At least 1 survey response	At least 1 survey response	At least 1 survey response

*The dimensions in these categories i.e. ʻACE Core dimensions – self and peer reviewersʻ and ʻACE Core dimensions – public respondentsʼ are shown in Appendix B. (In the platform these appear as drop- down menus)

** Standard demographic questions are shown in Appendix C

 

Flexible evaluation type

Respondent Group (See Appendix A)	Self Prior	Self Post	Peer Post	Public
Required Questions	At least 4 dimensions from the Cultural Experience or ACE Participatory categories***	At least 4 dimensions from the Cultural Experience or ACE Participatory categories***	At least 4 dimensions from the Cultural Experience or ACE Participatory categories***	At least 4 dimensions from the Cultural Experience or ACE Participatory categories***
Required Responses	At least 1 survey response	At least 1 survey response	At least 1 survey response	At least 1 survey response

***The dimensions in these categories i.e. ʻCultural Experienceʻ and ʻACE Participatoryʻ are shown in Appendix B. (In the platform these appear as drop- down menus). Please note, in a Flexible evaluation at least 4 of the flexible dimensions must be the same across all respondent groups

 

4. NPO event/activity choice

We encourage NPOs to evaluate a range of their work (e.g. theatre productions in a main house and studio). NPOs can use the Impact & Insight Toolkit to evaluate any public facing work, including participatory work and work that is delivered online or outdoors.

NPOs will find the Toolkit most valuable if they use it to evaluate work where they can learn something interesting about the experiences of their audiences, or where they have a particular hypothesis about programming or marketing that they wish to test. NPOs can discuss their event choices with their Relationship Manager if that would be helpful.

 

5. NPO special circumstances

Where an organisation is primarily participatory, the NPO should discuss with their Relationship Manager whether more than two of their evaluations can be built around the Participatory dimensions instead.

Touring and receiving NPOs can use the Culture Counts platform to carry out a shared evaluation of

the same event. Each organisation will need to submit a separate report on the event to their Relationship Manager; this will count towards the quota of mandatory evaluations for both organisations.

Band 2 and 3 NPOs that do not have four events per year to evaluate, such as biennials or those with a limited public programme, should discuss suitable usage with their Relationship Manager.

 

6. Sampling guidance

Counting What Counts has produced guidance specifically on appropriate sample sizes for different types of event, including participatory events which typically involve smaller groups of people. The minimum number of public responses required in order to meet the mandatory requirements is one; however, it is advised that NPOs follow the guidance available to ensure maximum insight is gained.

 

7. What does ‘Submit Data’ mean?

By choosing to ‘Submit’ this data, you are agreeing for certain data within this evaluation to contribute towards the public-facing, anonymised, aggregate dataset.

Data that will be submitted comprises:

• Dimensions data (e.g. numerical scores for ‘Captivation: It was absorbing and held my attention’)
• Demographic data (age, gender and postcode)
• Data properties (e.g. artform)

This means that data from custom questions will not contribute to the public-facing, anonymous aggregate dataset.

 

If you are choosing to submit this data and:

1. You are submitting it on behalf of an organisation whose use of the Toolkit is mandated
2. The data is from an evaluation considered to be ‘complete’*

The submission will also mean that you have contributed towards your mandatory requirements.

 

The anonymised aggregate dataset powers two tools for you to explore, helping you and others to understand more about the impact of different arts and cultural experiences in the UK:

• Benchmarking Dashboard
• Insights Dashboard

All data collected and processed on behalf of Arts Council will be done adhering to their Data Principles.

 

If you would like more information on the technical aspect of submitting your data, please do get in touch and we will be happy to help.

 

*A ‘complete’ evaluation will include at least:

1 self prior response

1 self post response

1 peer post response

1 public response

 

Appendix A – Respondent groups

Public:

An Impact & Insight survey completed by a representative sample of the audience.

Self prior:

An Impact & Insight survey from a relevant member of sta outlining the creative intentions for the work (ideally NPOs should engage at least three members of sta for each evaluation).

Self post:

An Impact & Insight survey completed by the same sta members that completed the Prior survey.

Peer post:

An Impact & Insight survey completed by relevant peers (ideally NPOs should engage at least three peers for each evaluation).

Please know that it is possible to conduct the Self post and Peer post surveys within one survey.

 

Appendix B – Question categories

The following are displayed as ʻdrop downʼ menus in the Culture Counts platform.

ACE Core Dimensions – self and peer reviewers

• Captivation: It was absorbing and held my attention
• Challenge: It was thought-provoking
• Concept: It was an interesting idea
• Distinctiveness: It was different from things Iʼve experienced before
• Relevance: It has something to say about the world in which we live
• Rigour: It was well thought through and put together
• Excellence: Itʼs one of the best examples of its type that I have experienced
• Originality: It was ground-breaking
• Risk: The artists/curators really challenged themselves with this work

ACE Core Dimensions – public respondents

• Captivation: It was absorbing and held my attention
• Challenge: It was thought-provoking
• Concept: It was an interesting idea
• Distinctiveness: It was different from things Iʼve experienced before
• Relevance: It has something to say about the world in which we live
• Rigour: It was well thought through and put together

ACE Participatory

• Organisation: ʻThe project was well organisedʼ
• Responsiveness: ʻThe organisers responded well to the needs of the groupʼ
• Support: ʻPeople in the group supported each otherʼ
• Acceptance: ʻI felt like I could be myselfʼ
• Belonging: ʻThey made me feel part of the teamʼ
• Voice: ʻMy ideas were taken seriouslyʼ
• Enjoyment: ʻI had a good timeʼ
• Experimenting: ʻI felt comfortable trying new thingsʼ
• Friendship: ʻI felt close to other people involved in the projectʼ
• New People: ʻI got to know people who are different to meʼ
• Contribution: ʻI felt that my contribution matteredʼ
• Feedback: ʻI got helpful feedbackʼ
• Artistic Skills: ʻI improved my artistic skillsʼ
• Confidence: ʻI feel more confident about doing new thingsʼ
• Creativity: ʻI feel more able to express myself creativelyʼ
• Empathy: ʻIt helped me understand other peopleʼs points of viewʼ
• Identity: ʻIt helped me to see myself differentlyʼ
• Motivation: ʻI feel motivated to do more creative things in the futureʼ
• Skills: ʻI gained new skillsʼ
• Stretch: ʻI did something I didnʼt know I was capable ofʼ

Cultural Experience

• Concept: It was an interesting idea
• Distinctiveness: It was different from things Iʼve experienced before
• Challenge: It was thought-provoking
• Captivation: It was absorbing and held my attention
• Relevance: It has something to say about the world in which we live
• Rigour: It was well thought through and put together
• Risk: The artists/curators really challenged themselves with this work
• Originality: It was ground-breaking
• Excellence: Itʼs one of the best examples of its type that I have experienced
• Local Impact: ʻItʼs important that itʼs happening hereʼ
• Presentation: ʻIt was well produced and presentedʼ
• Imagination: ʻI expect it will open my mind to new possibilitiesʼ
• Cultural Contribution: ʻIt provides an important addition to the cultural life of the areaʼ
• Inquisitiveness: ʻIt made me want to find out more about the artworkʼ
• Enthusiasm: ʻI would come to something like this againʼ
• Escapism: ʻI was so immersed that I lost sense of timeʼ
• Coherence: ʻAll the different elements of the piece came together wellʼ
• Resonance: ʻIt will stay with me for a long timeʼ
• Beauty: ʻIt was visually beautifulʼ
• Relatability: ʻI could relate to the emotions expressed by the charactersʼ
• Independent Interpretation: ʻI felt free to make my own meaning with the workʼ
• Intimacy: ʻI felt close to the artists/performersʼ
• Appropriately Pitched: ʻIt respected the emotional maturity of its audienceʼ

*Please note that the wording of these statements is that of a post-event public survey and that within the platform, wording will vary slightly depending on the type of survey being created.

 

Appendix C – Standard demographics question set

Age: What is your age?

Gender: How would you describe your gender?

Postcode: What is your postcode?