Arts Council England – Mandatory Requirements for National Portfolio Organisations Using the Impact & Insight Toolkit; Financial year 2021-22
The information on this page was written by Arts Council England for NPOs using the Impact & Insight Toolkit. View/download a PDF version of this page here.
It is a mandatory requirement for band 2 and 3 National Portfolio Organisations (NPOs) to undertake a minimum amount of evaluation activity each year using the Impact & Insight Toolkit (the Toolkit).
This document provides a full description of what is required of band 2 and 3 NPOs with respect to the Toolkit, and guidance as to what the expectations are for band 1 NPOs.
Mandatory evaluation expectations for band 2 and 3 NPOs
For the year 2021-22, the mandatory evaluation expectations have been developed to provide more flexibility for NPOs using the Toolkit.
A band 2 or 3 NPO must complete four evaluations, where:
- At least two adhere to the Core evaluation type
- The remaining two adhere to either the Core or the Flexible evaluation type
The Flexible evaluation type is new and allows the NPO to choose their own dimensions for that evaluation. A comprehensive description of what both the Core and Flexible evaluation types must include is given below.
Within a month of completing each of the four evaluations, the NPO must then:
- Create an Insights Report and submit the data to Arts Council England (ACE) – this process occurs entirely within the Culture Counts platform.
- Share the Insights Report with their Relationship Manager via email – this requires the NPO to save the created Insights Report and send it to their Relationship Manager manually via email.
Band 2 and 3 NPOs must also create a second report type – the Annual Summary Report. This is an annual, combined report of the NPO’s four submitted evaluations. It shows comparisons between the different evaluated works and provides opportunities for reflection.
In preparation for June’s Grantium submission, the NPO must:
- Create an Annual Summary Report – this occurs within the Culture Counts platform.
- Submit the Annual Summary Report to ACE – this process occurs via Grantium as part of an NPO’s June submission.
NPOs are encouraged to include additional questions within their surveys, and there are suggestions made within the platform to make this simpler for specific types of work (e.g. Online Works or Participatory). However, this is not mandatory.
Beyond the minimum requirement of four mandatory evaluations, all NPOs are welcome to undertake as many additional evaluations as they like. There is no requirement to share these evaluations with ACE and they can be designed in any way that NPOs choose.
Evaluation Type Definitions
There are two types of evaluation which may contribute to band 2 and 3 NPO mandatory requirements, these are Core and Flexible.
For both evaluation types, survey data resulting from specific questions must be collected from four respondent groups. These requirements are shown in the tables below.
Core Evaluation Type
Respondent Group | Self Prior | Self Post | Peer Post | Public |
Required Questions | *ACE Core Dimensions – self and peer reviewers | *ACE Core Dimensions – self and peer reviewers | *ACE Core Dimensions – self and peer reviewers | *ACE Core Dimensions – public respondents
***Standard demographics |
Required Responses | At least 1 survey response | At least 1 survey response | At least 1 survey response | At least 1 survey response |
Flexible Evaluation Type
Respondent Group | Self Prior | Self Post | Peer Post | Public |
Required Questions | At least 4 dimensions from the **Cultural Experience or ACE Participatory categories
| At least 4 dimensions from the **Cultural Experience or ACE Participatory categories | At least 4 dimensions from the **Cultural Experience or ACE Participatory categories | At least 4 dimensions from the **Cultural Experience or ACE Participatory categories
|
Required Responses | At least 1 survey response | At least 1 survey response | At least 1 survey response | At least 1 survey response |
*The contents of question categories are given in the appendix
**At least 4 flexible dimensions must be the same across all respondent groups
*** Standard demographic questions are shown in the appendix
Event/activity choice
We encourage NPOs to evaluate a range of their work (e.g. theatre productions in a main house and studio). NPOs can use the Impact & Insight Toolkit to evaluate any public facing work, including participatory work and work that is delivered online or outdoors. NPOs will find the Toolkit most valuable if they use it to evaluate work where they can learn something interesting about the experiences of their audiences, or where they have a particular hypothesis about programming or marketing that they wish to test. NPOs can discuss their event choices with their Relationship Manager if that would be helpful.
Special Circumstances
Where an organisation is primarily participatory, the NPO should discuss with their Relationship Manager whether more than two of their evaluations can be built around the Participatory dimensions instead.
Touring and receiving NPOs can use the Culture Counts platform to carry out a shared evaluation of the same event. Each organisation will need to submit a separate report on the event to their Relationship Manager; this will count towards the quota of mandatory evaluations for both organisations.
Band 2 and 3 NPOs that do not have four events per year to evaluate, such as biennials or those with a limited public programme, should discuss suitable usage with their Relationship Manager.
Sampling Guidance
Counting What Counts has produced guidance specifically on appropriate sample sizes for different types of event, including participatory events which typically involve smaller groups of people. The minimum number of public responses required in order to meet the mandatory requirements is one; however, it is advised that NPOs follow the guidance available to ensure maximum insight is gained.
Peer Review
In addition to including at least one peer review in each evaluation, it is a mandatory requirement for all band 2 or 3 NPOs to register at least one peer reviewer from their organisation to the Peer Matching Resource, available via the Toolkit. This peer reviewer may be invited by other NPOs to provide a review. It is expected that each NPO will provide four peer reviews over the course of each evaluation year. In some instances, this may not be possible. If the NPO is uncertain whether they will be able to provide four peer reviews per year, they should discuss this with their Relationship Manager.
Evaluation expectations for band 1 NPOs
There is no requirement for band 1 NPOs to use the Impact & Insight Toolkit but, if they do choose to use it, we expect them to follow the standards set out here for at least one Core evaluation each year. Findings from this evaluation should then be shared with their Relationship Manager, using the agreed reporting template.
Appendix
Survey Type Definitions
Public | An Impact & Insight survey completed by a representative sample of the audience
|
Self Prior | An Impact & Insight survey from a relevant member of staff outlining the creative intentions for the work (ideally NPOs should engage at least three members of staff for each evaluation)
|
Self Post | An Impact & Insight survey completed by the same staff members that completed the Prior survey
|
Peer Post | An Impact & Insight survey completed by relevant peers (ideally NPOs should engage at least three peers for each evaluation) |
Please know that it is possible to conduct the Self Post and Peer Post surveys within one survey, as demonstrated in the original Mandatory Evaluation Template.
Question Categories
Dimension Category | Dimension | Statement |
ACE Core Dimensions – self and peer reviewers | Captivation | It was absorbing and held my attention |
Challenge | It was thought provoking | |
Concept | It was an interesting idea/programme | |
Distinctiveness | It was different from things I’ve experienced before | |
Relevance | It had something to say about the world in which we live | |
Rigour | It was well thought through and put together | |
Excellence | It was one of the best examples of its type that I have seen | |
Originality | It was ground-breaking | |
Risk | The artists/curators really challenged themselves with this work | |
ACE Core Dimensions – public respondents | Captivation | It was absorbing and held my attention |
Challenge | It was thought provoking | |
Concept | It was an interesting idea/programme | |
Distinctiveness | It was different from things I’ve experienced before | |
Relevance | It had something to say about the world in which we live | |
Rigour | It was well thought through and put together | |
ACE Participatory | Skills | I gained new skills |
Enjoyment | I had a good time | |
Experimenting | I felt comfortable trying new things | |
Authenticity | It felt like a real artistic experience | |
Motivation | I feel motivated to do more creative things in the future | |
Contribution | I felt like my contribution mattered | |
Confidence | I feel more confident about doing new things | |
Support | People in the group supported each other | |
Artistic Skills | I improved my artistic skills | |
Acceptance | I felt like I could be myself | |
Voice | My ideas were taken seriously | |
Responsiveness | The organisers responded well to the needs of the group | |
Organisation | The project was well organised | |
Stretch | I did something I didn’t know I was capable of | |
Feedback | I got helpful feedback | |
Empathy | It helped me understand other people’s points of view | |
Friendship | I felt close to other people involved in the project | |
New People | I got to know people who are different to me | |
Belonging | They made me feel part of the team | |
Creativity | I feel more able to express myself creatively | |
Identity | It helped me to see myself differently | |
Cultural Experience | Captivation | It was absorbing and held my attention |
Challenge | It was thought provoking | |
Concept | It was an interesting idea/programme | |
Distinctiveness | It was different from things I’ve experienced before | |
Relevance | It had something to say about the world in which we live | |
Rigour | It was well thought through and put together | |
Excellence | It was one of the best examples of its type that I have seen | |
Originality | It was ground-breaking | |
Risk | The artists/curators really challenged themselves with this work | |
Enthusiasm | I would come to something like this again | |
Presentation | It was well produced and presented | |
Local Impact | It’s important that it’s happening here | |
Cultural Contribution | It provides an important addition to the cultural like of the area | |
Imagination | It opened my mind to new possibilities | |
Innovation | It was introduced to the audience in a new way | |
Inquisitiveness | It made me want to find out more about the artwork | |
Authenticity | It had a connection to the State/Country we live in |
Standard demographics question set:
Age: What is your age?
Gender: How would you describe your gender?
Postcode: What is your postcode?
The information on this page was last updated on 26 March, 2021.