IIT Mandatory Evaluation Guidance (21-22)

If you are unsure about your organisation’s band, please check your funding agreement.

Band 2 and 3 NPOs

Requirements

1. For the year 2021-22, the mandatory evaluation expectations have been developed to provide more flexibility for NPOs using the Toolkit.

A band 2 or 3 NPO must complete four evaluations, where:

  • At least two adhere to the Core evaluation type
  • The remaining two adhere to either the Core or the Flexible evaluation type

The Flexible evaluation type is new and allows the NPO to choose their own dimensions for that evaluation. A comprehensive description of what both the Core and Flexible evaluation types must include is given below.
 

2. Within a month of completing each of the four evaluations, the NPO must then:

  • Create an Insights Report and submit the data to Arts Council England (ACE) – this process occurs entirely within the Culture Counts platform.
  • Share the Insights Report with their Relationship Manager via email – this requires the NPO to save the created Insights Report and send it to their Relationship Manager manually via email.

 
3. Band 2 and 3 NPOs must also create a second report type – the Annual Summary Report. This is an annual, combined report of the NPO’s four submitted evaluations. It shows comparisons between the different evaluated works and provides opportunities for reflection. In preparation for June’s Grantium submission, the NPO must:

  • Create an Annual Summary Report – this occurs within the Culture Counts platform.
  • Submit the Annual Summary Report to ACE – this process occurs via Grantium as part of an NPO’s June submission.

NPOs are encouraged to include additional questions within their surveys, and there are suggestions made within the platform to make this simpler for specific types of work (e.g. Online Works or Participatory). However, this is not mandatory.

Beyond the minimum requirement of four mandatory evaluations, all NPOs are welcome to undertake as many additional evaluations as they like. There is no requirement to share these evaluations with ACE and they can be designed in any way that NPOs choose.
 

Band 2 & 3 peer review

In addition to including at least one peer review in each evaluation, it is a mandatory requirement for all band 2 or 3 NPOs to register at least one peer reviewer from their organisation to the Peer Matching Resource, available via the Toolkit. This peer reviewer may be invited by other NPOs to provide a review.

It is expected that each NPO will provide four peer reviews over the course of each evaluation year. In some instances, this may not be possible. If the NPO is uncertain whether they will be able to provide four peer reviews per year, they should discuss this with their Relationship Manager.
 

 

Band 1 NPOs and Sector Support Organisations (SSOs)

Requirements

There is no requirement for band 1 NPOs or SSOs to use the Impact & Insight Toolkit but, if they do choose to use it, we expect them to follow the standards set out here for at least one Core or one Flexible evaluation each year. Findings from this evaluation should then be shared with their Relationship Manager, using the agreed reporting template.

If you are a Band 1 or SSO and are finding it difficult to secure a Peer Reviewer, please contact Counting What Counts to discuss your options.

 

Core and Flexible evaluation type  

There are two types of evaluation which may contribute to NPO mandatory requirements, these are Core and Flexible. For both evaluation types, survey data resulting from specific questions must be collected from four respondent groups. These requirements are shown in the tables below.

Core evaluation type

Flexible evaluation type

*The contents of these question categories i.e. ACE Core dimensions – self and peer reviewers; ACE Core dimensions – public respondents; Cultural Experience; and ACE Participatory are shown in Appendix B. (In the platform these appear as drop-down menus)

**In a Flexible evaluation at least 4 of the flexible dimensions must be the same across all respondent groups

*** Standard demographic questions are shown in Appendix C

 

NPO event/activity choice

We encourage NPOs to evaluate a range of their work (e.g. theatre productions in a main house and studio). NPOs can use the Impact & Insight Toolkit to evaluate any public facing work, including participatory work and work that is delivered online or outdoors.

NPOs will find the Toolkit most valuable if they use it to evaluate work where they can learn something interesting about the experiences of their audiences, or where they have a particular hypothesis about programming or marketing that they wish to test. NPOs can discuss their event choices with their Relationship Manager if that would be helpful.

 

NPO special circumstances

Where an organisation is primarily participatory, the NPO should discuss with their Relationship Manager whether more than two of their evaluations can be built around the Participatory dimensions instead.

Touring and receiving NPOs can use the Culture Counts platform to carry out a shared evaluation of the same event. Each organisation will need to submit a separate report on the event to their Relationship Manager; this will count towards the quota of mandatory evaluations for both organisations.

Band 2 and 3 NPOs that do not have four events per year to evaluate, such as biennials or those with a limited public programme, should discuss suitable usage with their Relationship Manager.

 

Creative People and Places (CPP)

Introduction

This Impact & Insight toolkit has been developed from the Quality Metrics pilot in 2016. Counting What Counts successfully tendered to provide the service to National Portfolio Organisations in our 2018-2022 round of funding.

Arts Council England have since, introduced it to be used by New Creative People and Places projects as a pilot, to test whether this model can be effectively used to address our second key CPP Research question:

To what extent is the aspiration for excellence of art and culture and excellence of the process of engaging communities achieved?

We are hoping use of the toolkit will assist Creative People and Places projects and Arts Council England in their evaluation of how far CPP activity contributes to the aims of the programme, including but not limited to:

  • Communities are empowered to take the lead in shaping local cultural provision:
  • The aspiration for excellence is central to the activity we will support – this covers both excellence of art and culture and excellence of the process of engaging communities;
  • Through these projects, we will demonstrate the power of arts and culture to enrich the lives of individuals and make positive changes in communities;
  • Activity will be radically different from what has happened before in the eligible places prior to the Creative People and Places programme.

The toolkit can be used to deepen your understanding of how well your intentions for your activity align with the experiences of your peers and your audiences.

 

Creative People and Places (CPP) – funded from October 2019

It is a mandatory requirement for New Creative People and Places projects (those funded from October 2019) to undertake a minimum number of Impact & Insight Toolkit evaluations each year and share findings with their Relationship Manager via an agreed reporting template. There is a set of standards that these mandatory evaluations must adhere to.

In each of the evaluation years, all new CPPs must carry out at least four evaluations using the Impact & Insight Toolkit and, as part of their payment condition in May of each year, which states:

Submission of your annual report summarising your Impact and Insights Toolkit evaluations

Each project should submit an Annual Evaluation Report which shares their findings with their Relationship Manager. Guidance about how to create an evaluation report is available on the Impact & Insight toolkit website.

The four evaluations should be representative of your programme across the year and can include:

  • Events: e.g., performances, festivals, exhibitions;
  • Creative workshops and participatory experiences: e.g., craft workshops, singing and or music groups;
  • Decision making activities: e.g., community commissioning panels;
  • Co-created work: e.g., a large-scale event which has been co-created by the community.

As a minimum each evaluation must consist of:

  • one pre-activity Impact & Insight survey from a relevant member of staff outlining the creative intentions for the activity (we strongly recommend that you encourage more than one member of staff to complete this);
  • post-event Impact & Insight survey completed by a representative sample of the audience (this can be audience in the traditional sense or participants)

You may also wish to include:

  • one post-event Impact & Insight survey completed by relevant peer
  • post-event Impact & Insight survey completed by a relevant member of staff.

CPPs that do not have four events per year available to evaluate, should discuss suitable usage with their Relationship Manager
 

Creative People and Places (CPP) – funded before October 2019

There is no requirement for Existing CPPs, funded before October 2019 to use the Impact & Insight Toolkit, but if they do choose to use it, we expect them to follow the standards set out for CPPs funded after October 2019. If an existing CPP chooses to use the toolkit, they should discuss how many evaluations it would be appropriate for them to carry out each year with their Relationship Manager and share findings from those evaluations with their Relationship Manager.

 

Activity choice and Evaluation templates (CPPs)

A mandatory evaluation should involve an activity that can be evaluated using one of the following evaluation templates available in the Toolkit:

• Core CPP Experience Dimensions for Audiences and/or participants
• Core CPP Experience Dimensions for Core CPP Experience Dimensions for co-creators and/or community decision makers

A comprehensive description of these templates is given below.

We encourage CPPs to evaluate a range of their work. CPPs will find the toolkit most valuable if they use it to evaluate events where they can learn something interesting about the experiences of their audiences or participants, or where they have a particular hypothesis about programming or marketing that they wish to test. CPPs can discuss their event choices with their Relationship Manager if that would be helpful.

CPPs and NPOs can evaluate the same event, if it was a joint commission, however they must carry out two separate evaluations, that meet the requirements and use the mandatory metrics for NPO and CPP evaluations. They must also submit a separate report on the event to their Relationship Manager; this will count towards the quota of mandatory evaluations for both organisations.

Beyond the four mandatory evaluations, all CPPs are welcome to undertake as many additional evaluations as they like.

 

Core CPP Experience dimensions for audiences and/or participants template

The following metrics are mandatory when evaluating things like:

  • Events: e.g., performances, festivals, exhibitions;
  • Creative Workshops and participatory experiences: e.g., craft workshops, singing and or music groups;

The questions should be asked of a random sample of a public audience. These mandatory questions could also be used in the context of an artist led creative workshop for participants, if they are relevant. If you are un-sure please check with your Relationship Manager. Please note these are the core mandatory questions, projects may choose to add any additional metrics you wish. The first three metrics are also mandatory for Band 2 and 3 National Portfolio Organisations.

Distinctiveness: It was different from things I’ve experienced before
Relevance: It had something to say about the world in which we live
Rigour: It was well thought through and put together
Local Impact: It’s important that it is happening here
Enthusiasm: I would come to something like this again
Cultural Contribution: It provides an important addition to the cultural life of the area.

 

Core CPP Experience dimensions for co-creators and/or community decision makers template

The following metrics are mandatory when evaluating decision making activities, for example like:

• Decision Making Activities: e.g., community commissioning panels or ‘Go and See’ type events;
• Co-created work: e.g., a large-scale event which has been co-created by the community / members of the public;

The questions would be asked of members of the panel or participants in a co-created project. Please note these are the core mandatory questions, projects may choose to add any additional metrics you wish.

Intention: I felt able to shape the intention of the project
Voice: My ideas were taken seriously
Experimenting: I felt comfortable trying new things
Intensity: I felt deeply involved in the process
Motivation: I felt motivated to do more things in the future

 

Guidance on pre-activity Impact & Insight survey from a relevant member of staff (for CPPs)

This survey should be carried out ahead of the activity, by a relevant staff member – usually the person who organised it. The purpose of this survey is to outline the creative intentions for the activity, which it will then be possible to compare with the outcomes and experiences of the audience/participants.

The survey respondent should answer the questions in a way that they would expect the audience/participants to respond, based on their intentions for the activity. For example:

If the activity intention is to allow co-creators a strong voice in the creation of a project, the self-respondent should award the metric ‘Voice: My ideas were taken seriously’ a high score.

Sampling guidance

Counting What Counts has produced guidance specifically on appropriate sample sizes for different types of event, including participatory events which typically involve smaller groups of people. The minimum number of public responses required in order to meet the mandatory requirements is one; however, it is advised that NPOs follow the guidance available to ensure maximum insight is gained.

Appendix A – Survey type definitions

Public

An Impact & Insight survey completed by a representative sample of the audience.

Self prior

An Impact & Insight survey from a relevant member of staff outlining the creative intentions for the work (ideally NPOs should engage at least three members of staff for each evaluation).

Self post

An Impact & Insight survey completed by the same staff members that completed the Prior survey.

Peer post

An Impact & Insight survey completed by relevant peers (ideally NPOs should engage at least three peers for each evaluation).

Please know that it is possible to conduct the Self post and Peer post surveys within one survey.

 

Appendix B – Question categories

The following are displayed as ‘drop down’ menus in the Culture Counts platform.

ACE Core Dimensions – self and peer reviewers

• Captivation: It was absorbing and held my attention
• Challenge: It was thought-provoking
• Concept: It was an interesting idea
• Distinctiveness: It was different from things I’ve experienced before
• Relevance: It has something to say about the world in which we live
• Rigour: It was well thought through and put together
• Excellence: It’s one of the best examples of its type that I have seen
• Originality: It was ground-breaking
• Risk: The artists/curators really challenged themselves with this work

 

ACE Core Dimensions – public respondents

• Captivation: It was absorbing and held my attention
• Challenge: It was thought-provoking
• Concept: It was an interesting idea
• Distinctiveness: It was different from things I’ve experienced before
• Relevance: It has something to say about the world in which we live
• Rigour: It was well thought through and put together

 

ACE Participatory

• Organisation: ‘The project was well organised’
• Responsiveness: ‘The organisers responded well to the needs of the group’
• Support: ‘People in the group supported each other’
• Acceptance: ‘I felt like I could be myself’
• Belonging: ‘They made me feel part of the team’
• Voice: ‘My ideas were taken seriously’
• Authenticity: ‘It felt like a real artistic experience’
• Enjoyment: ‘I had a good time’
• Experimenting: ‘I felt comfortable trying new things’
• Friendship: ‘I felt close to other people involved in the project’
• New People: ‘I got to know people who are different to me’
• Contribution: ‘I felt that my contribution mattered’
• Feedback: ‘I got helpful feedback’
• Artistic Skills: ‘I improved my artistic skills’
• Confidence: ‘I feel more confident about doing new things’
• Creativity: ‘I feel more able to express myself creatively’
• Empathy: ‘It helped me understand other people’s points of view’
• Identity: ‘It helped me to see myself differently’
• Motivation: ‘I feel motivated to do more creative things in the future’
• Skills: ‘I gained new skills’
• Stretch: ‘I did something I didn’t know I was capable of’

 

Cultural Experience

• Concept: It was an interesting idea
• Distinctiveness: It was different from things I’ve experienced before
• Challenge: It was thought-provoking
• Captivation: It was absorbing and held my attention
• Relevance: It has something to say about the world in which we live
• Rigour: It was well thought through and put together
• Risk: The artists/curators really challenged themselves with this work
• Originality: It was ground-breaking
• Excellence: It’s one of the best examples of its type that I have seen
• Local Impact: ‘It’s important that it’s happening here’
• Presentation: ‘It was well produced and presented’
• Imagination: ‘I expect it will open my mind to new possibilities’
• Authenticity: ‘It had a connection to the State/Country we live in’
• Innovation: ‘It was introduced to the audience in a new way’
• Cultural Contribution: ‘It provides an important addition to the cultural life of the area’
• Inquisitiveness: ‘It made me want to find out more about the artwork’
• Enthusiasm: ‘I would come to something like this again’
• Escapism: ‘I was so immersed that I lost sense of time’
• Coherence: ‘All the different elements of the piece came together well’
• Connectivity: ‘The work/experience spoke to me on a personal level’
• Resonance: ‘It will stay with me for a long time’
• Beauty: ‘It was visually beautiful’
• Relatability: ‘I could relate to the emotions expressed by the characters’
• Catharsis: ‘I felt differently after this experience’
• Independent Interpretation: ‘I felt free to make my own meaning with the work’
• Intimacy: ‘I felt close to the artists/performers’
• Appropriately Pitched: ‘It respected the emotional maturity of its audience’

*Please note that the wording of these statements is that of a post-event public survey and that within the platform, wording will vary slightly depending on the type of survey being created.

 

Placemaking

• Local Impact: ‘It’s important it’s happening here’
• Connection: ‘It helped me to feel connected to people in the community’
• Heritage: ‘It made me feel connected to a shared history/culture’
• Participation: ‘It encouraged me to participate in community activities’
• Celebration: ‘It celebrated my own cultural heritage’
• Nature: ‘It made me feel connected to nature’
• Access: ‘It gave me the opportunity to access cultural activities’
• Place: ‘It made me feel proud of my local area’
• Awareness: ‘It opened my eyes to issues in the environment’
• Understanding: ‘It helped me to better understand the place and people where I live’
• Content: It reflected a broad and inclusive range of voices’
• Respect: ‘It gave me greater respect for cultural diversity’
• Community: I feel a sense of community here’
• Pride: It strengthened my cultural pride’

 

Frequently Used Custom

• Heritage: ‘I was clear about what we were here to achieve’
• Insight: ‘It helped me gain new insight or knowledge’
• Interpretation: ‘The information about the exhibition was clear and helpful’
• Celebration: ‘It celebrated my own cultural heritage’
• Family: ‘I explored and learnt together as a family’
• Networks: ‘It connected me with other people in my field
• Achievement: ‘I was amazed by what we achieved’
• Intensity: ‘I felt deeply involved in the process’
• Opportunity: ‘The project opened up new opportunities for me’
• Intention: ‘I felt able to shape the intention of the project’
• Clarity: ‘I was clear about what we were all here to do’
• Creative Legacy: ‘I now have creative ambitions I didn’t have before’
• Worldview: ‘It helped me understand something new about the world’
• Mental Wellbeing: It had a positive impact on my mental wellbeing
• Trust: ‘I trusted the other people involved’
• Respect: ‘I was treated as an equal’
• Physical Health: ‘It has a positive impact on my physical health’
• Vibe in place: ‘I enjoy the vibrancy here’

 

ACE Literature Specific

• Completion: ‘I feel more confident about knowing when my work is finished’
• Language: ‘I enjoyed the author’s use of language’
• Skills (writing): ‘I improved my technical writing skills’
• Network: ‘I have increased my network of people to support with my writing’
• Illustration: ‘The pictures were just right for the story’
• Layout: ‘I liked the way the words and pictures appeared together on the page’

 

ACE Dance Specific

• Balance: ‘The elements were well-balanced’
• Cadence: ‘It was well-paced’

 

Combined Arts Specific

• Blend: ‘I appreciated the mix of art forms in this work’

 

ACE Music Specific

• Sound: ‘I loved the sound of this performance’
• Unity: ‘The musicians performed as one voice’
• Use of musical elements: ‘The piece exploited different musical elements in an interesting way (Tempo, timbre, structure, dynamics, pitch/harmony, texture, beat/rhythm)’

 

ACE Visual Art Specific

• Interpretation: ‘The information about the exhibition was clear and helpful’
• Place: ‘It gave me a new perspective on its location’
• Visual Impact: ‘It drew me in and made me look’

 

ACE Museum Specific

• Interpretation:’ The information about the exhibition was clear and helpful’
• Visual Impact: ‘It drew me in and made me look’

 

Appendix C – Standard demographics question set

Age: What is your age?
Gender: How would you describe your gender?
Postcode: What is your postcode?

Powered by BetterDocs

X
X