1. Introduction

The Impact & Insight Toolkit will assist Arts Council England and the CPP Programmes in the evaluation of the second key CPP Action Research question:

To what extent is the aspiration for excellence of art and culture and excellence of the process of engaging communities achieved?

The Toolkit will also assist Creative People and Places programmes and Arts Council England in their evaluation of how far CPP activity contributes to the aims of the programme, including but not limited to:

• Communities are empowered to take the lead in shaping local cultural provision
• The aspiration for excellence is central to the activity we will support – this covers both excellence of art and culture and excellence of the process of engaging communities
• Through these projects, we will demonstrate the power of arts and culture to enrich the lives of individuals and make positive changes in communities
• Activity will be radically different from what has happened before in the eligible places prior to the Creative People and Places programme

The Toolkit can be used to deepen your understanding of how well your intentions for your activity align with the experiences of your peers and your audiences.

 

2. Mandatory Requirements and Fulfilling Your Payment Condition

It is a mandatory requirement for all National Portfolio Creative People and Places Programmes (funded from 1 April 2022) to undertake a minimum number of Impact & Insight Toolkit evaluations each financial year.  This mandatory requirement links to the Payment Condition in your Funding Agreement for each May:

Data monitoring and quality evaluation monitoring reports for previous financial year in the format Arts Council England requests.

All CPPs must carry out at least four evaluations each financial year. The four evaluations should be representative of your programme across the year.  CPPs that do not have four events per year available to evaluate, should discuss suitable usage with their Relationship Manager

To fulfil your payment condition, by 31st May 2024 you must have:

• Created an Insights Report for each of your four evaluations
• Uploaded each of the reports to Grantium
• Have discussed your evaluations with your Relationship Manager

Please see the Creating an Insights Report guidance for support on this aspect of fulfilling the requirements.

For reference, please see our example of a CPP Insights Report.

 

3. What Should You Evaluate?

We encourage CPPs to evaluate a range of their work. CPPs will find the Toolkit most valuable if they use it to evaluate events where they can learn something interesting about the experiences of their audiences or participants, or where they have a particular hypothesis about programming or marketing that they wish to test. CPPs can discuss their event choices with their Relationship Manager if that would be helpful.

You can use the tool to evaluate any of the following activities:

• Events: e.g., performances, festivals, exhibitions
• Creative workshops and participatory experiences: e.g., craft workshops, singing and or music groups
• Decision making activities: e.g., community commissioning panels
• Co-created work: e.g., a large-scale event which has been co-created by the community

Beyond the four mandatory evaluations, all CPPs are welcome to undertake as many additional evaluations as they like.

A mandatory evaluation should involve an activity that can be evaluated using either the Core CPP Experience Dimensions for Audiences and/or participants (see below) OR the Core CPP Experience Dimensions for Core CPP Experience Dimensions for co-creators and/or community decision makers (see below).

The Core CPP Experience dimensions for audiences and/or participants

The following metrics are mandatory when evaluating things like:

• Events: e.g., performances, festivals, exhibitions
• Creative Workshops and participatory experiences: e.g., craft workshops, singing and or music groups

 

Distinctiveness: It was different from things I’ve experienced before

Relevance: It had something to say about modern society

Rigour: It was well thought through and put together

Local Impact: It’s important that it is happening here

Enthusiasm: I would come to something like this again

Cultural Contribution: It provides an important addition to the cultural life of the area

 

As a minimum each evaluation must consist of a pre-activity Impact & Insight survey from a relevant member of staff outlining the creative intentions for the activity (we strongly recommend that you encourage more than one member of staff to complete this) AND a post-event/activity Impact & Insight survey completed by a representative sample of the audience/participants/community decision makers/co-creators

You may also wish to include:

• One post-event Impact & Insight survey completed relevant peers
• One post-event Impact & Insight survey completed by a relevant member of staff

The Core CPP Experience dimensions for co-creators and/or community decision makers

The following metrics are mandatory when evaluating things like:

• Decision Making Activities: e.g., community commissioning panels
• Co-created work: e.g., a large-scale event which has been co-created by the community

 

Intention: I felt able to shape the intention of the project

Voice: My ideas were taken seriously

Experimenting: I felt comfortable trying new things

Intensity: I felt deeply involved in the process

Motivation: I felt motivated to do more things in the future

 

As a minimum each evaluation must consist of a pre-activity Impact & Insight survey from a relevant member of staff outlining the creative intentions for the activity (we strongly recommend that you encourage more than one member of staff to complete this) AND a post-event/activity Impact & Insight survey completed by a representative sample of the audience/participants/community decision makers/co-creators

You may also wish to include:

• One post-event Impact & Insight survey completed relevant peers
• One post-event Impact & Insight survey completed by a relevant member of staff

 

4. The Pre-activity Impact & Insight Survey (for staff)

The purpose of this survey is to outline the creative intentions for the activity. It will then be possible for these intentions to be compared with the outcomes and experiences of the audience/participants. This survey should be carried out ahead of the activity, by a relevant staff member – usually the person who organised it.  We strongly encourage more than one member of staff to complete this.

The survey respondent should answer the questions in a way that they would expect the audience/participants to respond, based on their intentions for the activity. For example:

If the activity’s intention is to allow co-creators a strong voice in the creation of a project, the self-respondent should award the metric ‘Voice: My ideas were taken seriously’ a high score.

 

5. The Post-activity Impact & Insight Survey (for Audiences/Participants/Community Decision Makers and Co-creators)

This survey should be carried out after the activity has taken place.  You should survey a representative sample of the audience/participants/community decision makers or co-creators.

Counting What Counts has produced guidance on appropriate sample sizes for different types of event, including participatory, which typically involve smaller groups of people. As a general rule of thumb, CPPs should aim for 70-100 public responses per event, where possible and depending on the type of activity you are evaluating.

 

6. Appendix – Respondent groups

• Self prior – An Impact & Insight survey from a relevant member of staff outlining the creative intentions for the work
• Self post – An Impact & Insight survey completed by the same staff members that completed the prior survey
• Peer post – An Impact & Insight survey completed by relevant peers
• Public – An Impact & Insight survey completed by a representative sample of the audience/participants/decision-makers/co-creators