1. Introduction

 

The Impact & Insight Toolkit has been developed from the Quality Metrics pilot in 2016. Counting What Counts successfully tendered to provide the service to National Portfolio Organisations in our 2018-2022 round of funding.

Arts Council England piloted the Toolkit with 12 New Creative People and Places programmes between 2019 – 22. Arts Council England would now like to roll the Toolkit out across the 39 Creative People and Places Programmes that are part of the CPP National Portfolio 2022 – 25.

The Toolkit will assist Arts Council England and the CPP Programmes in the evaluation of the second key CPP Action Research question:

 

To what extent is the aspiration for excellence of art and culture and excellence of the process of engaging communities achieved?

The Toolkit will also assist Creative People and Places programmes and Arts Council England in their evaluation of how far CPP activity contributes to the aims of the programme, including but not limited to:

• Communities are empowered to take the lead in shaping local cultural provision
• The aspiration for excellence is central to the activity we will support – this covers both excellence of art and culture and excellence of the process of engaging communities
• Through these projects, we will demonstrate the power of arts and culture to enrich the lives of individuals and make positive changes in communities
• Activity will be radically different from what has happened before in the eligible places prior to the Creative People and Places programme

The Toolkit can be used to deepen your understanding of how well your intentions for your activity align with the experiences of your peers and your audiences.

 

2. Mandatory evaluation expectations 2022-23

 

It is a mandatory requirement for all National Portfolio Creative People and Places Programmes (funded from 1 April 2022) to undertake a minimum number of Impact & Insight Toolkit evaluations for the financial year 2022 – 23 and share findings with their Relationship Manager via an agreed reporting template. This mandatory requirement links to the Payment Condition in your Funding Agreement for May 2023:

Data monitoring and quality evaluation monitoring reports for previous financial year in the format Arts Council England requests.

There is a set of standards that these mandatory evaluations must adhere to:During the period 1 April 2022 – 31 March 2023, all CPPs must carry out at least four evaluations. The four evaluations should be representative of your programme across the year and can include:

• Events: e.g., performances, festivals, exhibitions
• Creative workshops and participatory experiences: e.g., craft workshops, singing and or music groups
• Decision making activities: e.g., community commissioning panels
• Co-created work: e.g., a large-scale event which has been co-created by the community

As a minimum each evaluation must consist of:

• One pre-activity Impact & Insight survey from a relevant member of staff outlining the creative intentions for the activity (i.e self prior survey– we strongly recommend that you encourage more than one member of staff to complete this)
• Post-event Impact & Insight survey completed by a representative sample of the audience (i.e. public survey– this can be audience in the traditional sense or participants)

You may also wish to include:

• One post-event Impact & Insight survey completed relevant peers (i.e. peer post survey)
• One post-event Impact & Insight survey completed by a relevant member of staff (i.e. self post survey)

CPPs that do not have four events per year available to evaluate, should discuss suitable usage with their Relationship Manager

In May 2023, to fulfil your payment condition, you must:

• Submit your anonymised data to Arts Council
• Create an Insights Report for each of your four evaluations
• Email your four Insights Reports to your Relationship Manager

Please see the Creating an Insights Report guidance on submitting data to Arts Council and producing an Insights Report.

For reference, please see our example of a CPP Insights Report.

 

3. CPP activity and evaluation template choice

 

A mandatory evaluation should involve an activity that can be evaluated using one of the following evaluation templates which are available to you in the Toolkit:

• Core CPP Experience Dimensions for Audiences and/or participants
• Core CPP Experience Dimensions for Core CPP Experience Dimensions for co-creators and/or community decision makers

A comprehensive description of these templates is given below.

We encourage CPPs to evaluate a range of their work. CPPs will find the Toolkit most valuable if they use it to evaluate events where they can learn something interesting about the experiences of their audiences or participants, or where they have a particular hypothesis about programming or marketing that they wish to test. CPPs can discuss their event choices with their Relationship Manager if that would be helpful.

CPPs and NPOs can evaluate the same event, if it was a joint commission; however they must ensure that the shared evaluation meets the requirements and uses the mandatory metrics for both NPO and CPP evaluations. They must submit a separate report on the event to their Relationship Manager; this will count towards the quota of mandatory evaluations for both organisations.

Beyond the four mandatory evaluations, all CPPs are welcome to undertake as many additional evaluations as they like.

 

Core CPP Experience dimensions for audiences and/or participants template

This template contains the following metrics which are mandatory when evaluating things like:

• Events: e.g., performances, festivals, exhibitions
• Creative Workshops and participatory experiences e.g., craft workshops, singing and or music groups

Mandatory metrics:

• Distinctiveness: It was different from things I’ve experienced before
• Relevance: It had something to say about the world in which we live
• Rigour: It was well thought through and put together
• Local Impact: It’s important that it is happening here
• Enthusiasm: I would come to something like this again
• Cultural Contribution: It provides an important addition to the cultural life of the area

The questions should be asked of a random sample of a public audience. These mandatory questions could also be used in the context of an artist led creative workshop for participants, if they are relevant. If you are unsure, please check with your Relationship Manager.

Please note these are the core mandatory questions; projects may choose to add any additional metrics that they wish. The first three metrics are also considered ‘core’ for Band 2 and 3 National Portfolio Organisations.

A spreadsheet of the metrics available in the Toolkit can be found in our Dimensions Framework guidance.

 

Core CPP Experience dimensions for co-creators and/or community decision makers template

 

This template contains the following metrics which are mandatory when evaluating things like:

• Decision Making Activities: e.g., community commissioning panels or ‘Go and See’ type events
• Co-created work: e.g., a large-scale event which has been co-created by the community / members of the public

Mandatory metrics:

• Intention: I felt able to shape the intention of the project
• Voice: My ideas were taken seriously
• Experimenting: I felt comfortable trying new things
• Intensity: I felt deeply involved in the process
• Motivation: I felt motivated to do more things in the future

The questions should be asked of members of the panel or participants in a co-created project. Please note these are the core mandatory questions; projects may choose to add any additional metrics they wish. A list of the metrics available in the Toolkit can be found here.

 

4. Guidance on pre-activity survey from a relevant member of staff (for CPPs)

 

This survey should be carried out ahead of the activity, by a relevant staff member – usually the person who organised, developed or facilitated it. The purpose of this survey is to outline the creative intentions for the activity. It will then be possible for these intentions to be compared with the outcomes and experiences of the audience/participants.

The survey respondent should answer the questions in a way that they would expect the audience/participants to respond, based on their intentions for the activity. For example:

If the activity’s intention is to allow co-creators a strong voice in the creation of a project, the self-respondent should award the metric ‘Voice: My ideas were taken seriously’ a high score.

 

5. Sampling guidance

 

Counting What Counts has produced Sample Size guidance specifically on appropriate sample sizes for different types of event, including participatory events which typically involve smaller groups of people. As a general rule of thumb, CPPs should aim for 70-100 public responses per event, where possible to ensure maximum insight is gained.

 

6. Appendix A – Respondent groups

 

• Self prior – An Impact & Insight survey from a relevant member of staff outlining the creative intentions for the work (ideally NPOs should engage at least three members of staff for each evaluation).
• Self post – An Impact & Insight survey completed by the same staff members that completed the Prior survey.
• Peer post – An Impact & Insight survey completed by relevant peers (ideally NPOs should engage at least three peers for each evaluation).
• Public – An Impact & Insight survey completed by a representative sample of the audience.

Please know that it is possible to conduct the Self post and Peer post surveys within one survey.