Now that we are 18 months into the Impact & Insight Toolkit 23+ project (Toolkit), we thought it would be a good time to gather some information about participation in the Toolkit project to share with you.

We hope that you’ll find it useful to see both who is using the Toolkit and how they’re using it.

Having this information may shape how you wish to further your use of the Toolkit and gather additional insight.

 

Some context to bear in mind

This blogpost will focus on the current Toolkit project (2023+), but, before we look at the figures, there are a few things we would like you to bear in mind:

1. During the 2019-2023 project, band 2 and 3 National Portfolio Organisations (NPOs)  (those that receive most funding) were mandated to participate in the project and complete 4 evaluations per year. There were 256 of these organisations.
2. During the 2019-2023 project, NPOs that were mandated to use the Toolkit were also mandated to use specific dimensions in their surveys (Captivation, Concept, Rigour, Distinctiveness, Challenge [now called Thought-Provoking], Relevance [wording has since been updated and so this version is now archived]).
3. Creative People and Places (CPPs) have been mandated to use the Toolkit in both projects and have two sets of dimensions they can choose from (set one: Distinctiveness, Relevance, Rigour, Local Impact, Enthusiasm, Cultural Contribution; set two: Intention, Voice, Experimenting, Intensity, Motivation).
4. In both projects, organisations outside of those mandated to use the Toolkit have been encouraged to register and to participate.
5. The data informing this blogpost was collected in August 2024. Therefore, there will likely be some developments, depending on when you’re reading this.

It might be that you are already familiar with the above points, but we wanted to remind you before we proceed…

 

The quantity of organisations registered to the Toolkit project

We have some approximate figures for users of the current Impact & Insight Toolkit…

714 organisations have registered to the Toolkit 23+, since its commencement in April 2023.

These organisations are made up of:

• National Portfolio Organisations (NPOs)
• Creative People and Places (CPPs)
• National Lottery Project Grant recipients (NLPGs)
• Museum Development Network museums (MDNs)
• Investment Principle Support Organisations (IPSOs)

… and many more!

 

NPOs are, by far, the largest group of users by funding type. We’re very encouraged that so many of them have chosen to participate in this project, after mandated use of the Toolkit for NPOs ceased in April 2023.

The Toolkit strives to support Arts Council England’s Ambition and Quality Investment Principle (IP). The overall registered number of organisations to the Toolkit indicates that this IP is deemed important and worth investing time in for a large array of organisations, with all different funding types and amounts. The level of interest in the Toolkit is a promising sign for Arts Council England (ACE), as it indicates that their funded organisations are keen to engage with the Ambition and Quality IP.

 

 

Artforms, Evaluations and Public Surveys

 

Let’s move on to the quantity of data that has been collected throughout this project so far…

Number of organisations registered	Evaluations created	Public surveys (with 4 or more responses)	Public responses	Total responses	Evaluations per organisation (mean)	Public surveys (mean)
714	3,490	3,088	240,293	245,889	4.89	4.21

 

And consider a breakdown of participants in the Toolkit project since April 2023, by Arts Council England recorded artform…

Artform	Number of organisations registered	Number of evaluations (total created)	Number of public surveys (with 4 or more responses)	Number of public responses	Total number of responses	Evaluations per organisation (mean average)	Public surveys per organisation (mean average)
Combined Arts	124	517	380	18,633	19,146	4.17	3.06
Dance	53	260	211	14,562	15,054	4.91	3.58
Libraries	5	30	34	949	974	6	6.8
Literature	23	74	103	5,755	5,866	3.22	4.48
Museums	60	407	336	45,373	45,769	6.78	5.6
Music	90	440	566	38,357	39,296	4.89	6.29
Theatre	141	695	605	71,911	73,344	4.93	4.29
Visual Arts	79	407	257	18,392	19,404	5.15	3.25
Not Artform Specific	12	71	52	1,761	1,770	5.92	4.33
Uncategorised	127	588	465	24,604	25,270	4.63	3.66

 

Organisations that ACE has recorded as Theatre or Combined Arts use the Toolkit the most; this is in terms of the number of organisations signed up, and by evaluations created. Although Combined Arts organisations have the second highest number of registrations at 124, they have far fewer public responses than the 90 Music organisations and 60 Museums organisations. This might be because combined arts organisations are choosing to use the Toolkit to evaluate lots of smaller works, in comparison to music and museums organisations choosing to evaluate fewer but larger works.

We also have many ‘uncategorised’ users – these are made up of organisations in receipt of other types of ACE funding, or those that are eligible for funding but do not receive it.

The average number of evaluations and surveys created across all organisations is above 4 per organisation. In the previous Toolkit project, 4 evaluations (and therefore public surveys within those evaluations) were the mandatory number required for organisations such as NPOs and CPPs. So, it is interesting to see that the numbers in the non-mandatory project have remained similar.

 

It’s important to recognise when considering numbers of evaluations and public surveys that:

• Surveys with 3 responses or fewer are not included in the figures.
• Some organisations use one evaluation for multiple public surveys.
• The organisation counts include all those who have signed up, but some users are more active than others and could be creating lots of evaluations whereas others fairly few.

 

Dimension usage

In the majority of instances, there is no requirement for a specific set of dimensions to be used by those participating in the Toolkit project. In fact, usually, there is no requirement for dimensions to be used whatsoever. However, project participants are encouraged to select dimensions which align most with their mission; this supports Arts Council England’s Ambition and Quality IP. With that in mind, we felt it would be interesting for you to know which are the 10 most commonly-used dimensions. Having this information might be particularly useful if you are interested in benchmarking and would like to have a larger pool of organisations to benchmark against.

Domain	Outcome area	Dimension label	Dimension statement	Count of use in evaluations, since 01/04/2023
Qualities	Perception	Rigour	It was well thought through and put together	1,109
Qualities	Experience	Distinctiveness	It was different from things I’ve experienced before	935
Qualities	Experience	Captivation	It was absorbing and held my attention	794
Qualities	Experience	Enthusiasm	I would come to something like this again	745
Qualities	Local Impact	Cultural Contribution	It provides an important addition to the cultural life of the area	620
Qualities	Content	Thought Provoking	It was thought provoking	568
Qualities	Local Impact	Local Impact	It’s important that it’s happening here	558
Qualities	Content	Concept	It was an interesting idea	526
Qualities	Content	Relevance	It had something to say about modern society	436
Qualities	Experience	Enjoyment	I had a good time	400

There are a couple of highlights that we will mention, reflected in the data shown in the table above:

• All dimensions in the top-ten most commonly-used list are from the Qualities domain. This is a really ‘stand out’ takeaway, when there are 5 other domains: Economic Outcomes, Environmental Outcomes, Community Outcomes, Social Outcomes, Cultural Outcomes. Whilst we wouldn’t want to assume why this might be, it is certainly something worth noting.
• All of the 6 dimensions that were mandated for NPOs in the previous project (2019-23) feature in this ‘top ten’ list. This might be because many of the organisations participating in the previous project found that the mandated dimensions were capturing insightful feedback, or maybe it’s because those organisations value continuity in their data collection more than the freedom offered via the ‘choose your own’ approach, or maybe there’s a whole other reason!

 

It might also be helpful to see a visual representation of the outcome areas that people have chosen…

 

This implies that organisations are most interested in receiving feedback about an individual’s experience of a work, rather than their wider reflections. It will be interesting to see how this evolves over the course of the project, alongside the domains from which dimensions have been chosen from.

 

Comparisons with the Toolkit Project 2019-23

You cannot directly compare the two Toolkit projects. Before it started, we even had discussions about using a different name for the new project to distinguish it from the Toolkit 19-23! However, for continuity purposes, we felt it would be more recognisable for those in the sector if we continued with the same name.

That said, we know that some of you are interested and we are therefore presenting you with a select few comparisons…

Point of Comparison	2019-2023 Project	2023+ Project
NPO sign ups	374	529
Total number of sign ups	568	714
Public responses in one year (‘22-23 vs ‘23-24)	223,045	240,293
Created evaluations in one year (‘22-23 vs ‘23-24)	2,492	2,823

Furthermore, 359 of the organisations that were registered to the 2019-23 project have chosen to continue to the 2023+ project. We are very happy that so many of our users from the original project decided to continue using the new Toolkit into the current funding cycle!

 

Closing words

Overall, as you can see, the engagement with Impact & Insight Toolkit project has increased and we are pleased to see that so many organisations across the country and artforms are placing quality evaluation in a prominent position within their organisation. Whilst it may feel that each organisation is operating independently, considering their quality-focussed data in silo, the stories we will be able to tell around the importance of arts and cultural experiences are ever-increasing. Arts Council England’s Ambition and Quality Investment Principle highlights that it is necessary for an organisation in receipt of public funds to define what quality means to them and to evidence the extent to which they’re achieving that. The Impact & Insight Toolkit offers one method of doing this and we are keen to continue to improve our offer, delivering maximum opportunity for insights and data driven decision making.

 

 

Image credit: Ryan Quintal on Unsplash