Now that we are 18 months into the Impact & Insight Toolkit 23+ project (Toolkit), we thought it would be a good time to gather some information about participation in the Toolkit project to share with you.
We hope that you’ll find it useful to see both who is using the Toolkit and how they’re using it.
Having this information may shape how you wish to further your use of the Toolkit and gather additional insight.
To make this blogpost easier to navigate, please see links to sections within this post:
Some context to bear in mind
This blogpost will focus on the current Toolkit project (2023+), but, before we look at the figures, there are a few things we would like you to bear in mind:
- During the 2019-2023 project, band 2 and 3 National Portfolio Organisations (NPOs) (those that receive most funding) were mandated to participate in the project and complete 4 evaluations per year. There were 256 of these organisations.
- During the 2019-2023 project, NPOs that were mandated to use the Toolkit were also mandated to use specific dimensions in their surveys (Captivation, Concept, Rigour, Distinctiveness, Challenge [now called Thought-Provoking], Relevance [wording has since been updated and so this version is now archived]).
- Creative People and Places (CPPs) have been mandated to use the Toolkit in both projects and have two sets of dimensions they can choose from (set one: Distinctiveness, Relevance, Rigour, Local Impact, Enthusiasm, Cultural Contribution; set two: Intention, Voice, Experimenting, Intensity, Motivation).
- In both projects, organisations outside of those mandated to use the Toolkit have been encouraged to register and to participate.
- The data informing this blogpost was collected in August 2024. Therefore, there will likely be some developments, depending on when you’re reading this.
It might be that you are already familiar with the above points, but we wanted to remind you before we proceed…
The quantity of organisations registered to the Toolkit project
We have some approximate figures for users of the current Impact & Insight Toolkit…
714 organisations have registered to the Toolkit 23+, since its commencement in April 2023.
These organisations are made up of:
- National Portfolio Organisations (NPOs)
- Creative People and Places (CPPs)
- National Lottery Project Grant recipients (NLPGs)
- Museum Development Network museums (MDNs)
- Investment Principle Support Organisations (IPSOs)
… and many more!
NPOs are, by far, the largest group of users by funding type. We’re very encouraged that so many of them have chosen to participate in this project, after mandated use of the Toolkit for NPOs ceased in April 2023.
The Toolkit strives to support Arts Council England’s Ambition and Quality Investment Principle (IP). The overall registered number of organisations to the Toolkit indicates that this IP is deemed important and worth investing time in for a large array of organisations, with all different funding types and amounts. The level of interest in the Toolkit is a promising sign for Arts Council England (ACE), as it indicates that their funded organisations are keen to engage with the Ambition and Quality IP.
Let’s move on to the quantity of data that has been collected throughout this project so far…
| Number of organisations registered | Evaluations created | Public surveys (with 4 or more responses) | Public responses | Total responses | Evaluations per organisation (mean) | Public surveys (mean) |
| 714 | 3,490 | 3,088 | 240,293 | 245,889 | 4.89 | 4.21 |
And consider a breakdown of participants in the Toolkit project since April 2023, by Arts Council England recorded artform…
| Artform | Number of organisations registered | Number of evaluations (total created) | Number of public surveys (with 4 or more responses) | Number of public responses | Total number of responses | Evaluations per organisation (mean average) | Public surveys per organisation (mean average) |
| Combined Arts | 124 | 517 | 380 | 18,633 | 19,146 | 4.17 | 3.06 |
| Dance | 53 | 260 | 211 | 14,562 | 15,054 | 4.91 | 3.58 |
| Libraries | 5 | 30 | 34 | 949 | 974 | 6 | 6.8 |
| Literature | 23 | 74 | 103 | 5,755 | 5,866 | 3.22 | 4.48 |
| Museums | 60 | 407 | 336 | 45,373 | 45,769 | 6.78 | 5.6 |
| Music | 90 | 440 | 566 | 38,357 | 39,296 | 4.89 | 6.29 |
| Theatre | 141 | 695 | 605 | 71,911 | 73,344 | 4.93 | 4.29 |
| Visual Arts | 79 | 407 | 257 | 18,392 | 19,404 | 5.15 | 3.25 |
| Not Artform Specific | 12 | 71 | 52 | 1,761 | 1,770 | 5.92 | 4.33 |
| Uncategorised | 127 | 588 | 465 | 24,604 | 25,270 | 4.63 | 3.66 |
Organisations that ACE has recorded as Theatre or Combined Arts use the Toolkit the most; this is in terms of the number of organisations signed up, and by evaluations created. Although Combined Arts organisations have the second highest number of registrations at 124, they have far fewer public responses than the 90 Music organisations and 60 Museums organisations. This might be because combined arts organisations are choosing to use the Toolkit to evaluate lots of smaller works, in comparison to music and museums organisations choosing to evaluate fewer but larger works.
We also have many ‘uncategorised’ users – these are made up of organisations in receipt of other types of ACE funding, or those that are eligible for funding but do not receive it.
The average number of evaluations and surveys created across all organisations is above 4 per organisation. In the previous Toolkit project, 4 evaluations (and therefore public surveys within those evaluations) were the mandatory number required for organisations such as NPOs and CPPs. So, it is interesting to see that the numbers in the non-mandatory project have remained similar.
It’s important to recognise when considering numbers of evaluations and public surveys that:
- Surveys with 3 responses or fewer are not included in the figures.
- Some organisations use one evaluation for multiple public surveys.
- The organisation counts include all those who have signed up, but some users are more active than others and could be creating lots of evaluations whereas others fairly few.
Dimension usage
In the majority of instances, there is no requirement for a specific set of dimensions to be used by those participating in the Toolkit project. In fact, usually, there is no requirement for dimensions to be used whatsoever. However, project participants are encouraged to select dimensions which align most with their mission; this supports Arts Council England’s Ambition and Quality IP. With that in mind, we felt it would be interesting for you to know which are the 10 most commonly-used dimensions. Having this information might be particularly useful if you are interested in benchmarking and would like to have a larger pool of organisations to benchmark against.
| Domain | Outcome area | Dimension label | Dimension statement | Count of use in evaluations, since 01/04/2023 |
| Qualities | Perception | Rigour | It was well thought through and put together | 1,109 |
| Qualities | Experience | Distinctiveness | It was different from things I’ve experienced before | 935 |
| Qualities | Experience | Captivation | It was absorbing and held my attention | 794 |
| Qualities | Experience | Enthusiasm | I would come to something like this again | 745 |
| Qualities | Local Impact | Cultural Contribution | It provides an important addition to the cultural life of the area | 620 |
| Qualities | Content | Thought Provoking | It was thought provoking | 568 |
| Qualities | Local Impact | Local Impact | It’s important that it’s happening here | 558 |
| Qualities | Content | Concept | It was an interesting idea | 526 |
| Qualities | Content | Relevance | It had something to say about modern society | 436 |
| Qualities | Experience | Enjoyment | I had a good time | 400 |
There are a couple of highlights that we will mention, reflected in the data shown in the table above:
- All dimensions in the top-ten most commonly-used list are from the Qualities domain. This is a really ‘stand out’ takeaway, when there are 5 other domains: Economic Outcomes, Environmental Outcomes, Community Outcomes, Social Outcomes, Cultural Outcomes. Whilst we wouldn’t want to assume why this might be, it is certainly something worth noting.
- All of the 6 dimensions that were mandated for NPOs in the previous project (2019-23) feature in this ‘top ten’ list. This might be because many of the organisations participating in the previous project found that the mandated dimensions were capturing insightful feedback, or maybe it’s because those organisations value continuity in their data collection more than the freedom offered via the ‘choose your own’ approach, or maybe there’s a whole other reason!
It might also be helpful to see a visual representation of the outcome areas that people have chosen…
This implies that organisations are most interested in receiving feedback about an individual’s experience of a work, rather than their wider reflections. It will be interesting to see how this evolves over the course of the project, alongside the domains from which dimensions have been chosen from.
Comparisons with the Toolkit Project 2019-23
You cannot directly compare the two Toolkit projects – see the ‘Some context to bear in mind’ section for more information. Before it started, we even had discussions about using a different name for the new project to distinguish it from the Toolkit 19-23! However, for continuity purposes, we felt it would be more recognisable for those in the sector if we continued with the same name.
That said, we know that some of you are interested and we are therefore presenting you with a select few comparisons…
| Point of Comparison | 2019-2023 Project | 2023+ Project |
| NPO sign ups | 374 | 529 |
| Total number of sign ups | 568 | 714 |
| Public responses in one year (‘22-23 vs ‘23-24) | 223,045 | 240,293 |
| Created evaluations in one year (‘22-23 vs ‘23-24) | 2,492 | 2,823 |
Furthermore, 359 of the organisations that were registered to the 2019-23 project have chosen to continue to the 2023+ project. We are very happy that so many of our users from the original project decided to continue using the new Toolkit into the current funding cycle!
Closing words
Overall, as you can see, the engagement with Impact & Insight Toolkit project has increased and we are pleased to see that so many organisations across the country and artforms are placing quality evaluation in a prominent position within their organisation. Whilst it may feel that each organisation is operating independently, considering their quality-focussed data in silo, the stories we will be able to tell around the importance of arts and cultural experiences are ever-increasing. Arts Council England’s Ambition and Quality Investment Principle highlights that it is necessary for an organisation in receipt of public funds to define what quality means to them and to evidence the extent to which they’re achieving that. The Impact & Insight Toolkit offers one method of doing this and we are keen to continue to improve our offer, delivering maximum opportunity for insights and data driven decision making.
Image credit: Ryan Quintal on Unsplash