Having recently been chatting more about the ‘new’ Impact & Insight Toolkit, some great questions have been presented to us.  Whilst we have responded directly, we would like to share these questions and our answers with all interested parties.  These questions are about both the methodology and the online platform it utilises, Culture Counts.

 

This is admittedly a verrrrrrry long (!) blogpost, but we think it’s important to acknowledge the questions asked and give you as much information as possible as we transition from the 2019-23 Toolkit project to the 2023-26 Toolkit project.

 

Before we go into specific questions, let’s address two important overarching questions:

 

What is the Impact & Insight Toolkit (Toolkit)?

The Toolkit is a three-year long evaluation project, funded by Arts Council England.  It has been developed to support ACE-funded organisations demonstrate their commitment to Arts Council’s Ambition and Quality Investment Principle (IP).  This means that there is no financial cost to an ACE-funded organisation wishing to use the Toolkit.  The Toolkit project enables organisations to use data-driven methods to evaluate their work, and measure progress against their ambitions, by gathering feedback from the people who experience their work.

 

How do I register our organisation to participate in the Toolkit project?

One person from your organisation needs to complete an online Expression of Interest.  We will then be in touch accordingly.

 

Now moving onto the more detailed questions…

 

For ease, we are dividing this blogpost into sections, under the following headings:

Mandated Use and Arts Council Requirements

Culture Counts and Account Management

PwC and Illuminate; The Audience Agency and Audience Finder

Accessibility, Touring and Other Complexities

 

Photo by vnwayne fan on Unsplash

 

Mandated Use and Arts Council Requirements

 

Is it mandatory to sign up to the Toolkit and use Culture Counts?

For the majority of ACE-funded organisations (e.g., NPOs), it is not mandatory to sign up to the Toolkit and use Culture Counts.

Although registering to the Toolkit is not mandated, it is a requirement of you to develop an evaluation framework and to evidence your commitment to fulfilling the Ambition and Quality Investment Principle.  The Toolkit has been developed to directly support ACE-funded organisations in evidencing their commitment to the Ambition and Quality IP and, therefore, the Toolkit will be a very useful resource in fulfilling the IPs for many of you.

Please know that this is not the case for those in the Creative People and Places portfolio, for whom this is a funding requirement.  If you are a CPP, please look at the appropriate funding requirements guidance to ensure you are meeting these.

 

If we choose to not participate in the Toolkit project, how do we go about developing an evaluation framework?

Whilst we are not familiar with other approaches or tools that might support you in developing you framework or evidencing ambition and quality, our blogpost on evaluation frameworks would be a good place to start.

 

If use isn’t mandated, does data collected get sent to Arts Council England in some way?

In short, the answer to this question is ‘no’.  The new Toolkit is more about engaging your board and Senior Management Team in understanding evaluation and the data that’s collected, than reporting to ACE.  If you use Culture Counts, you will be provided with Ambition Progress Reports and will also be able to generate Insights Reports.  These reports should fuel board meetings and provide a conversation kick-starter.  Your Relationship Manager will make a note of these discussions either through their attendance at your Board meetings and/or from your Board meeting minutes. Whilst you do not need to send reports or data to ACE, they will expect to see that you are discussing the Investment Principles and utilising data collected.

 

Who is the owner of any data we collect?

You are!  Culture Counts is a data processor.  The user of Culture Counts, you, owns the data.  It is not owned by Arts Council England.

 

If we register to the Toolkit project, are we committing to use Culture Counts?

No.  Registering to join the Toolkit project will give you access to evaluation guidance and resources, as well as the Culture Counts platform.  If you choose to not use Culture Counts but do want to engage with other aspects of the Toolkit project, that is fine.

 

Photo by Tim Arterbury on Unsplash

 

Culture Counts and Account Management

 

If we have participated in the ‘previous’ Toolkit project (2019-2023), and have used Culture Counts, do we still need to register to use the new Toolkit?

Yes.  The ‘new’ Toolkit project is based on a ‘new’ contract and therefore you will need to sign up.

 

If we have participated in the ‘previous’ Toolkit project (2019-2023), and have used Culture Counts, will my previously collected data still be accessible?  What about if I am running an evaluation over the transition period?

Any data collected as part of the ‘19-23 project can still be accessed via your Culture Counts account.  No data will be ‘lost’, and your account will still be fully functioning.  This means that any evaluations that are currently running will continue to collect data.  Please know that if you do not transition to the new Toolkit project, access to Culture Counts will be discontinued from May 31st 2023.

 

In addition to utilising inbuilt dimensions and templates, are we able to add in our own questions to surveys on Culture Counts?

Yes, absolutely.  One of the great things about Culture Counts is that it is a flexible tool.  Therefore, you can create evaluations based solely on the inbuilt questions; ‘write your own’ questions; or use a combination of both.  It really is up to you.

 

If our organisation registers, do we just get one login to Culture Counts?

As part of the new Toolkit project, there is a ‘multiuser’ feature, which means that each organisation can have up to three logins.  You can choose to collaborate on evaluations or to work separately.  Please know that only one person from an organisation needs to register.

 

Are we able to access results and reports in an automated way through Culture Counts, or do we need to wait for reporting from CWC?

CWC is going to be providing users with reports on, most likely, a quarterly basis.  We do not know exactly what these will look like and are keen to engage with NPOs to ensure that these reports are genuinely insightful and helpful.  However, you can generate automated reports through Culture Counts at any time.  These are called Insights Reports.  You can also extract the raw data any time you like.

 

Can we export data collected in Culture Counts and use it outside of the platform?

Yes, there is a CSV download feature.  This feature extracts all data relating to a specific evaluation.  You can then manipulate the data in a spreadsheet and use it as you see fit.  Some people might choose to use their data to power interactive dashboards, such as Power BI; others might choose to upload data to other platforms.

 

Can we import data into Culture Counts that we have collected outside of the platform?

Yes, there is a CSV import feature.  After setting up the evaluation in Culture Counts, you download a template.  You populate this template with your data collected elsewhere and then import.

 

Once we select and write questions for our evaluations, are we able to change these for different projects or are we wedded to those initially chosen?

You can absolutely change questions if that’s what you choose to do.  We recommend that you have different dimension ‘lists’ for different types of work (e.g., you could have a participatory-works list and a presentational-works list).  You can then select whichever is most appropriate and, of course, add and/or remove questions as you wish.

 

Photo by Joshua Mayo on Unsplash

 

PwC and Illuminate; The Audience Agency and Audience Finder

 

Does the Toolkit replace The Audience Agency’s Audience Finder?

No.  The Toolkit responds to a completely different contract set out by ACE.  PwC has been commissioned to offer a data collection service and to produce audience development focussed insights; this service is called Illuminate.

 

What is the difference between Illuminate and the Toolkit?

In a nutshell, Illuminate is about understanding who your work reaches; the Toolkit is about understanding how your works contribute to you achieving your overall ambition.

The objective of both the Toolkit and Illuminate is in providing target data insight to arts & cultural organisations.  Whilst both projects involve some similar data collection tools, the purpose behind the data insight is completely different.  This means that the way the data is analysed and presented, and the questions that it can help organisations to answer, is unique for the two projects.

 

Photo by Vardan Papikyan on Unsplash

 

Accessibility, Touring and Other Complexities

 

Are we able to use paper surveys?

The Culture Counts platform is an online, digital, evaluation tool.  Whilst it is possible to collect survey responses via paper surveys, it is not recommended as transposing and importing the data to Culture Counts can be resource-heavy.

 

Are we able to change the wording of inbuilt questions?

If you want to use the inbuilt questions to Culture Counts, the wording is fixed.  This is to enable research and analysis further down the line; these will also help fuel your reports.  You can add in your own custom questions if the inbuilt ones are unsuitable.

 

How do we use Culture Counts if our audiences have disabilities and/or are neurodiverse?

Firstly, it is important to remember that it is not mandatory for most ACE funded organisations to use this tool. There may be others available that are more appropriate for your organisation and audience.  If you do use this tool, you can write your own questions.  The platform supports the use of emojis in its surveys which we have found to be quite popular with some audiences.

 

Can we do a mixed-method approach to support our audiences?

Yes, of course!  It might be that you want to collect online, digital responses from a certain subset of your audience, whereas a more interactive and/or physical method of collecting feedback might be more appropriate for others.  You could then transpose the feedback into a method suitable for Culture Counts, meaning that all data is kept in one place.

 

What accessibility needs have been taken into account with Culture Counts?

The surveys generated through Culture Counts are AA accessibility certified.  This means that they are screen reader accessible and that the contrasting used is considered appropriate.  The inbuilt questions are written to a reading age of 12, meaning that most young adults and adults can respond to these questions.  You can write in your own questions if those inbuilt are not appropriate.  The platform also supports the use of emojis, which has proven to be quite popular.

 

Peer review is considered to be a positive thing in evaluation and is encouraged in the Toolkit, but how do we find suitable peers for our work?

You should look in your own professional network and give the Peer Matching Resource a try.

Firstly, a peer reviewer can be anybody who is not involved in the curation or creation of your work AND whose professional opinion you would respect on it. Therefore, a suitable peer reviewer for one work might not be suitable for a different work.  A peer reviewer does not need to work in the same artform as the work that’s being reviewed; a peer reviewer does not need to work in the arts and cultural sector.  What’s important is that you believe that their professional perspective on your work will provide value to your understanding of its effectiveness.  The Peer Matching Resource is an inbuilt resource to Culture Counts, helping you to connect with those across the sector that have expressed an interest in being a peer reviewer.  Using this resource is optional; you could connect with peers through your pre-existing professional networks.

 

We don’t have access to box office data or Internet access – how do we collect responses?

There is an offline data collection feature, built into Culture Counts.  This means that you can digitally collect 1 to 1 interviews in your space that does not have Internet.  When you go somewhere that does have Internet access, you can sync the responses collected to your Culture Counts account.

 

As a touring company, how do we approach data collection at host venues?

If the touring company and host venue are both ACE-funded, this can be helpful as both parties will likely be familiar with the Toolkit and, if they were band 2 or 3 NPOs in previous funding rounds, they will already know how to use Culture Counts and understand how the data is saved and presented.  There are a few tips we have:

• Communication – openly explain what you need and why you need it
• Clarity and understanding – people generally find that having a data sharing agreement in place is useful to ensure that there is clear understanding
• Compromise – when it comes to choosing the dimensions and other questions to include, you will likely need to compromise to ensure that all parties achieve data they need
• Be creative in data collection – even if you can’t access box office data, you could do things like sharing QR codes on event programmes or posters; you may be able to conduct 1 to 1 interviews ‘on the ground’
• Share evaluations and data – all collected data is anonymised, meaning that there is no reason that data cannot be shared

Remember, if both parties have a Culture Counts account, evaluations can be ‘shared’, meaning that they both have access and can use the data accordingly.  This reduces the survey fatigue issue as the audience only receive the one survey and there is no duplication for them.

 

Photo by Daniel Schludi on Unsplash

 

Have you not been able to find the answer to your question? 

Here are some handy links to written guidance and/or reports:

Sample size and the Toolkit

Accessibility of dimensions

Data protection

Timeline of launches and other handy information

Ambition and Quality Investment Principle (ACE authored)

PwC and Illuminate (ACE authored)

 

We look forward to working with you!

 

 

Featured image: Jac Alexandru on Unsplash