We’ve learned a lot and we’re making changes! Read on to understand the new things that are coming your way from April 2023…

 

In July 2022, CWC decided it was high time we conducted our own survey and asked Impact & Insight Toolkit (Toolkit) users for feedback on their Toolkit experience.  We created a survey in the Culture Counts platform and invited all registered users to take part and let us know how we were doing and where we could make improvements.

We’d like to thank the 144 Toolkit users that responded to the survey – 69% of these being a Band 2 or 3 NPO Toolkit user and 31% were a mix of Band 1 NPOs, CPPs, SSOs, and other recipients of ACE funding. We really appreciate the time you took to complete the survey, and an extra special thanks to those that were happy to be contacted for a more detailed discussion. We learned a lot from you and we have made, or are in the process of making, some changes.

 

So, what did we learn?

 

We were delighted that the majority of users felt the level of support from CWC met their needs. Our Engagement and Support team love speaking to users of the Toolkit, whether it’s to guide you through the process or help you problem solve. We’re glad to hear that you feel we’re doing a good job!

 

We were also pleased to see the positive response to our live online platform demonstrations.  Mind you, there’s always room for improvement so, if you attend a session in the future, do feel free to drop us an email afterwards to provide constructive feedback.

Similarly, we were pleased to see that:

• 80% of survey respondents were satisfied or very satisfied with our support materials, with 18% neutral
• 78% were satisfied or very satisfied with the web content, with 18% neutral
• 54% were satisfied or very satisfied with the data collection platform, Culture Counts, with 35% neutral
• 65% were satisfied or very satisfied with the research resources with a surprising 58% of respondents having used the Insights Dashboard and 41% having used the Benchmarking dashboard

 

We know that exploring the Insights Dashboard and/or Benchmarking Dashboard is an additional step in the Toolkit evaluation process, but we really feel that they are worth the time and are pleased to see that you think so too.  These two dashboards help you to contextualise the results you’ve received by looking at the bigger picture of submitted data.

 

Room for improvement

 

Finally, we asked respondents to let us know if there were things we could improve or change or if there was anything missing in our guidance and support. Below are examples of some of the key topics that came up more than once:

• Peer review
• Touring organisations
• Culture Counts account management and flexibility
• Paper surveying
• Offline survey collection
• A broader range of ‘dimension’ questions

It is important to know that resolving some of these issues are beyond our capabilities in this funding round as service providers, but we really value you telling us about these challenges as it will help to inform any future guidance and also help to frame conversations with partners.  However, that is certainly not the case for all the issues raised, and these have directly informed platform development and evaluation approach for the coming funding period…

What’s changing?

 

As many of you will know by now, CWC has been procured by ACE to offer all NPOs new tools to help them evaluate their work and report against ACE’s Ambition & Quality Investment Principle – an activity that is now an important part of NPOs’ 2023-2026 funding agreements.

Since the new contract was confirmed, we’ve been busy building on your feedback to make improvements in the Culture Counts platform, to the website and the project approach. From April you can expect to see a number of improvements, including the following:

Dimensions

We’ll be adding more dimensions to the platform. You’ll find dimensions for a range of outcomes including Cultural, Economic, Place and Social outcomes. In addition, the surveys will no longer display the dimension outcome and you will now only see the statement. For example, the Toolkit used to display the survey question as Outcome + Statement as follows;

‘Distinctiveness – It was different from things I’ve experienced before’

And now, only the statement will be displayed:

‘It was different from things I’ve experienced before’

This development is to address the question of accessibility and the use of plain English.  Having the outcome word (e.g., Distinctiveness) can feel very ‘artspeak’ and may put potential survey respondents off from answering.

 

Flexibility in choosing dimensions

There are no longer any mandatory questions for NPO users.  Therefore, if you are from an NPO, you will be able to select the dimensions that you feel are most relevant to your organisation, its mission statement and ambitions for your work.

You will be encouraged to select your chosen dimensions based on your organisation’s ambitions overall and for the work you’re evaluating.  This means that the dimensions chosen will speak directly to your organisation.

If you are from a funded organisation that does have a mandatory Toolkit question set as part of your funding requirements, you are still able to add your own selected dimensions into the survey, ensuring that you are fulfilling both ACE’s requirements and tailoring the evaluation to meet your organisational needs.

 

Multiple users

Some Toolkit users mentioned that it was difficult to manage evaluation sharing across different accounts. Currently, Toolkit users have to know who they are sharing their evaluations with and, if a new person in their organisation signs up to the Toolkit, they have to go through their evaluations and re-share everything with them.  With the new Toolkit, evaluation sharing will be made simpler and easier, enabling more people in your team to see and collaborate on evaluations.

The evaluation permissions will be relabelled to make sharing much more intuitive. There will be three types:

Full Access: can edit and share with others

Can edit: can edit but not share with others

Can view: can view but cannot edit or share with others

You will be able to manage this through a new user management portal. The benefit of this portal also means it will be easier to manage and administer user accounts across your organisation. You will be able to use this portal to add new users to your account, or revokeaccess to those that may have left your organisation.

 

New templates

If you’re not sure how to get started or need some inspiration, you can select from some ready-to-use evaluation templates in the platform.  The contents of these templates have been informed through research and our experience of working with arts and cultural organisations.  You can edit these to ensure it is appropriate for your work and organisation.

This development has occurred to make it quicker and easier for you to create all-encompassing evaluations that capture a variety of insight, whilst still enabling customisation.

 

Offline survey feature

This feature will allow Toolkit users to digitally collect survey responses on a smart device or tablet when not connected to WiFi.  Responses will be saved to your device; you can then upload those responses to your Culture Counts account once an Internet connection is restored.

This means that when surveying in rural areas, outdoors or other Internet-less locations, you can collect responses immediately and do not have to concern yourself with the resource-heavy process of manually entering paper survey responses.

 

Reporting data filters (coming soon)

The Culture Counts platform will feature new filters and charts, allowing users to explore data collected and pivot by different factors, including survey/activity comparison, identity, or overall experience. These will give users more flexibility in understanding the insights collected through the platform.

We know that a lot of value comes from looking beyond the top-level dimension results, and the ability to pivot dimension results based on other factors will help you to learn how the impact of your work differs across your audience/participants.

 

Ongoing development

We are not stopping here!  CWC is keen to continue making developments to our service, so that our users can get optimal value out of the Impact & Insight Toolkit.  We will need your support and input for this, so do keep an eye out for opportunities to work with us.

 

And a few words of thanks…

Once again, thank you to those that responded to the survey and to those that have provided feedback through the development strands implemented in phase one of the Toolkit: Strategic Development Strand; Artform and Museum Metrics Strand; Participatory Metrics Strand.  Your knowledge and expertise have been crucial in shaping the Toolkit project since its launch in 2019 and we hope you’ll join us and continue to collaborate in 2023.

 

 

Author: Gurvinder Bans, Project Director

Featured image credit: Ross Findon on Unsplash