With many thanks to Z-Arts for sharing their insights on using the Impact & Insight Toolkit (Toolkit) at their festival, Big Imaginations: theatre, arts and music for children and their grown ups.

Z-arts is a new Band 2 National Portfolio Organisation (NPO). A major part of our funding agreement is to manage and support the Big Imaginations network of 22 venues and programmers across the North West who programme brilliant theatre for children and families. We support and deliver a children’s theatre programme all year round, often in areas where there is little existing provision and in venues that range from small and technically-challenged village halls to large auditoriums.

As well as a year-round programme, we programme a bi-annual festival, probably the only region-wide festival of its kind in the UK. With ten different shows, performed a total of 111 times over 35 venues and reaching over 3,300 audience members, this was the largest project for Z-arts this year, so it made sense to try and run an Impact & Insight Toolkit evaluation for the entire festival. In other ways, this presented us with challenges, so I’m hoping that sharing some of those challenges and solutions will help spread the learning.

Initial challenges:

• How to ensure that the data was relevant and shareable for all 22 venues, most of whom are not NPOs and certainly not Band 2 or 3. We had to consider the various requirements each partner had to report to other funders, as well as the mandatory questions required by Arts Council England (ACE). Many of these mandatory questions didn’t feel relevant to us, but other inbuilt dimension questions like ‘We explored and learned together as a family’ felt more valuable and we used them across all festival surveys.

• Whilst the simplicity of the survey is very appealing, both Z-arts and many of our partner venues found it odd that we weren’t encouraged to use the Toolkit to gather data for mailing lists or demographic information. Venues were resistant to making audiences complete more than one evaluation form and so we did add some demographic questions to the survey, but this did feel a bit clumsy. We have yet to find the right compromise.

• Some of the partners (rural venues) didn’t even have internet access, so Counting What Counts (CWC) was able to make a paper version of the survey for the venues to complete, and manually upload later. Whilst we did manage to create a survey that worked as a compromise for most partners, it still wasn’t ideal for everyone. One partner, who is a Creative People and Places (CPP) scheme, did require an additional question which we managed to add at the end without having a negative impact on the overall evaluation.

• We found some of the language used in the core dimensions obscure for family audiences or people with English as a second language (especially the dimension titles), and as our primary audiences were children and families, we wanted to make a child-friendly version of the survey. CWC recommended the report by Shared Intelligence which tested the accessibility and legibility of the metric statements with children and young people as well as older people and people with learning disabilities. Upon further inspection, this research had been focused on young people rather than children. I know CWC is planning to bring some of the new suggestions into a future model, but obviously these things take time. As a solution, we tried using the ‘interview’ model with family audiences, which worked well with children. Anyone who wasn’t sure about the heading word, did understand when we read the description below.

• In terms of evaluation and sharing the data meaningfully across the network, we created a different link for each venue, so that we were able to create distinct reports for each venue, as well as one that combines all of them. That way we could send each venue their own report, either linking to their own dashboard if they had a Toolkit account, or via a spreadsheet if not.

The main challenge was that we were learning as we went along, but each mistake we made was multiplied by 22! Luckily the team at CWC was very understanding and supportive, but I’m sure they would have rather helped solve one error rather than a multiple of 22.

The biggest lesson we learned is that we are constantly under-estimating our value to others as we tend to score ourselves lower that our peers and the public. The one instance that this was not the case was our local impact (important it’s happening in this area) when our peers ranked us lower than ourselves and the public – this may however be down to our partners being in areas of low engagement, and reflecting the perception of what matters to audiences (who scored us higher than we scored ourselves).

A lot of things were confirmed through the survey – for example that the majority (81%) of our bookers who responded are female, and that the largest age group of bookers within the 35 – 44 age range (43%), which makes sense given that our work is aimed at children. We were surprised to find how highly others rated us compared to our own opinions of ourselves – something which does however fit in with a lot of what Z-arts does. We tend not to realise that what we often do, for example in terms of access and adaptability is not always the industry norm. We were pleased to be rated so highly in terms of captivation (87 by the public, 91 by peers as opposed to 72 by ourselves). This was one of the biggest gaps in perception for us, and we were pleased to see that our focus on quality programming has achieved its desired effect.

This will inform our marketing for future festivals– it was difficult to get a gauge of individual results because of the survey covering the whole festival, but it has been helpful to see attitudes towards the festival as a whole.

The marketing team have looked over the results in a season feedback session and have taken them on board for future Big Imaginations marketing. As a result, we’ve also continued/restarted social media plans on Big Imaginations Social Media and are looking at how we can improve for next festival. Responses were generally very positive, so we are pleased with this and looking to build on it for the future.

If there is one message to leave readers with, it’s don’t be afraid to experiment with the Toolkit to make it fit your own style and audiences, and don’t be afraid to ask for help from the CWC team. They’ve been fantastic!

Written by Liz O’Neill, Chief Executive and Artistic Director of Z-Arts

Photo credit: Kaleidoscope, by Filskit Productions – photo Zoe Manders