With many thanks to Talawa for sharing their insights on using the Impact & Insight Toolkit (Toolkit) across several events in the Summer of 2020. Written by Melise Bicep, Development & Marketing Coordinator at Talawa Theatre Company.

Background

In March 2020, Talawa Theatre Company was about to open its new Studio in Croydon with Run it Back. It was an exciting time for us as it would have been our first production in our own space. It was also a chance to get to grips with the Impact & Insight Toolkit, which, as a Band 2 National Portfolio Organisation (NPO), is part of our funding agreement with Arts Council England(ACE).

As a touring theatre company which usually doesn’t control its box office, data collection has always been challenging for us. We perform well on collecting case studies, quotes, images from our artists and creatives, but March was the occasion to go beyond qualitative data and dive deeper into numbers. Then lockdown happened.

Five months later, it feels like we’ve never been busier and we now have four new evaluations on the platform, with two live ones. Although they cover various projects (a production, free artists’ development programmes and public training sessions), all of them happened online during or following the UK lockdown. Now, it seems fitting to talk about our overall experience of the evaluation process.

We started lockdown wondering how to best deliver on our mission to support Black artists & creatives during this unparalleled time. We conducted the first free Talawa Cafés via Zoom with the view of doing just that, checking in with the Talawa family, providing moral support and building a better understanding of how to be of further help. From this, we developed a series of One-to-one support sessions alongside group masterclasses in Directing, Writing & Producing, delivered by the Talawa team. With 14 group sessions delivered to 104 artists and creatives, this has been our main and most successful online offering to date.

The demand for our Unconscious Bias, Diversity and Anti-Racism Training has also soared in the wake of the Black Lives Matter protests, and we adapted the training to an online format. On the artistic side, we are currently working on delivering Tales from the Front Line, which will bring to life the real-world testimony and experiences of Black people on the front line of the pandemic in a multi-disciplinary online format.

Dimensions and learnings

The concept of ‘Relevance’ instantly sung to us when designing the survey for Tales from the Front Line, which is still ongoing. Through interviews with Black frontline and key workers, the production explores the historic moment of the Covid-19 crisis and its impact on them; evaluating its perceived relevance (the average agreement level from respondents to this dimension was 89/100) felt apposite.

For our three training-related evaluations, we focussed on Participatory dimensions from which the ‘Contribution’ metric clearly stood out and was used in all of our surveys. It seemed directly connected to Talawa’s mission – to champion Black artists & creatives to enrich cultural life for all. Ensuring that participants in our projects felt like their contribution mattered (the average level of agreement was 96/100) was therefore paramount to us. For Talawa Cafés, it is interesting to note that our expectations closely matched participants’ experience on this dimension; self-assessors gave an equally high level of agreement with the statement on ‘Contribution’ (95/100).

The same is true for ‘Clarity’ where the average level of agreement was 94/100 . They strongly believed that participants would have a clear idea of what they were here to do and participants felt the same, giving an average level of agreement of 96/100.

In contrast, we underestimated ourselves on ‘Feedback’ (75/100) as respondents felt positively about this, giving an average level of agreement of 96/100. We conducted the first Talawa Cafés while still adjusting to the online format which might explain why we rated ourselves lower there.

One metric I wished we used more is ‘Enthusiasm: I would like to see more work like this’ which was only used in our latest evaluation although, as I later realised, it is mentioned in our Theory of Change!

Overall, I think the main challenge for us was to overcome an underlying perception of evaluating as a box-ticking exercise; especially as our initial encounters with the toolkit led us to believe it was better suited to large venues with their own box office rather than touring companies like us.

The support and flexibility of the Counting What Counts (CWC) team has been instrumental in enabling us to truly own our evaluation process and make the toolkit work for us. We received invaluable advice on the dimensions best suited to each project’s aims but also had the opportunity to design surveys from scratch, as needed for the Unconscious Bias, Diversity and Anti-Racism Training for instance.

Not having to provide an evaluation report for this year also allowed us to truly focus on what we wanted to evaluate without worrying too much about our performance or our sample size; although we didn’t perform too badly there (our lowest response rate being 27% for an ongoing evaluation, for example).

Sharing and reflecting on the results of the surveys with the team really helped to make the process and value tangible. It was also for us the occasion to celebrate; it was a real feel-good moment to discover the comments and reflect on what the data told us.

The greatest surprise was the reach of our artists’ development sessions which we were aiming to broaden. For Talawa Cafes, we anticipated that at least 40% of participants would be new to us, which was exceeded with a whopping 76%. We were thrilled to see that we expanded our pool while being able to support more Black artists.

This tendency was confirmed with the regional reach; although most participants were from London (71%), we engaged with 26% of people outside of London, which might not have been possible with in-person sessions.

It was also exciting to see that 100% of Talawa Cafes respondents would attend further online events like this one even when they would be able to attend physical ones. The success of the online format for artists development programmes will certainly inform our future thinking.

It was furthermore valuable to have the make-up of our audience confirmed – on average across our artists’ development sessions, respondents are Black or Mixed-Race (100%); female (83%), within the 25-34 age range (71%); self-identify as early career artists (76%) and as belonging to the working class (68%).

When we know that the total percentage of the NPO workforce in the UK with a Black and Minority Ethnic background is just 11%[1] while ‘16% of those employed in creative roles in 2019 were from working-class backgrounds’[2] this is a promising contribution towards building an inclusive creative economy.

The process has truly been a learning curve; we delved into it quite swiftly as our first project, Talawa Cafes had a very quick turnaround, and there was definitely room for improvement in the evaluation process. As a team, we are now more attuned to the process and the way that it can be adapted, which is great.

We are getting better at building the surveys; taking the habit to go to our Theory of Change, which states clearly what we wish to achieve and measure as an organisation, then designing an outcome matrix to ensure each objective is paired with a dimension.

We are now compiling the results in an Impact document to be shared with our stakeholders, and as we will shortly be reviewing our fundraising strategy, we are aiming at building an evaluation strategy in tandem. For Talawa, it feels like we’re going beyond the ‘let’s explore the toolkit’ phase to something more mature which goes further and deeper and is allied to our business and artistic aims.

If I had one message to leave you with, it would be this: Manipulate the raw data for yourself. It can feel quite daunting to face a full spreadsheet with no idea where to begin; but once you dive into it, it gives you the flexibility to make the data speak in line with your organisation’s mission and objectives. Although the platform provides a comprehensive analysis which is an excellent starting point, you will always know your organisation better than it does!

Photo credit: Talawa Theatre Company, Tales from the Frontline, 2020

[1] Arts Council England, 2020, Equality, Diversity and the Creative Case

[2] Creative Industries Policy and Evidence Centre, August 2020: Getting in and getting on: Class, participation and job quality in the UK Creative Industries