Evaluating Participatory Work Using the Impact & Insight Toolkit

This page provides a starting point for evaluating participatory projects using the Arts Council England Impact & Insight Toolkit. A participatory project is normally one in which members of the public – adults, children or young people – are involved in and interact with the creative process. They may be involved in making artistic work; help to shape the creative inquiry; and make decisions about the direction and design of the process. A participatory project may or may not result in an ‘end product’ – a final performance, exhibition or other kind of event.

Group of people huddling

The Impact & Insight Toolkit can be used to evaluate the experience of people involved in a participatory project. The method is similar to that used for evaluating an audience-facing event such as a theatre production or museum exhibition. It can involve self-assessment by the people or organisation running the participatory project, peer review and collecting feedback from participants themselves.

Here we offer guidance on carrying out a participatory evaluation, including selecting dimensions to evaluate the project and designing the evaluation process to ensure that you collect the information that will give you the most insight. If you have further questions about evaluating participatory work, please contact support@countingwhatcounts.co.uk.

Jump to different sections of this page by clicking on the headings below or view/download a PDF version of this page here.

    1. Participatory Dimensions
    2. Designing Your Participatory Evaluation
    3. Participatory Work with Children and Young People
    4. Developing the Participatory Dimensions

1. Participatory Dimensions

The Culture Counts platform contains a set of dimensions for evaluating participatory projects. These dimensions have been developed and piloted by cultural organisations in previous projects[1] and further tested among different participant groups including children and young people, people with disabilities and people living in areas of low socioeconomic status[2].

The Participatory Dimensions were developed by asking cultural organisations and practitioners to describe the outcomes that are most important to them in their participatory work. The dimensions measure three inter-connected clusters of outcomes, as show in Figure 1 below:

venn diagram

Figure 1: Impact & Insight Toolkit Participatory Dimensions

There are four dimensions that relate to the creation of a ‘conducive environment’ – a safe space where the project can operate successfully for both the artists and participants involved:

Responsiveness: ‘The organisers responded well to the needs of the group’

Support: ‘People in the group supported each other’

Clarity: ‘I was clear about what we were all here to do’

Organisation: ‘The project was well organised.’

 

There are a further six dimensions that relate to the environment but also capture participant experience:

Acceptance: ‘I felt like I could be myself’

Belonging: ‘They made me feel part of the team’

Voice: ‘My ideas were taken seriously’

Intention: ‘I felt able to shape the intention of the project’

Respect: ‘I was treated as an equal’

Trust: ‘I trusted the other people involved’

 

The following six metrics relate solely to the quality of the participants’ experience:

Authenticity: ‘It felt like a real artistic experience’

Enjoyment: ‘I had a good time’

Experimenting: ‘I felt comfortable trying new things’

Friendship: ‘I felt close to other people involved in the project’

Intensity: ‘I felt deeply involved in the process’

New People: ‘I got to know people who are different to me’

 

There are two dimensions that relate to both participant experience and participant development:

 

Contribution: ‘I felt that my contribution mattered’

Feedback: ‘I got helpful feedback’

 

Finally, there are twelve dimensions that focus on development outcomes for participants resulting from their involvement:

Achievement: ‘I was amazed by what we achieved’

Artistic Skills: ‘I improved my artistic skills’

Confidence: ‘I feel more confident about doing new things’

Creativity: ‘I feel more able to express myself creatively’

Creative Legacy: ‘I now have creative ambitions I didn’t have before’

Empathy: ‘It helped me understand other people’s points of view’

Identity: ‘It helped me to see myself differently’

Motivation: ‘I feel motivated to do more creative things in the future’

Opportunity: ‘The project opened up new opportunities for me’

Skills: ‘I gained new skills’

Stretch: ‘I did something I didn’t know I was capable of’

Worldview: ‘It helped me understand something new about the world’

 

Selecting Participatory Dimensions

Generally, organisations select a subset of these dimensions to evaluate a participatory project. The best way to select dimensions is to ask yourself: ‘what are we trying to achieve with this project? Which of these outcomes are most important to us in meeting our aims?’

Other useful questions to inform your dimension choice include:

  • Which of these dimensions are most relevant and meaningful to the people taking part in this project?
  • Which of these dimensions are most important for the artists we are working with on this project, and for their personal practice?
  • Which of these dimensions are most important for the partners we are working with on this project?
  • Which of these dimensions will most help us to improve our work?
  • Are there outcomes important to this project that are not captured by the Participatory Dimensions? If so, how can we measure them? (Remember that you can create your own custom questions in the Culture Counts platform)

Once you have selected your dimensions, you may want to consider how your selections could shape the design of your participatory process and inform your choices about which artists and partners to work with.

 

2. DESIGNING YOUR PARTICIPATORY EVALUATION

The process for evaluating a participatory project using the Impact & Insight Toolkit is broadly the same as the process for evaluating an audience-facing work, which is described in the Impact & Insight Toolkit User Guide. However, there are some additional considerations when evaluating a participatory project:

 

Participatory Projects With ‘End Products’

Some participatory projects result in an ‘end product’ such as a performance, exhibition or other kind of event. In this case you may want to evaluate both the participatory element of the project using the Participatory Dimensions and the final performance using the Cultural Experience Dimensions. This means surveying both participants in the process and audiences or visitors at the final event. This kind of evaluation can provide useful insight into whether a work created through a participatory process has the same kind of impact on an audience as a professionally produced work, and the extent to which the participatory nature of the project is understood and valued by audience members.

 

Self Assessment

The most straightforward way to carry out self assessment for a participatory project is as a prior objective-setting exercise. You can ask your self assessors to complete a survey to show what they hope the project will achieve for participants. If you are also evaluating a final performance or exhibition, you may want your self-assessors to complete an additional prior survey to show what they hope the event will achieve in terms of audience experience. In some cases it may also be appropriate to ask self assessors to complete a post-project survey to reflect on what they think the project achieved in reality.

 

Peer Review

You may find it useful to invite a peer to observe and give feedback on elements of the participatory process. However, depending on the circumstances it may not feel appropriate or helpful to include an outside observer, particularly if you are working with very small groups, over long time periods and/or on sensitive themes.

If the participatory project results in a final performance or exhibition, you may want that event to be reviewed by a suitable peer using the normal method (as described in the Impact & Insight Toolkit User Guide).

 

Working With Artists

If your organisation has brought in artists or other external practitioners to help run your participatory project then you have some choices about whether and how to involve those people in your evaluation. You could consider your partner artists as:

  • self assessors – in which case you would ask them to complete a prior survey to show what they are hoping the project will achieve in terms of participant experience and outcomes
  • peer reviewers – in which case you would ask them to complete a post-project survey to show what they think the project achieved in terms of participant experience and outcomes
  • a type of participant – in which case you would ask them to complete a survey to show what the experience of working on the project was like for them and how their practice has developed as a result

Please contact support@countingwhatcounts.co.uk to discuss how you can best define and collect feedback from the various stakeholder groups involved in your participatory project.

 

Evaluation Timepoints

‘Summative evaluation is after the fact, post event. Formative can be at any time (before, during, and after), the attraction being that formative evaluation that takes place during an activity can allow feedback to be immediately incorporated back into the practice so that the activity dynamically evolves based on the needs of those involved.’

A further issue to consider is when to collect feedback from the participants in your project. You can limit the feedback process to just one post-project survey, in which you ask participants to reflect on what their experience was like and what the outcomes have been for them. Alternatively, you could collect participant feedback at multiple timepoints and compare the responses, feeding the results into a real-time reflective cycle. This allows you to evaluate the effectiveness of any responsive changes that were made to the project in addressing the needs of participants.

This approach is particularly useful for organisations with theory of change models, which lend themselves well to deploying surveys at the beginning, intermediate and end points of a project. Some organisations have also found that if dimensions are used strategically and appropriately at various points in the activity, the reflection process and results can give participants a ‘sense of control’.

 

Participatory Projects and Mandatory Evaluations

The Arts Council requires Band 2 and 3 NPOs to carry out at least four evaluations per year using the Impact & Insight Toolkit and to share their findings with their Relationship Managers. For the most part these will be evaluations of audience-facing work using the core Cultural Experience Dimensions. However, if your organisation is primarily participatory, or you have a key piece of participatory work that you wish to evaluate, please contact your Relationship Manager to discuss whether some or all of your mandatory evaluations can be built around the Participatory Dimensions instead.

 

3. Participatory Work with Children and Young People

The Participatory Dimensions were developed in consultation with organisations that specialise in carrying out participatory work with children and young people, and were designed to complement the Arts Council’s Quality Principles, which aim to raise the standard of work produced by, with and for children and young people. The Impact & Insight Toolkit can therefore be used to collect feedback directly from young participants as well as from intermediaries such as parents and teachers.

In 2017 the accessibility of the Participatory Dimensions was tested with around 40 young people aged 9 to 18 in a research project for the Arts Council by Shared Intelligence, The Mighty Creatives and Sarah Pickthall[3]. The research found that ‘the young people in all age groups could respond to use the statements to reflect on their experiences, and were able to hold discussion and provide explanation that supported this conclusion’. The researchers went on to note that ‘while the majority of the metrics were well understood by the young people, there were some which were more challenging’ and ‘required further refinement’. Young people worked with the researchers to suggest some amendments that could be made to the dimensions to ensure that they are easily understood by young participants.

In 2019-20 we will be working with relevant NPOs to refine the Participatory Dimensions, incorporating the recommendations made by the children and young people that took part in the accessibility testing described above. We also plan to explore whether and how the dimensions and data collection process could be adapted to capture feedback from children aged 8 and under.

*Please note, the links above will take you away from the Impact and Insight Toolkit and to YouTube. Please ensure you are happy for this to happen before clicking and have familiarised yourself with their cookie and privacy policies.

 

4. Developing the Participatory Dimensions

In 2019-20 we will launch the Artform and Participatory Metric Strand of the Impact & Insight Toolkit. Within in this workstrand, NPOs that specialise in participatory work will have an opportunity to share their experiences of evaluating work using the Participatory Dimensions and work with us to further develop the dimensions. This will include exploring the possibility of identifying a smaller, core set of Participatory Dimensions for use in all NPO participatory evaluations, as well as refining the dimensions to ensure they are accessible to different groups including children and young people, people with disabilities and people with little or no involvement in mainstream arts. If you have not yet registered for the Artform and Participatory Strand but would like to take part, please contact support@countingwhatcounts.co.uk

[1] You can read the Participatory Metrics Report here and the Quality Metrics Final Report here.

[2] See Testing the Accessibility of Arts Council England’s Quality and Participatory Metrics

[3] See Testing the Accessibility of Arts Council England’s Quality and Participatory Metrics

 

The information on this page was last updated on 17 June, 2019.

Image Credit: Perry Grone via Unsplash

X
X