Evaluating Participatory Work Using the Toolkit

1. Introduction

This guidance document provides a starting point for evaluating participatory projects using the Arts Council England Impact & Insight Toolkit. A participatory project is normally one in which members of the public – adults, children or young people – are involved in and interact with the creative process. They may be involved in making artistic work; help to shape the creative inquiry; and make decisions about the direction and design of the process. A participatory project may or may not result in an ‘end product’ – a final performance, exhibition or other kind of event.

The Impact & Insight Toolkit can be used to evaluate the experience of people involved in a participatory project. The method is similar to that used for evaluating an audience-facing event such as a theatre production or museum exhibition. It can involve self-assessment by the people or organisation running the participatory project, peer review and collecting feedback from participants themselves.

This document provides guidance on carrying out a participatory evaluation, including selecting dimensions to evaluate the project and designing the evaluation process to ensure that you collect the information that will give you the most insight. If you have further questions about evaluating participatory work, please contact support@countingwhatcounts.co.uk.

 

2. Participatory dimensions – background

The Culture Counts platform contains a set of dimensions for evaluating participatory projects. These dimensions have been developed and piloted by cultural organisations in previous projects[1] and further tested among different participant groups including children and young people, people with disabilities and people living in areas of low socioeconomic status[2].

The Participatory dimensions were developed by asking cultural organisations and practitioners to describe the outcomes that are most important to them in their participatory work. The dimensions measure three inter-connected clusters of outcomes, as show in Figure 1 below:

[1] You can read the Participatory Metrics Report here and the Quality Metrics Final Report here.

[2] See Testing the Accessibility of Arts Council England’s Quality and Participatory Metrics

Figure 1: Impact & Insight Toolkit Participatory Dimensions

Figure 1: Impact & Insight Toolkit Participatory Dimensions

There are three dimensions that relate to the creation of a ‘conducive environment’ – a safe space where the project can operate successfully for both the artists and participants involved:

Responsiveness: ‘The organisers responded well to the needs of the group’

Support: ‘People in the group supported each other’

Organisation: ‘The project was well organised.’

 

There are a further three dimensions that relate to the environment but also capture participant experience:

 

Acceptance: ‘I felt like I could be myself’

Belonging: ‘They made me feel part of the team’

Voice: ‘My ideas were taken seriously’

 

The following five metrics relate solely to the quality of the participants’ experience:

 

Authenticity: ‘It felt like a real artistic experience’

Enjoyment: ‘I had a good time’

Experimenting: ‘I felt comfortable trying new things’

Friendship: ‘I felt close to other people involved in the project’

New People: ‘I got to know people who are different to me’

 

There are two dimensions that relate to both participant experience and participant development:

 

Contribution: ‘I felt that my contribution mattered’

Feedback: ‘I got helpful feedback’

 

Finally, there are eight dimensions that focus on development outcomes for participants resulting from their involvement:

 

Artistic Skills: ‘I improved my artistic skills’

Confidence: ‘I feel more confident about doing new things’

Creativity: ‘I feel more able to express myself creatively’

Empathy: ‘It helped me understand other people’s points of view’

Identity: ‘It helped me to see myself differently’

Motivation: ‘I feel motivated to do more creative things in the future’

Skills: ‘I gained new skills’

Stretch: ‘I did something I didn’t know I was capable of’

 

3. Selecting participatory dimensions

Generally, organisations select a subset of these dimensions to evaluate a participatory project. The best way to select dimensions is to ask yourself: ‘what are we trying to achieve with this project? Which of these outcomes are most important to us in meeting our aims?’

Other useful questions to inform your dimension choice include:

  • Which of these dimensions are most relevant and meaningful to the people taking part in this project?
  • Which of these dimensions are most important for the artists we are working with on this project, and for their personal practice?
  • Which of these dimensions are most important for the partners we are working with on this project?
  • Which of these dimensions will most help us to improve our work?
  • Are there outcomes important to this project that are not captured by the Participatory Dimensions? If so, how can we measure them? (Remember that you can create your own custom questions in the Culture Counts platform)

Once you have selected your dimensions, you may want to consider how your selections could shape the design of your participatory process and inform your choices about which artists and partners to work with.

 

4. Designing your participatory evaluation

The process for evaluating a participatory project using the Impact & Insight Toolkit is broadly the same as the process for evaluating an audience-facing work. However, there are some additional considerations when evaluating a participatory project:

A. Participatory project with ‘end products’

Some participatory projects result in an ‘end product’ such as a performance, exhibition or other kind of event. In this case you may want to evaluate both the participatory element of the project using the Participatory Dimensions and the final performance using the Cultural Experience dimensions. This means surveying both participants in the process and audiences or visitors at the final event. This kind of evaluation can provide useful insight into whether a work created through a participatory process has the same kind of impact on an audience as a professionally produced work, and the extent to which the participatory nature of the project is understood and valued by audience members.

For an NPO looking to meet their ACE funding requirements this could mean carrying out a Flexible evaluation when evaluating the participatory part of the project and a Core evaluation when evaluating the final outcome i.e., exhibition or performance.

B. Self assessment

The most straightforward way to carry out self assessment for a participatory project is as a prior objective-setting exercise. You can ask your self assessors to complete a survey to show what they hope the project will achieve for participants. If you are also evaluating a final performance or exhibition, you may want your self-assessors to complete an additional prior survey to show what they hope the event will achieve in terms of audience experience. In some cases it may also be appropriate to ask self assessors to complete a post-project survey to reflect on what they think the project achieved in reality.

Please ensure that, if you choose to not obtain a post self-response, you are meeting any mandatory requirements through other evaluations.

C. Peer review

You may find it useful to invite a peer to observe and give feedback on elements of the participatory process. However, depending on the circumstances it may not feel appropriate or helpful to include an outside observer, particularly if you are working with very small groups, over long time periods and/or on sensitive themes.

If the participatory project results in a final performance or exhibition, you may want that event to be reviewed by a suitable peer using the normal method (as described in the Impact & Insight Toolkit Getting Started with Thoughtful Evaluation Guide.

Please remember, if you are planning to submit your participatory evaluation to Arts Council to contribute towards your funding requirements, please ensure that the participatory evaluation you create complies with your mandatory requirements. Any deviation from the requirements must be approved by your Relationship Manager at Arts Council.

For example, if you are a Band 2 or 3 NPO your mandatory requirements include a minimum of one peer review per evaluation, whether they are Core or Flexible evaluations. If you have a valid reason for not including a peer review in your participatory evaluation please discuss this with your Relationship Manager.

 

D. Working with artists

If your organisation has brought in artists or other external practitioners to help run your participatory project then you have some choices about whether and how to involve those people in your evaluation. You could consider your partner artists as:

  • self assessors – in which case you would ask them to complete a prior survey to show what they are hoping the project will achieve in terms of participant experience and outcomes
  • peer reviewers – in which case you would ask them to complete a post-project survey to show what they think the project achieved in terms of participant experience and outcomes
  • a type of participant – in which case you would ask them to complete a survey to show what the experience of working on the project was like for them and how their practice has developed as a result

Please contact support@countingwhatcounts.co.uk to discuss how you can best define and collect feedback from the various stakeholder groups involved in your participatory project.

E. Evaluation timepoints

‘Summative evaluation is after the fact, post event. Formative can be at any time (before, during, and after), the attraction being that formative evaluation that takes place during an activity can allow feedback to be immediately incorporated back into the practice so that the activity dynamically evolves based on the needs of those involved.’

A further issue to consider is when to collect feedback from the participants in your project. You can limit the feedback process to just one post-project survey, in which you ask participants to reflect on what their experience was like and what the outcomes have been for them. Alternatively, you could collect participant feedback at multiple timepoints and compare the responses, feeding the results into a real-time reflective cycle. This allows you to evaluate the effectiveness of any responsive changes that were made to the project in addressing the needs of participants.

This approach is particularly useful for organisations with theory of change models, which lend themselves well to deploying surveys at the beginning, intermediate and end points of a project. Some organisations have also found that if dimensions are used strategically and appropriately at various points in the activity, the reflection process and results can give participants a ‘sense of control’.

 

5. Participatory projects and NPO mandatory evaluations

If you are an NPO, you will be expected to complete a specific number of core and/or flexible evaluations each financial year. If you are unsure of the type and the number you need to complete, please read the mandatory requirements for your specific funding status.

  • A Flexible evaluation allows you to select a minimum of 4 dimensions from the ACE Participatory dimensions category or Cultural Experience dimensions category
  • A Core evaluation must include the ACE Core dimensions: 6 dimensions to be completed by self, peers and public (Captivation, Concept, Rigour, Relevance, Distinctiveness and Challenge) and an additional 3 dimensions that need to be completed by self assessors and peer reviewers (Excellence, Originality and Excellence) 

As mentioned above (section 4A), an NPO looking to meet their ACE funding requirements could carry out a Flexible evaluation when evaluating the participatory part of the project and a Core evaluation when evaluating the final outcome i.e., exhibition or performance.

For more information about Core and Flexible evaluations please click here.

If your organisation is primarily participatory, please contact your Relationship Manager to discuss whether some or all of your mandatory evaluations can be built around the Participatory Dimensions instead.

 

6. Participatory work with children and young people

The Participatory Dimensions were developed in consultation with organisations that specialise in carrying out participatory work with children and young people, and were designed to complement the Arts Council’s Quality Principles, which aim to raise the standard of work produced by, with and for children and young people. The Impact & Insight Toolkit can therefore be used to collect feedback directly from young participants as well as from intermediaries such as parents and teachers.

In 2017 the accessibility of the Participatory dimensions was tested with around 40 young people aged 9 to 18 in a research project for the Arts Council by Shared Intelligence, The Mighty Creatives and Sarah Pickthall[1]. The research found that ‘the young people in all age groups could respond to use the statements to reflect on their experiences, and were able to hold discussion and provide explanation that supported this conclusion’. The researchers went on to note that ‘while the majority of the metrics were well understood by the young people, there were some which were more challenging’ and ‘required further refinement’. Young people worked with the researchers to suggest some amendments that could be made to the dimensions to ensure that they are easily understood by young participants.

[1] See Testing the Accessibility of Arts Council England’s Quality and Participatory Metrics

 

7. Developing the Participatory dimensions

In 2019-20 we launched the Artform and Participatory Metric Strand of the Impact & Insight Toolkit. Within this workstrand, NPOs that specialise in participatory work had an opportunity to share their experiences of evaluating work using the Participatory dimensions and work with us to further develop the dimensions. As a result of their work we identified the smaller, core set of Participatory dimensions that are currently available in the platform for use in all NPO participatory evaluations. We are continuing to work on refining the dimensions to ensure they are more accessible to different groups including children and young people, people with disabilities and people with little or no involvement in mainstream arts. To read more about the history of the participatory dimensions, read our Participatory Metrics Report.

[3] See Testing the Accessibility of Arts Council England’s Quality and Participatory Metrics

 

Powered by BetterDocs

X
X