1. Introduction

 

Use this guidance as a starting point for evaluating projects where members of the public are involved in, or interact with, the creative process. They could be involved in the making of the artistic work, decisions around the design or process, or they might be asked to carry out an activity, for example a workshop or training. Sometimes these works or projects lead to an ‘end product’ and sometimes they don’t.

 

2. Participatory Dimensions

Background

We know evaluating participatory experiences requires a different approach to more presentational work. That’s why we worked with arts and cultural organisations to develop, pilot and curate a set of participatory dimension questions. Read about their:

• Origins in the Participatory Metrics Report, 2016
• Development in the Quality Metrics Final Report, 2016
• Accessibility in Shared Intelligence and The Might Creatives’ Testing the Accessibility of Art Council England’s Quality and Participatory Metrics Report, 2017
• Curation in the Artform and Museum Metric Strand Summary Report, 2019

We are continuing to work on refining the dimensions to ensure they are more accessible to different groups including children and young people, people with disabilities and people with little or no involvement in mainstream arts.

 

The dimensions

Prior to April 2023 participatory dimensions were located in their own ‘Participatory’ category on the Culture Counts platform. This changed during the development of our Dimensions Framework, and they are now part of the ‘Qualities’, ‘Cultural Outcomes’, ‘Social Outcomes’ and ‘Economic Outcomes’ categories:

Qualities

• Relevance: It had something to say about modern society
• Enjoyment: I had a good time
• Experimenting: I felt comfortable trying new things
• Peer support: I felt supported in the group
• Local Impact: It’s important that it’s happening here
• Presentation: It was well presented

Cultural Outcomes

• Worldview: It helped me to understand something new about the world
• Motivation: It made me feel more motivated to do creative things in the future

Social Outcomes

• Acceptance: I felt like I could be myself
• Equality: I felt I was treated as an equal
• Respect: I felt I was treated with respect
• Voice: I felt like my ideas were taken seriously
• Achievement: I was amazed by what we achieved
• Confidence: It made me feel more confident about doing new things
• Stretch: I did something I didn’t know I was capable of

Economic Outcomes

• Opportunity: It opened up new opportunities for me

 

Accessing the dimensions

In the Culture Counts platform, navigate to the ‘Design’ page of your survey, click on ‘Dimensions’ and then select the relevant category from the ‘Dimension Library’. Scroll through the list of dimension questions and select the ones you want to add to your survey.

 

Using the dimensions

When selecting your participatory dimensions, ask yourself, “What are the intentions for this work?”.

Other useful questions to inform your dimension choice include:

• Which of these dimensions are most relevant and meaningful to the people taking part in this project?
• Which of these dimensions are most important for the artists we are working with on this project, and for their personal practice?
• Which of these dimensions are most important for the partners we are working with on this project?
• Which of these dimensions will most help us to improve our work?

Once you have selected your dimensions, you may want to consider how your selections could shape the design of your participatory process and inform your choices about which artists and partners to work with.

 

3. Participatory question bank

Background

We have produced a small set of standardised questions to support you in your evaluation of participatory experiences, located in the platform’s question bank.

These questions were developed by:

1. Exploring custom questions added by National Portfolio Organisations (NPOs) to their participatory-focussed surveys within the Toolkit
2. Finding common links and themes between the custom questions
3. Developing standardised versions of the commonly asked custom questions
4. Condensing the list of questions to avoid overlap and to increase opportunity for future research
5. Seeking approval from Arts Council England (ACE)

 

The questions

Question subject	Question wording	Instruction text	Time to answer (s)	Question Type	Answer Options
Experience benefits	Which of the following benefits do you feel this experience has provided you with?	Select from the list below	10	Multiple choice Question	· Increased physical health · Increased sense of wellbeing · Learned something new · Developed skills/knowledge I already had · Interacted with new people · Interacted with people I’ve known before · Inspired me to do more creative activities in the future · Inspired me to further my pre-existing creative practice
Free text improvement	How could your experience have been improved?	Enter your answer below	15	Free text Question	Freetext
Prior participation	Have you taken part in an activity provided by this organisation before?	Please choose from the options below	5	Yes/No Question	· Yes · No
Prior participation this org	Have you taken part in an activity provided by this organisation before?	Please choose from the options below	5	Yes/No Question	· Yes · No
Short text highlight	In a few words, what has been the highlight of your experience?	Enter your answer below	10	Short text Question	Freetext
Wellbeing increase	Do you feel an increase in any of the following?	Please select as many of the below that you feel have increased since your experience.	10	Multiple choice Question	· Optimism for the future · Feeling of usefulness · Feeling of relaxation · Interest in other people · Energy levels · Ability to deal well with problems · Clarity of thoughts · Feeling good about myself · Closeness to other people · Confidence · Ability to make up my own mind about things · Love from others · Interest in new things · Cheerfulness · None of the above

 

Accessing the question bank

In the Culture Counts platform, navigate to the ‘Design’ page of your survey, click on ‘Question Bank’ and then select ‘Participatory’. Scroll through the list of questions and select the ones you want to add to your survey.

 

Using the question bank

When selecting your questions from the question bank, ask yourself “What other quantifiable factors will help me understand the impact of this work?”

 

Each question you ask should help you to understand:

1. Whether your intentions for the work you’re evaluating were met
2. What impact your work has had on its participants

There is no ‘one size fits all’ approach and we would encourage you to consider and select the questions that you feel will genuinely support you in your evaluation. Don’t ask questions for the sake of it!

Consider including a combination dimensions and question bank questions. A well-considered combination will not only provide you with more context and understanding for interpreting your results, but also provide some variety for your respondents, avoiding survey fatigue.

 

4. Evaluation design

Summative vs formative evaluations

Summative evaluations collect feedback from one post-project survey, which asks participants to reflect on what their experience was like and what the outcomes have been for them.

Formative evaluations collect participant feedback at multiple timepoints and compare the responses.

Formative evaluations can be useful for organisations with theory of change models, which lend themselves well to deploying surveys at the beginning, intermediate and end points of a project. Some organisations have found that if dimensions are used strategically and appropriately at various points in the activity, the reflection process and results can give participants a ‘sense of control’.

To analyse formative evaluation data use the ‘Compare’ feature in the Analytics Dashboard or export the data from the Culture Counts platform.

 

5. Survey delivery

Peer reviewer

We are often asked how to implement peer review when delivering and evaluating participatory activities. With such a wide range of participatory work, there isn’t a ‘one size fits all’ answer.

You may find it useful to invite a peer to observe and give feedback on elements of the participatory process. However, depending on the circumstances, it may not feel appropriate or helpful to include an outside observer, particularly if you are working with very small groups, over long time periods and/or on sensitive themes.

If the participatory project results in an ‘end product’ for example a final performance or exhibition, you may choose for this output to be reviewed, rather than the process itself, by a suitable peer.

 

Artists and external practitioners

If your organisation has brought in artists or other external practitioners to help run your participatory project, you can choose whether and how to involve them in your evaluation. You could consider your partner artists as:

• Self assessors – ask them to complete a prior and post survey to show what they are hoping the project will achieve, and how they felt it went, in terms of participant experience and outcomes
• A type of participant – ask them to complete a survey to show what the experience of working on the project was like for them and how their practice has developed as a result

 

Children and young people

The Participatory Dimensions were developed in consultation with organisations that specialise in carrying out participatory work with children and young people, and were designed to complement the Arts Council’s Quality Principles, which aim to raise the standard of work produced by, with and for children and young people. The Impact & Insight Toolkit can therefore be used to collect feedback directly from young participants as well as from intermediaries such as parents and teachers.

In 2017 the accessibility of the Participatory dimensions was tested with around 40 young people aged 9 to 18 in a research project for the Arts Council by Shared Intelligence, The Mighty Creatives and Sarah Pickthall: Testing the Accessibility of Arts Council England’s Quality and Participatory Metrics. The research found that ‘the young people in all age groups could respond to use the statements to reflect on their experiences, and were able to hold discussion and provide explanation that supported this conclusion’. The researchers went on to note that ‘while the majority of the metrics were well understood by the young people, there were some which were more challenging’ and ‘required further refinement’. Young people worked with the researchers to suggest some amendments that could be made to the dimensions to ensure that they are easily understood by young participants.

The team at Counting What Counts is ready to support you in evaluating participatory work. We can advise on question choice, delivery methods and which respondent categories you may wish to use. Please do not hesitate to get in touch with any questions.