An Introduction to Peer Review

Jump to a particular section by clicking on its heading below. Or, view/download a PDF version of this pagehere.

blank sheet of paper and writing materials

What is peer review? 

Peer review is a vital element of the Impact & Insight Toolkit. The Toolkit works by asking audiences, participants and peer reviewers what they think of a cultural event or project and comparing this with the organisation’s original objectives for the work.

Peer review provides fresh professional perspectives from people that aren’t involved in the internal development of projects and enables organisations to understand how their work is perceived within their artform or specialist field. Organisations are interested to see how peer responses compare with the views of regular audience members and participants.

The ‘triangulation’ of self, peer and public views gives organisations a rounded appraisal of their work; it helps them to understand whether they are achieving what they set out to achieve, and how they can develop their work in the future.

Peer review is also valuable for peers themselves. It enables them to experience different types of work, and to explore how their own practice relates to what others in their field are doing. We hope that by expanding and diversifying peer review, the Impact & Insight Toolkit will help cultivate stronger professional networks across the sector, creating a supportive, challenging and stimulating environment for arts and culture to thrive.

This document provides guidance to organisations that are looking for peers to review their work; to organisations that would like to nominate staff members to act as peer reviewers; and to individuals who are interested in becoming peer reviewers. It provides an overview of the peer review process, from start to finish, and concludes with some guiding principles of peer review.

How do I select a peer to review my work? 

If you have an event or project that you are planning to evaluate using the Impact & Insight Toolkit, you will need to identify and invite suitable peers to review the work. You need at least one peer reviewer per event; we normally recommend that you engage three to five peers.

A. Who is a suitable peer reviewer? 

In a nutshell, a peer reviewer is someone whose opinion you value. You are looking for someone who you think can offer useful insight as well as a fair, informed critique of your work.  You should choose individuals who are not invested in the specific work you are evaluating, and you may want to look beyond the ‘usual suspects’ to access perspectives on your work that you might not normally hear.

The type of professional who you would like to hear from will vary from event to event. For example, if you are evaluating an education performance or project involving schoolchildren, teachers observing the work may be ideal peer reviewers. A company that is evaluating a piece of physical theatre around environmental themes may want to approach a physical theatre practitioner and/or someone with expertise in communicating environmental issues to the public.

If you decide to approach multiple peer reviewers, try and choose people with a variety of expertise, experience and specialisms. Having a diversity of peers reviewing your event will offer you a well-rounded perspective of your evaluation. It is also worth considering peers’ locations when choosing; if they are far away, you will need to consider the cost of reimbursing them for their travel costs and time.

B. How do I find peer reviewers? 

To find a suitable peer to review an event, the first place we would encourage you to look is within your own professional networks.  Your networks may contain people from local arts organisations, touring companies, independent artists and academic institutions, amongst others.  By engaging with your current contacts, you are acknowledging the value you place on their professional opinions of your work. 

Within the Culture Counts platform, the peer matching resource allows you to search for peers. You may browse and filter a list of creative professionals who have registered their interest in participating as a peer reviewer. They may be working with a National Portfolio Organisation or Sector Support Organisation that is registered to use the Impact & Insight Toolkit, or they may have signed up as an external individual interested in being part of the project.

Upon registering, peer reviewers will have created a personal profile which will give information about their job role, organisation, location, artform specialisations and additional expertise along with a short biography.

The peer matching feature will allow you to browse all the registered peer reviewers, filtering by artform and location. The list view shows basic details of each peer reviewer’s profile, such as artform, location and job title. Once you click through to a peer’s card, you will see their full profile including their name and a short biography giving you more information about their interests and experience. Names are omitted from the initial list view, encouraging you to browse peer reviewers based on their job role and specialism, rather than names you may know. For more information about how to identify peers using the portal, please see our peer review support video: https://youtu.be/lAq0ZYUoHsQ

Remember that you are not obliged to use the Peer Portal to identify a suitable peer to review your event; you are welcome to approach people from your own networks.  If they are interested in peer reviewing the work of other organisations using the Toolkit, they are welcome to register to the Peer Portal.  They would then be visible in the Peer Portal and other Toolkit users would be able to approach them to ask if they would be a peer reviewer for their work.  When selecting your peer reviewers, we would recommend that you engage with peers from both your own network and the new resource available to you, in order for you to achieve a well-rounded range of professional perspectives on your work.

C. How do I invite a peer to review an event? 

The Culture Counts platform enables you to select a peer from the Peer Portal and send them an invitation to review your event. You can select a peer reviewer by using the checkbox to the left of each user, then clicking ‘Invite’ at the top of the page. You can then personalise the message each peer reviewer receives, providing details of the event.  Each peer reviewer who receives this message will have the opportunity to accept or decline the request.   For more information about how to invite peers using the resource, please see our peer review support video: https://youtu.be/lAq0ZYUoHsQ

If you are approaching a potential peer reviewer who is not registered on the Impact & Insight Toolkit Portal, you may need to tell them a bit about the Impact & Insight Toolkit and the peer review process. You might want to share our document Becoming a Peer Reviewer which explains what the toolkit is, the benefits of being a peer reviewer, what peer review involves and how information provided by peers will be used. You will need to explain what event you are asking them to review and provide some basic information about the event such as date and location. If a peer who is not registered on the Peer Portal agrees to review your event, you can add their email address to the Culture Counts platform using the box towards the bottom of the ‘Invite’ page of your dashboard. This will enable you to later distribute surveys to your peer.

Once a peer has agreed to review your event, you may choose to email the reviewer with further information about the event. We suggest that you provide enough information to enable peers to understand the aims and intentions behind the work, and where you see the work sitting within your overall programme. Some events may need more explanation than others; some peers – particularly those who don’t know your organisation well – may need more information than others.

It is likely that you will need to arrange a time for each peer to come and see the work and make tickets available as required.

Peers on the Peer Portal are under no obligation to accept invitations to review work; if you approach someone who is unable to review your event, we suggest that you look again within your existing networks or the Peer Portal.

D. How do I send a survey to my peer reviewers? 

Once a peer has attended your event, you will need to send a survey to capture their feedback. The ‘Summary’ page of your dashboard will show all the peers in the portal who have accepted the invitation to review your event, and any other peers who you have approached outside of the portal and whose email addresses you provided on the ‘Invite’ page. From the ‘Summary’ page you can email the survey to each peer using a URL specific to that individual peer. Detailed information about how to distribute surveys to peers is provided in the Impact & Insight Toolkit Platform Guide.


Peer Review of Participatory Work

If you are using the Impact & Insight Toolkit to evaluate a participatory project then you will need to consider whether peer review is appropriate. If the participatory process results in a performance or exhibition, you may want that event to be reviewed by a suitable peer. Alternatively, you may find it useful to invite a peer to observe and give feedback on elements of the participatory work. However, in some cases it may not feel appropriate or useful to include an outside observer – for example if you are working with very small groups, over long time periods and/or on sensitive themes. In these instances, your participatory evaluation would include self and participant assessment only. Please contact support@countingwhatcounts.co.uk if you would like more advice on evaluating participatory work.


How do I nominate people within my organisation to become peer reviewers?

Organisations may be keen to nominate staff members to act as peer reviewers for other NPOs, both to provide individual professional development opportunities and to bring learning back into the organisation. If you are a Band 2 or 3 NPO, the Arts Council requires your organisation to be available to provide a minimum of four peer reviews per year. These assessments can be carried out by one staff member or shared out between multiple staff members. There is no limit to the number of people you can nominate from your organisation to act as peer reviewers – you may want to put forward someone from your creative team, say, and someone with education or audience development expertise.

To nominate someone from your organisation, you will need to tell them a bit about the Impact & Insight Toolkit and the peer review process. You might want to share our document Becoming a Peer Reviewer which explains what the Toolkit is, the benefits of being a peer reviewer, what peer review involves and how information provided by peers will be used.

If your staff member is happy to become a peer reviewer, they will need to register their details. In your Culture Counts dashboard, open an evaluation and go to the ‘Invite’ page. Here you can register yourself or a colleague as a peer reviewer by clicking ‘Register a Peer’. If you would like your staff members to upload their own details, you can copy and share your organisation’s bespoke peer registration link.

After uploading their details to the Peer Portal, individuals may be approached by other NPOs with invitations to review events. Some peer reviewers will be approached more than others, particularly if they have specialist knowledge of an artform or type of practice. Individual peer reviewers are under no obligation to review work; they can choose which invitations to accept based on their availability and interest. The Arts Council suggests:

As a recommendation towards meeting your minimum requirements as an NPO (4 peer evaluations which can be carried out by different nominated peers from your organisation), that you accept peer evaluation requests across each year based on capacity and other commitments. You may choose to balance evaluation visits equally across the year, for example by accepting one each quarter and planning in advance when throughout the 12-month period you are available to accept requests. Provided that you have completed your minimum reviews as stated in your funding agreement, you are under no obligation to accept further requests where capacity and resource are limited. However, where peers do have capacity available, you can accept and complete a number of additional reviews for other organisations to support the initiative.’

If you would like to register someone from outside your organisation to the Peer Portal, please contact support@countingwhatcounts.co.uk.

men working at a table

How do I become a peer reviewer?

If you are a practitioner working in or with the arts and cultural sector, you are welcome to register as an Impact & Insight Toolkit peer reviewer, regardless of whether you work at an Arts Council funded organisation. You might be interested in becoming a peer reviewer if you:

  • would like to use your knowledge and skills to support the development of the arts and cultural sector
  • have professional experience of one or more artforms or creative fields
  • are committed to providing honest feedback and constructive criticism to help support an organisation’s development

If you would like to register as an Impact & Insight Toolkit peer reviewer, please contact support@countingwhatcounts.co.uk to arrange for your details to be uploaded to the Impact & Insight Toolkit Peer Portal. This can only be accessed by organisations that have registered to use the Impact & Insight Toolkit. Once your details have been uploaded, you can be contacted by organisations that are interested in your professional opinion on their work.

If you are invited by an organisation to attend and review a performance, exhibition or project, you can accept or decline depending on your interest and availability. When you accept an invitation to review an event, you will need to arrange to attend the event and may be asked to read some background information in advance. After attending the event, you will be asked to complete a short online survey. The survey will take about ten minutes to fill in, depending on how much information you wish to provide.

If you are interested in reviewing a particular performance, exhibition or project, but have not been approached by the organisation in question, you are welcome to contact the organisation directly and ask to be included as a peer reviewer.

The Peer Review Process

The chart below provides a summary of the overall peer review process, distinguishing between actions for organisations and actions for peer reviewers.

Figure 1: Summary of Impact & Insight Toolkit peer review process

The Principles of Peer Review

This final section proposes some principles of peer review that relate to everyone involved in the Impact & Insight Toolkit, including the Arts Council as funder; Counting What Counts as provider; NPOs looking for peers to review their work; individuals wishing to contribute as peer reviewers.

1. Organisations need different perspectives

The peer review element of the Impact & Insight Toolkit will work best if it enables organisations to understand the perspectives of practitioners from a wide range of backgrounds

If you are recruiting people to join the database of peer reviewers, can you prioritise people whose backgrounds and viewpoints are not well-represented in the cultural sector at present?

If you are looking for someone to review your work, can you enable more diverse voices to be heard through your selection of peers? Can you create valuable professional development opportunities for practitioners at different stages of their careers

2. Contributions to peer review 

NPOs and individual peers are asked to contribute to the development and embedding of an effective peer review system within the arts and cultural sector

If you are a peer reviewer, you will contribute your time – both to attend events and to provide feedback afterwards via a short online survey

If you are an organisation that has asked a peer to review your work, you will need to contribute a ticket to your event, and you may wish to make a contribution towards their travel expenses.

3. All expertise is valid

  • The Impact & Insight Toolkit works on the assumption that organisations are best placed to decide who has the necessary expertise and insight to review their work
  • Peers with considerable experience in their artform or specialist field can expect to receive more invitations to review work; however, organisations often want to hear feedback from emerging as well as established practitioners
  • The opinions of one peer group are no more valid than another; they are simply different perspectives. When interpreting peer feedback, organisations need to reflect on who their reviewers are: why did these peers respond to this work in this way?

4. Feedback to support practice

  • The primary aim of the Impact & Insight Toolkit is to help organisations to improve their practice rather than prove their worth; peer review is only valuable if it offers a fair critique of an organisation’s work
  • Peer review needs to be approached with a spirit of enquiry, with organisations approaching peers who can challenge their thinking and peers committed to providing honest, supportive and constructive feedback
  • A peer review is a snapshot of an organisation’s work as experienced by one individual, and will not have a direct impact on any decisions made about an organisation’s funding

5. Appropriate use of data

  • Peer reviews are ‘pseudonymised’, which means that when survey responses are downloaded, personal information about peers (e.g. email address) is not shown
  • Organisations cannot see which survey response came from which peer reviewer unless personally identifiable questions are asked and answered
  • When analysing data, organisations will typically look at the average scores given by their peer reviewers, and compare them with average audience scores and their own self-assessment
  • Data from all survey respondents will be anonymised and provided in a large, public dataset that anyone can analyse to better understand the value of publicly funded arts and culture

6. Local peer clusters

  • Peer review can be a useful way of connecting organisations in geographical areas and encouraging collaboration
  • Local peer clusters offer opportunities for peers to learn from each other, and for organisations to develop a collective as well as individual approach to improvement
  • If you are working with NPOs in a region, what can you do to encourage a joined-up approach to peer review?

7. Enabling great peer review

  • The Impact & Insight Toolkit is an opportunity for us all to learn more about how to give and receive useful feedback on arts and cultural events
  • Counting What Counts will work with NPOs to produce guidance and resources to share learning about the purposes, processes and outcomes of peer review
  • We can learn from existing research, peer review systems in other countries and best practice in other sectors, particularly education

 

Go back to Support Materials

 

The information on this page was last updated on 1 July, 2019.

Image Credit: Kelly Sikkema (image 1) and Nik MacMillan (image 2) via Unsplash

X
X