Impact & Insight Toolkit Glossary

To view/download a PDF version of this page, click here.

Impact & Insight Toolkit: The Impact & Insight Toolkit project has been commissioned by Arts Council England and is delivered by Counting What Counts, licensing the Culture Counts platform.

It aims to address long-standing challenges in the arts and cultural sector:

  • Measuring artistic, social and economic outcomes that are meaningful to and shaped by arts and cultural organisations
  • Building sector-wide dataset on the impact of publicly funded arts and culture.
  • Giving audiences a voice in debates about the value and impact of arts and culture
  • Strengthening peer review within the arts and cultural sector
  • Building evaluation capacity and capability
  • Encouraging cross-sector collaboration on the collection, interpretation and use of data

Ultimately, the Impact & Insight Toolkit aims to help arts and cultural organisations to understand people’s perceptions of their work and how well this aligns with their creative intentions. It informs and enriches self-evaluation and the conversation between the Arts Council and NPOs about the impact of funded work.

 

Who’s Who

Counting What Counts (CWC): A team of Consultants and Researchers who specialise in impact evaluation and measurement.

Culture Counts (CC): An international performance evaluation platform that has developed research, methodologies and tools to assist state and local governments, councils, cultural organisations and other third sector agencies to implement outcomes-driven and evidence-based decision-making processes.

Arts Council England (ACE): An executive non-departmental public body, sponsored by the Department for Digital, Culture, Media & Sport. ACE champions, develops and invests in artistic and cultural experiences to enrich people’s lives. It supports a range of activities across the arts, museums and libraries – from theatre to digital art – reading to dance, music to literature, and crafts to collections.

National Portfolio Organisations (NPOs): A core group of arts organisations that receive ACE funding. They are split into 3 funding ‘Bands’: Band 1, Band 2 and Band 3. Band 2 and Band 3 organisations are required to use the Toolkit as per their funding agreements; Band 1 organisations are not required to use it but may opt do so if they choose.

Creative People and Places (CPP): ACE-funded organisations whose work centres on place.  In accordance with their funding agreements, some CPPs are required to use the Toolkit.

Sector Support Organisations (SSO): ACE-funded organisations that include Bridge Organisations, Museum Development Providers, umbrella and networking organisations, strategic library and museum partnership bodies, and other organisations that support the arts and cultural sector as part or all of their function.

 

How it Works

Evaluation – An evaluation is the folder which contains the collection of surveys (Public, Peer and Self-Assessor Surveys) used to assess one specific piece of work.

Mandatory Evaluation – Mandatory evaluations are evaluations you must complete in order to meet your funding conditions. Your funding agreement will set out the number of evaluations you must complete each financial year and the minimum content you must include.

For more information on the number and content of your mandatory evaluations see below:

  • Band 2 & 3 see here
  • CPPs (funded post October 2019) see here

Pre event survey – surveys issued before a work/performance/exhibition occurs are referred to as pre event surveys

Post event survey – surveys issues after a work/performance/exhibition occurs are referred to as post event surveys

Self-Assessor Survey – Surveys issued to self-assessors are referred to as self-assessor surveys.  Self-assessors are usually surveyed pre-event survey and post-event survey. The pre-event survey measures the self-assessors’ expectations on the work.  The post-event survey measures the self-assessors’ reflections on the work.

Self-Assessors – Self assessors are usually staff members within the hosting organisation, or better still, a range of key individuals involved in the event’s curation or creation. Organisations can think broadly about the individuals who have contributed to the making of your event; in this way, self-assessors might include the artists, curators, marketing or education staff within your organisation.

Peer Survey – Surveys issued to invited peer reviewers after the work are called ‘Peer’ surveys. Peer reviewers are usually surveyed after they have experienced a work (post-event survey).  This gives them the opportunity to reflect on their experience and subsequently provide feedback.

Peer Reviewers – Peer reviewers are individuals who are deemed to offer useful insight as well as a fair, informed critique of a piece of work.  A Peer Reviewer is anyone that has not been directly involved in the curation or creation of the work that is being evaluated, but that the receiving NPO values and respects their professional opinion. See here for more information on who can be a Peer Reviewer

Public Surveys – Surveys issued to members of the public that have experienced the work are referred to as ‘Public’ surveys.

Public – The public is the general public that experienced the work.  They might be audience members, visitors or participants.

 

Evaluation Content

Dimensions – Metrics used to measure an intrinsic quality of a work. Within the Impact & Insight Toolkit, a dimension statement is made up of an outcome and a statement that captures that outcome. For instance:

Pride:              It strengthened my cultural pride

Core Cultural Experience Dimensions: A set of dimensions that have been identified as having the potential to offer art and cultural organisations greater insights into what people value about their work; allow them to gauge how they are meeting their creative intentions; benchmark against similar organisations; and to help sector professionals talk about quality in a more consistent and confident way.

Custom Questions – These are original questions created by the user and included in a Toolkit survey. Users can add these on the Design page of the survey builder, using a range of content types, including drop-down and multiple choice. Users are encouraged to include some custom questions in each evaluation to ensure that the evaluations speak directly to the unique outcomes of their work and organisation’s mission.

Evaluation Properties – Metadata tags, or descriptors of the work which will enable the organisations to make more detailed comparisons between evaluations over time, or against aggregate datasets, like in the Dimensions Interpretation Tool. These tags include artform, sub-artform, location, and key words. These details are input by a user and, when submitted to Arts Council, feed into their Insights Report, giving valuable context.

Peer Matching Resource – The Peer Matching Resource is a database of over 600 peer reviewers. Within the Culture Counts platform, the Peer Matching Resource allows the user to search for peers to provide a review of their work. Peer reviewers that have registered to join the database may be browsed and filtered so that the user can find a peer reviewer that can provide most valuable and relevant insight. The majority of Peer Reviewers in this database work for an ACE funded organisation such as an NPO and are Toolkit users themselves.

 

Reporting Analysis

Insights Reports* – An Insights Report is a summary of an evaluation’s results, comprising of automatically generated graphs and statistics. It also offers space for the user to reflect on their creative intentions and insights achieved. This report should be submitted to their Relationship Manager within one month of the event’s end date.

Annual Summary Reports* – The Annual Summary Report compiles the evaluation data for four evaluations that an organisation has submitted to ACE. Similar to the Insights Reports, it presents the overall data for these evaluations, with particular focus on the core cultural experience dimensions. Organisations are also given the opportunity to reflect on their creative intentions and insights gained. These reports are to be submitted at the end of each evaluation year.

* Currently only encouraged for NPO use

Dimensions Interpretation Tool – The Dimensions Interpretation Tool is an interactive tool, developed to help NPOs understand what their scores to core dimension questions mean.  This tool allows users to compare their survey results to other Toolkit evaluations, providing artform-specific context for their individual results.

Aggregate data set – Aggregate data refers to information that is collected from multiple sources and/or on multiple measures, variables, or individuals and compiled into data summaries or summary reports. For more information please see https://www.edglossary.org/aggregate-data/

Sample Size –  Sample size refers to the segment of an audience that completes a given survey. The more audience members surveyed, the larger the sample size, and therefore the data is more accurately representative of the wider audience. For more information see: https://impactandinsight.co.uk/sample-size-guidance-2/

 

Development of the Impact & Insight Toolkit 

Strategic Development Strands (SDS) – The SDS is an optional area of work that NPOs can register to participate in.  The SDS provides the opportunity for NPOs to work with CWC to build the best possible Impact & Insight Toolkit to meet the needs of the sector. NPOs involved in the SDS have supported the development of the Toolkit via in-person workshops, webinars and other online development activities. For more information on activity so far please read the report on SDS activity here.

Artform & Museum Metric Strand (AMMS) – The AMMS is an optional area of work that NPOs can register to participate in.  It focuses on the development of artform and museum specific metric statements – dimensions. For more information on activity so far please read the report here.

 

Have we missed anything? Is there a term or word that you think we should include here? If so, please contact us at support@countingwhatcounts.co.uk

 

Go back to Support Materials

 

The information on this page was last updated on 17 March, 2021.

 

 

 

X
X