Does standardisation or customisation work best when evaluating your work? Which approach you take will depend on your aims for the evaluation.
Introduction
When designing surveys, there are three main approaches you can take, and importantly, they’re not mutually exclusive. In most cases, organisations do not need to choose just one route, and a blended approach is often the most effective way forward.
You can:
- Write customised questions for everything, across the board
- Write customised questions that work for your organisation and repeat them across your surveys
- Use standardised, inbuilt questions
In practice, most organisations benefit from a blend of all three. The key is understanding what each approach offers and choosing the right balance for your goals.
Standard Toolkit approach
The Impact & Insight Toolkit primarily uses option 3, drawing on both Dimensions and Question Bank (QB) standardised questions. This is the default approach because it offers several key advantages:
- Confidence in using tested and well-considered questions
- Varied and automated analysis and reporting options
- Strong benchmarking opportunities using quantitative data
- Ability to compare results across surveys
- Seamless integration with our AI Thematic Analysis tool
- Alignment with external frameworks and requirements (e.g., Illuminate)
- Contribute to sector-wide research into the value of arts and cultural experiences
For many organisations, these benefits increase efficiency, reduce uncertainty, and make evaluation findings simpler to establish and act upon.
However, as with everything, there are some trade-offs.
Considering customisation
Customised questions offer greater flexibility and precision. You can:
- Use language that aligns perfectly with your organisation’s brand
- Reflect a specific demographic’s or programme’s needs
- Reduce ambiguity through identifying specifics
This can be particularly valuable when you need to capture something unique about your work, or when you need to ensure the survey resonates with your intended respondents.
If you use a consistent custom question across your surveys, but change the subject every time, you maintain clarity. However, this approach loses the benefits of automated comparison and benchmarking that standardised questions provide. In other words, greater flexibility can sometimes mean giving up some rich analysis and internal efficiency.
Finding the right balance
Ultimately, this is all about balance.
At Counting What Counts, we have addressed the need for greater balance through enabling the subject in most standardised questions to be changed.
So, what does this mean in practice?
Consider the Challenge dimension, ‘It was different from things I’ve experienced before’.
Developments in Culture Counts enable you to specify what ‘it’ is; for instance, ‘The parade, Lottie’s Walk, was different from things I’ve experienced before’.
This also applies to many of the standardised question bank questions. For example, let’s look at the Staff helpfulness (Linguistically Easy Read) question, ‘How helpful were the people working at it?’.
This could be updated to, ‘How helpful were the people working at the House of Bea?’.
Regarding the balance of your evaluations, consider a traditional balance scale:
At one end: Customisation
At the other: Standardisation
The right position depends on what you need your evaluation to do.
To decide where you should sit on that scale, consider your intentions for the evaluation’s results.
If you don’t intend on using it for more formal reporting or comparing your data with other datasets (either your own, those available through the Toolkit or open-source), maybe fully-customised is the right route for you.
However, if you do want to use it for this, we would recommend moving a little more towards the standardised end of the scale.
A blended approach
Does that mean we think that you shouldn’t have any customised questions in your evaluation? Absolutely not!
A typical balanced survey might include:
- 4 standardised dimension questions, benefiting from the flexible subject field where available
- 3 standardised question bank questions, benefiting from the flexible subject field where available
- 2 customised questions that are used consistently across all surveys
- 1 customised question that is unique to this specific survey
The ‘right’ breakdown will look different for everyone.
If you need support in finding the right balance for your organisation, get in touch!
Featured photo by Shubham Sharan on Unsplash