The uses and misuses of sentiment measurement for cultural experiences It is a commonly held belief that cultural experiences should make us ‘feel’ something; indeed, this is often considered to be the purpose of the arts. As our previous blogpost ‘Exploring ‘feelings’ prompted by arts and cultural experiences’ explained, this was seen to be of particular value to organisations that use the Impact & Insight Toolkit (Toolkit) and something that hasn’t always been offered. Knowing how visitors, audience members or participants are feeling could be a potentially useful tool in reflecting on the efficacy of your programme, but the seemingly simple question of ‘how do you feel?’ is inextricably wrapped up in broader conversations about happiness and wellbeing. While we may want to capture positive affective states like ‘happiness’ and ‘joy’, even deciding what we mean by these terms can be complex. We have decided to address this complexity through the inclusion of a tried and tested sentiment framework that includes a variety of both positive and negative emotion. This second blogpost in the series of three intends to provide you with important information you should consider before implementing the framework. This blogpost might be a little longer than we normally aim for, but it’s all important! Please see the three questions addressed in this post: What is sentiment? What are the risks and caveats of trying to measure sentiment? How should you use sentiment measurement? What is sentiment? Sentiment is about positive and negative affect (PA and NA) or, in other words, positive emotions and negative emotions. We will go into more detail about the specifics of sentiment measurement frameworks, and the one that we will include in the platform, in the third blogpost of the series. For now, it is important to understand a little bit about where these measures come from and how they are used. Positive and negative affect (PANA) questionnaires are predominantly used in psychology to assess a person’s emotional state. Positive and negative affect are not seen as opposites but as separate scales; in other words, you can have both high positive affect and high negative affect at the same time. Emotions related to high PA include high energy, enthusiasm, and focus whereas low PA is characterised by listlessness or lethargy. High NA may include anger, contempt, disgust, guilt, fear and nervousness, with low NA being a state of calm[1]. However, alongside its use as a standalone questionnaire, PANA questions are often used as part of a Subjective-Wellbeing (SWB) measure[2]. This is used in combination with a measure of Life Satisfaction, either overall or domain specific e.g., work, social life. A positive assessment of SWB includes high positive affect, low negative affect and a high rating of life satisfaction. This may be more familiar to those working in the cultural sector as a set of questions measuring subjective wellbeing have been developed by the Office for National Statistics (ONS); these are used in national datasets, including those on cultural participation like ‘Taking Part’. These datasets have been used to advocate for the ‘culture-wellbeing’ relationship i.e., that taking part in cultural activity is ‘good’ for you[3]. What are the risks and caveats of trying to measure sentiment? Whilst it may be possible to use SWB measures to demonstrate that cultural participation is good for you, this is not an objective of the Toolkit. Our objective is to help you think reflectively about the effectiveness of your work. Effectiveness may include the emotional response which people have and adding survey questions asking about a broad spectrum of emotions would bring in new data. However, whether that data is an accurate representation of those emotions is not a question that can easily be answered. In addition to the challenge of capturing accurate data on a person’s emotions, there is the further challenge of how that data should then be interpreted. We might be inclined to assume that our works caused whatever reactions we find, but it is not so straightforward. Here are some of the risks and caveats which one needs to consider when trying to measure and interpret sentiment: Context With the sentiment measure, we hope to move away from making value judgments about whether a score is ‘good’ or ‘bad’ but instead focus on what we can learn about people’s experiences. There shouldn’t be a hierarchy of emotion with ‘happiness’ at the top. Some work, due to its purpose or content, may not make people happier and nor should it intend to. There may be many reasons why a person is experiencing a particular emotion on a particular day. These may be nothing to do with their experience of your organisation or programme. They may be happy because they are out with friends they haven’t seen for a long time, or they may be stimulated because it’s their favourite artist and they’ve been waiting months to see the show. They may be stressed or upset because their train was delayed, or a family member has been taken ill, or the subject of the exhibition or performance triggered memories of traumatic experiences. The limits of a survey like those in the Toolkit means these contextual factors are impossible to capture in their entirety. We may know how people feel but you have no idea why, unless we ask additional free text questions. Timescale In psychological assessment, the PANA questionnaire is often given a particular timeframe e.g., ‘in this moment’, ‘today’, ‘in the past week’ with the idea being that this data is collected from the same person multiple times. This is not possible within the Toolkit survey, which is asked post event. Moreover, the diversity of events being evaluated by the Toolkit means that even this can be varied. Some responses are submitted in the exhibition space; others at home after a performance; others after a series of events. Some emotional states may be fleeting whereas others may come into effect after further reflection, and this may influence the consistency of the results produced. Causality Again, due to the nature of the post-event survey and without a pre-event baseline measure, we are unable to assume any causality that a particular emotion was caused by the experience under evaluation. As discussed above, there are many contingent factors that may cause a person to feel a certain way. This may be complicated further by the potential of reverse causality[4]. Take an example of ‘enthusiasm’ as a positive emotion. It is possible that the subset of the population who has made it to your event may be pre-disposed towards ‘enthusiasm’ compared to those who are pre-disposed towards ‘lethargy’. I’m aware this is a sweeping generalisation and there will be many other factors, but it’s something to be mindful of when we interpret our results. So how should you use sentiment measures? We’ve looked at the risks and what not to do, so now let’s look at what you can do. The way that we capture sentiment provides a snapshot of the audience’s emotional state. Whilst with this snapshot we can’t describe a causal link between the work being experienced and the feelings of the audience, there are a few things that we can use this data for: Interpretation of dimensions results – Above we discuss how a lack of context can cause us to make the wrong assumptions about our works. Whilst we don’t have that context for the person’s sentiment, the sentiment of the person answering the survey is important context for the dimensions questions. If we capture their sentiment (as well as their age, gender, prior experience etc.) we can better understand whether we meet our expectations. Tracking overall audience mood – When comparing individual people, their sentiment might fluctuate significantly due to the factors influencing their lives. However, if we take a larger group of people and average across their sentiment, the individual differences will cancel out and the trends will remain. If we do this measurement consistently and over time, we can develop an understanding of how audience mood changes over time and across types of work. Note caveat 2: Causality – we can’t infer that the work has caused the difference, but we can still learn from it. Creation or identification of more specific goals or achievements – This follows on from the two other uses highlighted above. If we can learn what the overall audience mood typically is for certain types of work, or that people with a specific sentiment tend to respond in a certain way to the dimensions questions, then we can set specific goals which push these boundaries. For example, if we learn that people with high negative affect tend to give lower levels of agreement for the Captivation dimension, then we might set out to attain high levels of agreement to Captivation from those people as a stretch goal. In addition to these points, capturing the sentiment of our audiences in a consistent fashion will enrich the overall dataset and provide opportunities for analysis and discovery in the future. This is the second blogpost in a series of three. Please see links below: Exploring ‘feelings’ prompted by arts and cultural experiences The uses and misuses of sentiment measurement for cultural experiences Considering frameworks and the selection of LE-PANAVA Main image: Photo by Hello I’m Nik on Unsplash [1] Watson, David., Lee Anna Clark and Auke Tellegen (1988) Development and Validation of Brief Measures of Positive and Negative Affect: The PANAS Scales. Journal of Personality and Social Psychology. Vol 54 (6), 1063-1070 [2] Diener, Ed (2000) Subjective Well-Being: The Science of Happiness and a Proposal for a National Index. American Psychologist. Vol 55 (1). 34-43 [3] See Susan Oman’s thorough analysis and explanation of the “culture-wellbeing” relationship in her open access book published in 2021 Understanding Well-being Data: Improving Social and Cultural Policy, Practice and Research. Palgrave Macmillan [4] See Fujiwara, Daniel (2013) Museums and Happiness: The Value of Participating in Museums and the Arts. United Kingdom: The Happy Museum; Museum of East Anglian Life; Arts Council England.