Appraisal-based Emotion Analysis: An Overview
Emotion analysis and affective computing
Some important information that people share with each other are facts, like “She rides the bicycle” or “The election is over”. Such factual statements can also describe private states of a person, which cannot be directly observed, for instance “I am sad.”. The task of emotion analysis as an area of AI belongs to a broader research field called affective computing (Picard 2000). The goal is to enable computers to understand which emotion somebody expresses, or, more generally, which emotional state somebody is in. It also includes tasks such as understanding what the emotion is that a text might cause in a person, but we focus here on understanding the emotional state of a person based on observing their behaviour.
This can be done based on images of faces, automatically interpreting body postures, or the sound of a voice. Our research focuses, however, on interpreting text. We want to understand which emotion a real or a fictional person might be in, purely based on text, that describes what the person says, does, or how they interact. The challenge is, however, that not every description of an emotional private state is as explicit as the sentence mentioned above: “I am sad.” In everyday communication, we don’t need to state our emotion - when I tell somebody “I’ve been at a funeral” most people will understand that there is a reason to be sad. When I say “I just booked my next holidays”, people know that I am looking forward to something good, and I probably feel happy about that. And when I say: “I don’t know what will happen”, the feeling of insecurity that is expressed is an indicator for being afraid of something. Emotion analysis from text is challenging, in particular in cases in which the emotion is not mentioned (Casel, Heindl, and Klinger 2021).
Why is emotion analysis from text relevant?
Having robust systems that interpret emotions from text have a plethora of applications. Chatbots in customer care can, for instance, act empathetically when they understand if the person they talk to is angry or sad. Robots or intelligent agents could decide to adapt the way they speak or what to say. Large amounts of text can automatically be analyzed, for instance to see in social media a change in the general emotion of some part of a population - for instance how the emotions regarding the COVID virus changed throughout the years of the pandemic. Literary scholars could analyze the relationship between characters and observe similarities between books - for instance quantify how the complicated relation between Lotte, Werther, and Albert motivated other authors for similar settings (the so-called Wertheriaden).
Appraisal theories explain the link between the cognitive evaluation of events and the emotion
Emotion analysis systems build on top of emotion psychology. They use the set of possible emotions to be analyzed, for instance Ekman’s set of basic emotions (Anger, Fear, Joy, Disgust, Surprise, Sadness) or the set proposed by Plutchik (adding Anticipation and Trust). Predefining a set of emotions an automatic system should be able to work with is often fine, but sometimes it’s hard to come up with a complete set. That problem is, to some degree, solved by not just building a text classification system that assigns an emotion label to a text, but instead assigns values of valence and arousal. These concepts are continuous dimensions which are in fact what we perceive; how much we are activated and how much we perceive something to be positive or negative. The combination of the two values leads to a feeling that we often give a name: the emotion.
Appraisal theories go one step further: they were studied by psychologists particularly to understand how events are interpreted by people participating in them, and how that leads to an emotion. For instance, if somebody feels insecure, it is very unlikely the person is happy about it. If something happens that supports the goals of a person, that likely makes them happy. These links between event evaluations and emotions are not perfect - not every situation in which we don’t really know what’s going on makes us anxious, but there are correlations between the concepts of appraisals and emotions that explain the link between events and perception quite well. Smith and Ellsworth (1985) showed that a comparably small set of such appraisal dimensions explain a larger set of emotions. Scherer, Schorr, and Johnstone (2001) observed that the evaluation of appraisals takes place sequentially. Scherer is one of the key players in psychological research on emotions appraisals, and in fact he proposed that these concepts need to be considered in computer science (Scherer 2009).
Appraisal theories motivated some early work on automatic emotion analysis
The need for implementing emotion recognition systems with the help of appraisal theories has been recognized already a while ago. Balahur, Hermida, and Montoyo (2012) developed EmotiNet, a database that dissects event action chains and links them to a particular emotion. Similarly, SenticNet lists events that lead to particular emotions.
Some other work made more direct use of appraisal concepts. The work by Shaikh, Prendinger, and Ishizuka (2009) and by Udochukwu and He (2015) build on top of the OCC model (Steunebrink, Dastani, and Meyer 2009), another appraisal model, which propose pretty clear rules how appraisals work together and lead to emotions. Both papers mentioned above make use of these rules to interpret events. They show that this works quite well for implicit expressions of emotions.
Our work on appraisal-based emotion analysis
Still, there was no work that made the appraisal concept in text explicit. All previous work was inspired by appraisal theories, and they made use of them to interpret text and output an emotion, but they did not output appraisal values. They could do that, but then we would not know how well that prediction works. To really understand what appraisal theories can do for textual emotion analysis, we needed a text corpus annotated with appraisal concepts.
We started our research in this direction with a study (Hofmann et al. 2020) in which we asked three people to annotate existing event descriptions (Troiano, Padó, and Klinger 2019) with appraisal concepts. That worked quite well, the annotators agreed with each other pretty well, but not always. Automatic systems that predict the appraisal dimensions also worked quite well. In a follow-up study, we showed that manual annotation might not even be required, because we could exploit the knowledge about correlations between emotions and appraisals (Hofmann, Troiano, and Klinger 2021).
The main disadvantage here was: We had no clue if the annotated appraisal concepts really described the event evaluation of the people who lived through the event. We needed another study in which we asked people to tell us about events that caused emotions in them and to tell us about their emotion and the appraisal. At the same time, we extended the comparably limited set to many more appraisal dimensions, so we started with a smaller annotation project to understand the challenges (Troiano et al. 2022). This study includes a couple of other interesting aspects regarding how events and people interact, but that is something I will describe in another blog post later. For now, I will summarize our most recent study on appraisal theories and text analysis (Troiano, Oberländer, and Klinger 2023).
This study can be intuitively summarized with the following depiction.
We asked people online to describe an emotional event for a specific emotion and then assign appraisal dimensions, how they perceived them in the respective situation. We then asked other people to read the texts and reconstruct the emotion and appraisal - without knowing anything about them, and without further context. Unsurprisingly, the readers sometimes misinterpreted an event. For instance the text “I put together a funeral service for my Aunt” is mostly interpreted as something sad, while the original author was actually proud about it. These differences in interpretation can also be seen in the appraisal variables - appraisals explain the differences in the event evaluation! The interpretations as being sad come with event evaluations as not being in control, while the interpretation as an event that made the person proud come with a perception of being in control.
The question is, of course, if that also helps automatic assignment of emotion labels. While appraisal dimensions can be predicted from text similarly good as humans can do the job (our two-phase data annotation project allowed exactly such analysis), appraisals do not help in all emotions, but in some they do. Particularly guilt, sadness, and anger can be predicted with the help of appraisals with better accuracy than without.
My favorite text example from the data we collected (which is, at the same time, the one that I find most … well, see for yourself) is: “His toenails were massive.” This text has been generated for the emotion disgust. All annotators (and perhaps you as well) agree that it’s disgusting, despite the problem that there is nothing known about the context. Our text classifier that does not know about appraisals, however, does not agree: It interprets the word “massive” as something good, and finds the emotion “pride” to be most appropriate. Only when we include appraisal concepts in our classification system, the model ‘understands’ that it’s not a pleasant event, that somebody else is not responsible, and that “disgust” is therefore a better output.
Summary
I discussed in this blog post our work on emotion analysis with appraisal theories.
Why is this important? Appraisal theories from psychology explain the link between a cognitive evaluation of an event and the emotion that somebody develops. In text analysis, they support the automatic interpretation of emotions in implicit expression, such as event descriptions. The data we collected is also the first data annotated with a large amount of appraisal dimensions.
What comes next? Appraisals explain one evaluation of an event, not many. Emotions are subjective experiences, and evaluations of events are based on previous experiences. We need to better understand which factors we need to implement in computer models to make more accurate and fitting predictions, based on other aspects. For instance, and old person might interpret an event differently and develop an emotion differently than a younger person. Somebody in the US might be more used to some event than somebody in Europe, and therefore better know how to deal with it. An important next question is: how do such demographic variables impact the emotion classification?
Further, the research area of emotion role labeling focuses on dissecting emotional events into their causes, targets, and emotion experiencers. This area handles emotions as events - while our work described above focuses on events as causes to emotions. These two fields need to be put together, they can very likely benefit from each other and provide a more holistic understanding how emotions are described in text.
Acknowledgements
I would like to thank all coauthors who contributed to our work on emotion analysis with the help of appraisal theories. These are (in no specific order) Laura Oberländer, Enrica Troiano, Kai Sassenberg, Jan Hofmann, Valentino Sabbatino, Max Wegge, Felix Casel, Amelie Heindl, Tornike Tsereteli,
This work has been funded by the German Research Council (DFG) in the project CEAT, KL 2869/1-2.
Bibliography
Balahur, Alexandra, Jesus M. Hermida, and Andrew Montoyo. 2012. “Building and Exploiting EmotiNet, a Knowledge Base for Emotion Detection Based on the Appraisal Theory Model.” IEEE Transactions on Affective Computing 3 (1): 88–101. https://doi.org/10.1109/T-AFFC.2011.33.
Casel, Felix, Amelie Heindl, and Roman Klinger. 2021. “Emotion Recognition Under Consideration of the Emotion Component Process Model.” In KONVENS 2021. https://aclanthology.org/2021.konvens-1.5.
Hofmann, Jan, Enrica Troiano, and Roman Klinger. 2021. “Emotion-Aware, Emotion-Agnostic, or Automatic: Corpus Creation Strategies to Obtain Cognitive Event Appraisal Annotations.” In Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, 160–70. Online: Association for Computational Linguistics. https://aclanthology.org/2021.wassa-1.17.
Hofmann, Jan, Enrica Troiano, Kai Sassenberg, and Roman Klinger. 2020. “Appraisal Theories for Emotion Classification in Text.” In Proceedings of the 28th International Conference on Computational Linguistics, 125–38. Barcelona, Spain (Online): International Committee on Computational Linguistics. https://doi.org/10.18653/v1/2020.coling-main.11.
Picard, Rosalind W. 2000. Affective Computing. The MIT Press. https://mitpress.mit.edu/9780262661157/affective-computing/.
Scherer, Klaus R. 2009. “Emotions Are Emergent Processes: They Require a Dynamic Computational Architecture.” Philos. Trans. R. Soc. Lond. B Biol. Sci. 364 (1535): 3459–74.
Scherer, Klaus R, A Schorr, and T Johnstone. 2001. Appraisal Considered as a Process of Multi-Level Sequential Checking. Appraisal Processes in Emotion: Theory, Methods, Research. Vol. 92. Oxford University Press.
Shaikh, Mostafa Al Masum, Helmut Prendinger, and Mitsuru Ishizuka. 2009. “A Linguistic Interpretation of the OCC Emotion Model for Affect Sensing from Text.” Edited by Jianhua Tao and Tieniu Tan. Affective Information Processing. https://doi.org/10.1007/978-1-84800-306-4_4.
Smith, Craig. A., and Phoebe. C. Ellsworth. 1985. “Patterns of Cognitive Appraisal in Emotion.” Journal of Personality and Social Psychology 48 (4). https://www.researchgate.net/publication/19274815_Patterns_of_Cognitive_Appraisal_in_Emotion.
Steunebrink, Bas R., Mehdi Dastani, and John-Jules Ch. Meyer. 2009. “The OCC Model Revisited.” Online: https://people.idsia.ch/~steunebrink/Publications/KI09_OCC_revisited.pdf.
Troiano, Enrica, Laura Ana Maria Oberlaender, Maximilian Wegge, and Roman Klinger. 2022. “X-EnVENT: A Corpus of Event Descriptions with Experiencer-Specific Emotion and Appraisal Annotations.” In Proceedings of the Thirteenth Language Resources and Evaluation Conference, 1365–75. Marseille, France: European Language Resources Association. https://aclanthology.org/2022.lrec-1.146.
Troiano, Enrica, Laura Oberländer, and Roman Klinger. 2023. “Dimensional Modeling of Emotions in Text with Appraisal Theories: Corpus Creation, Annotation Reliability, and Prediction.” Computational Linguistics 49 (1). https://doi.org/10.1162/coli_a_00461.
Troiano, Enrica, Sebastian Padó, and Roman Klinger. 2019. “Crowdsourcing and Validating Event-Focused Emotion Corpora for German and English.” In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4005–11. Florence, Italy: Association for Computational Linguistics. https://doi.org/10.18653/v1/P19-1391.
Udochukwu, Orizu, and Yulan He. 2015. “A Rule-Based Approach to Implicit Emotion Detection in Text.” In Natural Language Processing and Information Systems.