Why do we listen to music? While there are several answers to this question, one common answer is that listening to music provides us with an emotionally rich experience. If music is rich with emotional content, would it be possible to predict a listener’s emotional assessment of a specific song? Additionally, to what degree is this assessment of emotion common across listeners with different musical interests and backgrounds, and to what degree is it subjective to each listener? Gaining a better understanding of this relationship between music and the listener, in the context of emotion, is an important topic in the domain of music composition and music recommendation applications. A good understanding of this relationship would enable music software applications to be customized for different types of listeners. Emotion assessments are collected either as predetermined categories (e.g. happy, sad, peaceful, angry for a music excerpt) or as ratings on a continuous scale (e.g. calm-to-excited, unpleasant-to-pleasant).
Photo Source: www.brainpickings.org (Maria Popova)
Studies have shown that it is possible to successfully predict emotion assessments using computational modelling methods. These approaches involve gathering ratings from several listeners for large datasets of music samples, and applying machine learning algorithms to train predictive models on these labeled ratings using features. The features are a combination of low-, mid-, and high-level audio features (spectral and temporal) and music features (tonal and rhythmic). Previous work in the SMART lab involved using neural networks based on audio features to successfully predict emotion ratings of listeners for a small dataset of 12 classical music excerpts. While this approach is valuable for a general listener, it has its limits with respect to each specific listener.
Every listener brings his or her own cultural background, preferences, and familiarity to the music listening experience. In some cases, your sad could be my happy. One of the ways to hone in on a listener’s subjective emotional experience is through his or her physiological responses. Studies have shown that listeners experience physiological changes during music listening which parallel the kinds of physiological changes they experience with everyday emotion. This allows us to pursue the relationship between the types of physiological responses experienced by a listener and the reported emotion induced by the music they are listening to. An interesting question to consider is whether a listener's subjective assessments of felt emotion can be predicted from physiological responses occurring during music listening. In a paper we published in Frontiers in Psychology, neural networks were used to predict emotional responses of listeners based on five physiological features. Results from this paper suggest that physiological responses are powerful indicators of a listener’s emotional state, and a non-linear relationship may be used to map musical emotion onto physiological responses.
Further studies are currently in progress between the SMART lab and WaveDNA, a music software development company in Toronto, to explore and answer interesting questions at the intersection of musical emotion, musical features, listener physiological responses, and listener experience.
Written by: Naresh Vempala