The following piece is based around some research that was carried out into the effect sound design has on an audience’s emotional perception on a film. This work was for a module at the University of Salford Audio Production masters course. All papers and research can be found in the references at the end of the piece.

Abstract

The role of music and soundtracks in film has been a heavily investigated topic; researchers have conducted a number of studies both of a qualitative and quantitative nature to determine the impact music and soundtracks have on the audiences perception and emotive response. The present paper aims to outline a proposed experiment into this field of study; in particular trying to answer the following question “Can audiences perceived emotions of a moving image piece be changed due to the accompanying soundtrack?”. The paper will give an outline of the importance of music and soundtracks in a film and its link to the perceived emotive response of the audience. It will also attempt to explain the methodological considerations for this type of research. Finally it will aim to conclude the findings and evaluate the topic and explore further work which could be carried out in the subject.

Introduction The following studies discussed in this paper will highlight key research into the way sound

A number of considerations can be made when we analyse the emotional response of an audience who have watched a motion picture film. We can consider things from both a symbolic and technical stand point, the choices made by the creator are perceived on an individual basis by its audience.

Sound, which can follow both of these codes, can be through the dialogue which allows us to determine how characters interact with another. Sound effects can give an inanimate objects life and music can help to strengthen the narrative. Through a combination of diegetic and non diegetic sounds, a scene comes to life. Through an intricate and creative process, inanimate objects, characters and creatures from history are brought to life through the process of foley. Soundtracks are embedded within the moving image to support, contradict and exaggerate and in some cases have become as iconic as the film itself.

Sound is an integral aspect in the process of making a motion picture and it is also a widely studied subject in regards to the impact the sound design of a film has on the audience.

1. Literature Review

 

The following studies discussed in this paper will highlight key research into the way sound design has an impact on the emotive response of the audiences who view films.

The importance of sound working in parallel with the moving image has been highlighted by a number of different studies into the subject. Fahlenbrach (2007) discusses the cognitive and bodily associations sound has on a viewer and a motion picture. In her paper she discusses and elaborates on what she believes are the “Three Dimensions of Emotional Experience and Emotional Communication” (2007). These are the innate schema, which is concerned with the neurological dimension of emotion. The second aspect of this theory is the associative scheme which deals with the level of the emotional response and finally she outlines the symbolic aspect of the emotions. This refers to the cultural and perceptive emotive codes and rules and how we associate symbolism with specific emotions. Fahlenbrach’s paper demonstrates that sound is more than just an afterthought and there are a number of considerations that producers of soundtracks can adopt in order to support and strengthen narrative within moving image productions. These emotive descriptors are

present in all film productions and forms the basis for consideration in the present paper. This is further demonstrated in Boltz (2004) paper “The Cognitive Processing of Film and Musical Soundtracks”. The paper discusses the cognitive affect musical scores have on a viewer, and discusses in part the way in which the musical soundtrack is remembered depending on whether or not the soundtrack is purposefully congruent or incongruent. The paper follows on from research carried out by Cohen (2004) and Pezdek and Stevens (1984) which again, looked into the relationship between music and moving image.

Two experiments were carried out to demonstrate the relationship between film and music and the recall of these two stimulus. It was concluded a link was made between the congruent music and the accompanying moving image, however recall was significantly lower in incongruent music. The paper would suggest that music that is used to support the action of a moving image is processed far greater than that of music which does not support the image. However it is suggested that further testing could be carried out as this does not always account for feature films which use incongruent or contrapuntal sound such as that of “A Clockwork Orange” or “Reservoir Dogs”. In these films, there is an argument that would

suggest that the scenes which are visually graphic in nature are somewhat more memorable. This is an important aspect to consider when looking into the emotive perception our audience has whilst watching a film. Although this paper will consider the emotive response through the pleonastic aspect of a soundtrack, there is further study that has already been identified through the idea of using music and soundtracks as counterpoint to the moving image.

Cohen (2001,2005) had conducted studies into the concept of congruence adopting the term “The Congruence-Associationist Model” or as it will from hereon be referred to as C-A- M. The idea of C-A-M is that it is supportive in the idea of us decoding meaning from a film not only from its two visual cues (scenes and text) but also from auditory cues (voice, sound effects and music). She also suggests that we can use the auditory levels to associate the action of the two visual levels.

For this particular study Cohen, Drew and MacMillan (2006) looked at the three auditory levels and their link to engagement (absorption), with each being prominent in two films. The study used two one minute clips from ‘Witness’ (P.Weir,1985) and ‘Jackal’ (F. Zinneman,1973) as stimulus to test

engagement and the link between auditory cues and absorption. Alongside the original soundtrack, additional soundtracks were created with a focus on music, sound effects or speech and an additional two soundtracks with a combination of the auditory cues. The study found that through self assessment, absorption was impacted depending on the music, sound effects or speech. What was also noted from the study is that even if the music had no connection to the narrative, it still has a significance on engagement.

This provides some understanding of the impact that auditory cues within a film production has in supporting engagement within a film.

This study demonstrates the importance and consideration producers have for not only the visual aspects of a motion picture but also the sound design within that film.

Further implications that sound has in motion pictures can be extended to consider the emotive impacts of soundtracks on an audience.

Cohen’s chapter “Music as a Source of Emotion in Film” (2001) discusses and argues for music being an integral part of the emotive response in motion picture. Highlighting the humble beginnings of the silent era of film and the need to mask the sound of the projector

and in response the somewhat neglectful response sound has had when considering narrative dramas.
Cohen highlights further the functions of sound in film and broke these down into eight considerations. From the more functional aspects (masking, supporting continuity and drawing our attention to important aspects) to the more creative and emotive responses it instills in us ( communication of meaning, integration into a film, leitmotiv- a form of association and finally the artistic quality of the soundtrack or music). These functions support the argument that as well as having functional purposes, there is clear support for the emotive impact a well written and considered soundtrack has when accompanying moving image.

1.1 Considerations

The aforementioned studies and literature presented so far in this paper support the purpose of the proposed study. Further studies highlighted will be discussed in the context of the methodology in support of this subject.

First of all we need to consider the topic of emotion and the recording of this in the form of research.

It would be expected from the themes of this paper that emotion is a personal thing that is differentiated between individuals. It could therefore be suggested that a qualitative method of obtaining results would be appropriate for this present study. The justification behind this is that by gaining an insight into someones emotions we would need to consider why these subjects have this emotional response. It would be possible to construct material which would incorporate previously mentioned criteria into the design of the test materials, which would allow for a more in-depth response giving reason why the choices of the participants have been made. However quantitative methods can also be considered to get a numerical understanding and would allow for mathematical analysis of the information gained from studies.

Parke,R., Chew, E., and Kyriakikas, C. (2007) conducted a quantitative study into perception and emotion of film and music. In their study they used the quantitative means to identify patterns between the two stimulus. Scatter plots were created to illustrate participants responses, using a method created by Osgood (1956) which plotted responses against axis’ activity (low or high) and stress (increased and decreased) and four quadrants (exuberant, anxious, content and depressed). Using a web interface for the

participants to respond, they were asked to plot how they felt after engaging with the test stimulus.
The conclusion from this study in regards to the type of research carried out is that a combination of the present quantitative method and qualitative response could be correlated and used to gain a further understanding of the participants choices.

A further consideration of the study design is the choice of stimulus that is to be used in this type of study.
All the studies highlighted so far have used different types of visual media in combination with soundtracks. In certain studies, simplistic visual stimulus has been used in comparison to others, which have opted to use more realistic and commercial based releases.

The reasons for these choices have all had their own identified justifications. In the study by Parke,R., Chew, E., and Kyriakikas, C. (2007), films were chosen because they were considered to be lesser known; the justification behind this decision was that there would be a minimised chance of the participants having any previous emotional connection or have any prior memories of the stimulus chosen.

In Bullerjahn,C. and Guldenring,M. (1994) study, a film entitled “The Joker” was

specifically created for the purpose of the study. Considerations for the film were highlighted included the the stimulus resembling a likeness to that of an actual feature film and the ambiguity of the scenes to help enhance the accompanying soundtracks and the emotive response that was being studied.

In contrast to these types of stimulus used in this particular field of research, Cohen (1993) in one of three experiments graduated the stimulus from a simple visual of a ball bouncing, to a more complex animation and finally two short video clips were used.

As well as the choice in visual stimulus, considerations need to be made in regards to the soundtracks used. All of the literature reviewed demonstrates the importance of the creation of soundtracks. Creating soundtracks for film is a particular skill and the composer(s) involved will need to think about the way their score impacts on the audience.

2. Methodology

The following section will discuss the proposed method which will be implemented to try and answer the question “Can audiences perceived emotions of a moving image piece be changed due to the accompanying soundtrack?”.

2.1 Stimulus

The previous section aimed to reflect on the previous body of work and what criteria needs to be formulated for this type of study. An explanation and justification of the methods used in this study will be discussed with the purpose of the study being carried out at a later date or act as a guide for other researchers.

2.1.1 Visual Stimulus

A clear thing that needs to be considered for this study is the type of visual media that is to be used in helping to support the question answered at the start of this paper.

Previous studies have shown that the visual stimulus used needs to be fit for purpose; this means that in order to support the auditory aspect of the study the scenes selected need to be neutral in what they are presenting. Certain cues such as props which have clear connotations are best to be avoided, things like scenes depicting violence with weapons could potentially persuade a participant; where as the purpose of this study like others that have been conducted is to study the viewers own perceptions. Providing a short film of ambiguous clips would allow for a clearer response of interpretation by the participants.

The source of these clips is also a determining factor in this type of study. Too simplistic and the viewer can become disengaged (Marshall and Cohen, 1988) and (Sirius and Clark 1994). Using existing commercial productions would allow for a more realistic viewing experience of that at home or in the cinema but finding clips from these sources would need to factor in things like popularity and prior knowledge of the participants.

Duration also plays a factor into the design of this study. Whereas some studies showed short clips of thirty seconds long, others ranged between 3-4 minutes in length like Andreas G. Anestis, Christos A. Gousios (2015) and others showed short films of 10 minutes in duration like that of Bullerjahn,C. and Guldenring,M. (1994) study.

The length of the film, would need to account for the soundtrack that is being used and should allow participants enough time to be able to interpret what it is they are watching and listening to.

2.1.2 Auditory Stimulus

As we would be testing the response to the audio that is accompanying the visual stimulus we would also need to think about what is selected.

Music that is readily available to us on a consumer level could cause issues in the study. Referring back to a previous section of this paper; considering the symbolic and cultural impact of a piece of music and also the memory recall of certain songs could askew and have a negative impact on the study. An example of this is cited in Bullerjahn’s 1994 paper of a study conducted by Brosisus and Kepplinger (1991) who chose to use the theme from James Bond’s “Goldfinger” as the stimulus for the experiment. Although compared to non familiar music, using a well known song (especially one that contains a leitmotif) can raise questions and issues when designing methodology for a test of this nature.

Genre is also a consideration of the choice in music; memory and cliches will act as signifiers for the recipients of the study and therefore can be used to compare and contrast against a casual stimulus.

For the purpose of this study, taking influence from previous works it would be feasible to have music specifically composed for the study.

2.1.3 Participants

Subjects involved in the study can vary depending on what it is the experiment is setting out to achieve. Demographic factors including age, gender and location can be

considered to make quantitative comparisons between participants involved in the study. From previous studies the volume of participants has varied. Cohen’s (1993) experiment used only 12 subjects in comparison to that of Bullerjahn,C. and Guldenring,M (1994) which used 412 participants in their study. As this study is concerned with the emotional impact on audience members individuals with no prior knowledge or expertise in listening tests are not of a concern in this type of experiment.

3. Design

Following on from the considerations of the previous section, this part will aim to discuss the overall design of the proposed experiment.

A short film created for the purpose of this study would be used in order to alleviate and minimise the symbolic and memory based emotional criteria highlighted in previous studies. The importance of ambiguity in the shots would also need to be considered so as not to influence the viewers response and potentially disregard the soundtrack that is accompanying the moving image. This would need to be created with an emphasis on the overall aesthetic of the film. The need for the film to look as professional as possible would

allow for some replication and comparison of a professional output.
Auditory stimulus would need to be created in conjunction with a professional composer or composers. Original film scores can and have been used in similar experiments however considerations again need to be made for things such as expectancy, memory recall and symbolic nature of the score. The score(s) for the purpose of this type of design would need to have an emphasis on congruence in that they are used to support the moving image piece.

In terms of the sample it would like in all research be beneficial to have a sample that is representative of the population. However previous studies have demonstrated the use of a convenience sample primarily based around students in particular departments (film, music and media/psychology and musicology departments) although this does not represent and cannot necessarily be generalised to the wider population the convenience of this allows for a controlled and easily accessible method of gathering information. It could also be argued that the age demographic of these students falls in line with research carried out by the BFI which in a 2018 report found that the admission for film viewing in cinemas was highest in the 15-24 age category at 27.8%.

Participants will be required to fill out a questionnaire containing closed demographic questions (if results are wanted to be quantified and compared between participants) and open ended questions.

User based interfaces could potentially be designed for this type of study like that in Chew.E, Kryakakis.C, Parke.R. study but it would also suffice to use apparatus typical in most rooms that have a computer and speakers or headphones.

Staggering the questionnaire would also support the ambiguity of the project and minimise bias in the study.

4. Discussion

In designing this study there are a number of apparent things that can be highlighted from these types of work.
There has been a number of studies into this field of work despite earlier studies highlighting the lack of consideration to soundtrack and the motion picture.

The present study identified in this paper, looks to demonstrate how emotive response and perception can be changed when viewing a particular film with different soundtracks. The type of methodology outlined in this paper demonstrates the way in which this can be tested and experimented on although there

are considerations which need to be highlighted in this type of study.

There are clear variations in the stimulus material; lengths between minimal 30 second clips to a longer 10 minute short film would suggest that there is further research which could be carried out as these durations would suggest that although investigation is present into film and soundtrack, a lot of the research has looked into scenes and snapshots of films not a film as a whole entity.

The type of film that is being used as stimulus has to thought about, ambiguity of clips is a common theme in the work presented. Simplistic visuals like that of animated shapes or inanimate objects is not always considered to be suitable in the this type of work as we are trying to determine a link between the two stimulus. Having that visual connection between the two stimuli strengthens this type of study.

Congruent soundtracks have had a heavy focus on the this field of research, the supporting aspect of music in film is widely known and considered. Although congruent sound is integral to films there is further investigation to be he had in this field with other types of sound used in film.

The implications of the music used in these type of experiments is important to consider. Genres of film like music are clearly identifiable, which leads to the question being asked about changing the perception of the genre of a film through its soundtrack. The materials that need to be created need to be planned, considered and deemed fit for purpose also.

5. Conclusion

The field of emotion is a complex topic which in conjunction with film, music and personal circumstances crosses a number of different disciplines.

Further study in this field could reach out to determining the difference emotive response to a film which has the characteristics of one genre but the music is used to counterpoint the moving image on the piece.

As well as the source material being focused on the music of a film, there is further study into more specific sound design of the foley and sound effects that are used in films. If the expected sounds of a scene are changed can this also change the way in which perceived emotions are recorded?

Qualitative and quantitative results are important to determine the results of these types of studies. With the high level of

accessibility of technology, these types of studies can be implemented in a number of different settings. Simpler controlled ways of executing these type of studies to more advanced web based implementations makes this type of research flexible.

 

 

 

 

 

 

Bibliography

Anestis, A. G., & Gousios, C. A. (2015). How Cinema Sounds Affect the Perception of a Motion Picture. Universal Journal of Psychology, 3(5), 147-152.

BFI (2018). Statistical Yearbook. Accessed from https://www.bfi.org.uk/sites/bfi.org.uk/files/ downloads/bfi-statistical-yearbook-audiences-2018-12-17.pdf

Blumstein, D. T., Davitian, R., & Kaye, P. D. (2010). Do film soundtracks contain nonlinear analogues to influence emotion?. Biology letters, 6(6), 751-754.

Bolivar, V. J., Cohen, A. J., & Fentress, J. C. (1994). Semantic and formal congruency in music and motion pictures: Effects on the interpretation of visual action. Psychomusicology: A Journal of Research in Music Cognition, 13(1-2), 28.

Boltz, M. G. (2004). The cognitive processing of film and musical soundtracks. Memory & Cognition, 32(7), 1194-1205.

Boltz, M.G., 2001. Musical soundtracks as a schematic influence on the cognitive processing of filmed events. Music Perception: An Interdisciplinary Journal, 18(4), pp.427-454.

Bullerjahn, C., & Güldenring, M. (1994). An empirical investigation of effects of film music using qualitative content analysis. Psychomusicology: A Journal of Research in Music Cognition, 13(1-2), 99.

Cohen, A. J. (1993). Associationism and musical soundtrack phenomena. Contemporary music review, 9(1-2), 163-178.

Cohen, A. J., MacMillan, K., & Drew, R. (2006). The role of music, sound effects & speech on absorption in a film: The congruence-associationist model of media cognition. Canadian acoustics, 34(3), 40-41.

Cohen, A.J., 2001. Music as a source of emotion in film. Music and emotion: Theory and research, pp. 249-272.

Collins, K. (2009, February). Generating Meaningful Sound: Quantifying the Affective Attributes of Sound Effects for Real-Time Sound Synthesis in Audio-Visual Media. In Audio Engineering Society Conference: 35th International Conference: Audio for Games. Audio Engineering Society.

Fahlenbrach, K. (2008). Emotions in sound: Audiovisual metaphors in the sound design of narrative films.

Krosnick, J. A. (2018). Questionnaire design. In The Palgrave Handbook of Survey Research (pp. 439-455). Palgrave Macmillan, Cham.

Lipscomb, S. D., & Kendall, R. A. (1994). Perceptual judgement of the relationship between musical and visual components in film. Psychomusicology: A Journal of Research in Music Cognition, 13(1-2), 60.

Lipscomb, S. D., & Tolchinsky, D. E. (2005). The role of music communication in cinema. Musical communication, 383-404.

Parke, R., Chew, E., & Kyriakakis, C. (2007). Quantitative and visual analysis of the impact of music on perceived emotion of film. Computers in Entertainment (CIE), 5(3), 5.