Conjuring Cross-Modal Perception

By embracing video devoid of audio tracks it may be possible to engage with a uniquely personal and internal sound world, as Cage asserted in a discussion of the nature of sound and silence: ‘It is inextricably synchronous with all other, sounds, non-sounds, which latter, received by other sets than the ear, operate in the same manner’ (Cage & Gann, 2011, p. 14).

Human sensory perceptual experience arrives via external stimulation of the sensory organs which can be organised into several modes. The body is frequently subjected to multiple modal experience and research suggests that perceptual experience may not be isolated to one particular mode (McGurk & MacDonald, 1976; Krumhansl & Schenck, 1997; Berger, 2002; Guttman, Gilroy, & Blake, 2005; Bulkin & Groh, 2006; Skipper, van Wassenhove, Nusbaum, & Small, 2007; McAuley & Henry, 2010; Man, Kaplan, Damasio & Meyer, 2012; O’Callaghan, 2015; Vilhauer, 2015; Su & Salazar-López, 2016). Cross-modal perception offers an explanation as to why humans sometimes report experiential phenomenon with the features of another sensory experience, such as having a sensation of sound when viewing images. I first became aware of the potential for video to conjure cross-modal perception and non-cochlear sound following a performance and subsequent group discussion of my piece Syncretism, which was not intended to create cross-modal interactions. I followed this up with reading into ventriloquism, research into visual dominance over auditory stimulus in the McGurk effect and composers who intended to specifically engage with a non-cochlear sound.

Étude was the first piece in the portfolio to attempt the generation of a non-cochlear sound using video. My approach drew on techniques from experimental films of the early twentieth century and approaches to composing pieces of visual music, such as the use of montage to form large structural schemes (Evans, 1992; Turvey, 2003; Evans, 2005; Fuxjäger, 2012; Keefer, 2015). New visual rhythms were created from recorded footage by altering natural hand movements using interventional video editing techniques similar to those used by Arnold and De Bemels. Drawing on Eisenstein’s (1949) concept of metric montage, which is the arrangement of frames of the film within the shot (Eisenstein, 1949; Evans, 2005), the use of hard cutting, slow motion and extreme repetition at the local level resulted in an amplification and exaggeration of movement (Benjamin, 2005; Koupil & Vícha, 2011).

My approach to Étude did not connect directly with research into cross-modal perception which I later engaged with for Apprentice. However, I did draw heavily on experimental film pieces of the early twentieth century which focussed on manipulating light and various optical media as an abstraction of the treatment of sound by composers without generating non-cochlear sound (Turvey, 2003; Evans, 2005; Ox & Keefer, 2008; Alves, 2012; Fox-Gieg, Keefer, & Schedel, 2012; Keefer, 2015; Mollaghan, 2015). Although, McLaren’s Synchromy reportedly featured ‘pseudo-/culturally synaesthetic associations’ (Mollaghan, 2015, p. 12). Étude also drew on several pieces of dancefilm such as Rainer’s Hand Movie, and Robert’s Hands which were highly influential on the composition of the previous piece Syncretism. Even though the approach used in Étude may not produce non-cochlear sound the purposeful, musical arrangement and structuring of the moving hand in the visual domain should be evident (Krumhansl & Schenck, 1997; Su & Salazar-López, 2016).

That is not to say that experimental film and visual music pieces, as well as my own work, which organised moving imagery based on the techniques more readily associated with sound-based media would not generate any sort of non-cochlear sound. Some evidence suggests that, although it may not be automatic (McAuley & Henry, 2010), rhythmic patterns of light and video may be able to trigger forms of cross-modal perception (McGurk & MacDonald, 1976; Berger, 2002; Guttman, Gilroy & Blake, 2005; Skipper, van Wassenhove, Nusbaum & Small, 2007). Of course, works which do not attempt to generate a non-cochlear sound should, for the sake of the work, be considered within the realms for which they were intended (Cox, 2005; Shaw-Miller, 2014). Furthermore, it could be argued that moving imagery which depicted intentional and known bodily motions, such as playing the guitar, may initiate a form of kinesthetic empathy in a viewer (Fischman, 2011; Strukus, 2011), rather than a sense of sound. It may also be equally likely that a viewer would experience only the visual aspects of the work itself, such as the visual content and structure, rather than experiencing any sort of cross-modal perception (McAuley & Henry, 2010).

Apprentice adhered more closely to the research into cross-modal perception which studied rhythm perception in sound and light using very simple visual imagery (Guttman, Gilroy, & Blake, 2005; McAuley & Henry, 2010). My intention was to compose a piece of music as my final piece for the portfolio which would generate non-cochlear sound through visual stimulus. I attempted to engage with flashing images of hands in various poses associated with musical instruments to create rhythmic pulses but several issues remained with the approach.

Discussing the use of notation in musical practice could reveal why my pieces may fail to conjure non-cochlear sound. Just as one may be able to create an internal representation of written language while reading, which can be experienced in Ablinger’s Weiss/Weisslich 11B and Pisaro’s Braids : for (silent) reader (Gottschalk, 2016), musicians with a proficiency in reading musical notation may be able to create a ‘visualisation’ (Gieseking & Leimer, 1972, p. 18), an ‘audiation’ (Gordon, 1994, p. 39), or mental representation of a piece of notated music. Although applied specifically to the discussion of reading musical notation, Gordon (1994) asserts audiation as ‘the process that takes place when we hear and comprehend music for which the sound is no longer physically present’ (Gordon, 1994, pp. 39-40), an ability which can also be applied to non-reading musicians (Ginsborg, 2004; Duckworth, 2009; Gennet, 2016). These approaches are bound by the understanding of a fundamentally societally agreed upon set of rules of language and literature (Gordon, 1994), to which Apprentice may be ascribed through the representation of instrumentation but Étude may not.
© 2016 James A. Fox Contact Me