Gesture in Interactive Electroacoustic Music: the Role of the Body in Computer-Based Performance

One of the central challenges in interactive electroacoustic music concerns the role of gesture in performance. In acoustic instrumental practice, gesture is inseparable from sound production: the physical actions required to generate sound are visible, legible, and structurally meaningful. In contrast, computer-based music performance often obscures the relationship between movement and sonic outcome. This raises an important question:

How can gesture remain musically meaningful when sound is mediated through digital systems?

The problem of invisibility in computer-based performance

Acoustic instruments provide an immediate and intelligible relationship between action and sound. By contrast, laptop-based performance environments frequently conceal this relationship. While small-scale gestures such as clicking, typing, or trackpad movement may generate complex sonic transformations, they are often difficult for audiences to interpret.

This perceptual disconnect has implications for how performance is experienced.

Interactive electroacoustic systems provide one possible response. Technologies such as motion sensors, camera-based tracking, and custom-designed controllers allow performers to establish clearer relationships between bodily movement and sonic transformation. Through these systems, gesture can be enlarged, made continuous rather than discrete, and re-integrated into the perceptual field of performance.

Transferable instrumental gestures

Traditional instrumental performance requires extensive practice before expressive control can be achieved. Instrumental technique develops gradually through repeated physical interaction with a specific instrument.

In interactive systems, however, gesture tracking can be separated from sound production. As a result, instrumental gestures become transferable across different sonic environments. A single keyboard controller, for example, can be used to perform the sounds of multiple instruments through the same physical gesture vocabulary.

This separation represents an important shift in musical interaction. Rather than adapting technique to the physical constraints of a single instrument, performers can apply familiar gestures across multiple sound worlds through configurable mapping systems.

Non-instrumental gestures applied in music

Once gesture tracking and sound production are separated, composers and system designers gain greater freedom in defining gesture as an input to musical interaction.

In addition to traditional instrumental gestures, a wide range of everyday bodily actions can function as musical input. Typing on a computer keyboard, hand-waving, baton gestures, and even sign language movements can be incorporated into digital musical performance systems. Advances in motion-tracking technology have also enabled dance movements to function as gestural input within integrated audiovisual performance environments.

One early example is the Digital Dance Project (1998) at the Danish Institute of Electroacoustic Music, which explored the integration of movement tracking within interactive dance performance (Siegel & Jacobsen, 1998).

Another important development is the adoption of commercial gestural controllers in electroacoustic music practice. Devices originally designed for non-musical purposes are frequently adapted for performance through hardware modification or custom software development. This practice reflects a broader shift toward flexible and configurable interface design within digital musical instruments.

Facilitating of newly invented gestures

The separation of gesture tracking from sound production also enables the invention of entirely new gestural vocabularies.

Examples include Max Mathews’s Radio Drum and Radio Baton, as well as Michel Waisvisz’s The Hands and The Web, which introduced new interaction paradigms specifically designed for electronic performance environments. Although the Theremin was originally hard-wired to its sound-generation circuitry, later MIDI implementations allowed it to function explicitly as a controller interface.

Another important development in this area is the emergence of smart garments, or electronic textiles (e-textiles), as musical interfaces. Rooted in the belief that “the body is the most familiar of all communicative modes” (Marvin, 1990, p.109) and the assumption that “the most profound technologies are those that disappear” (Weiser, 1991), e-textiles enable continuous data collection from both performers and their environments (Toney et al., 2006).

Projects such as Drum Pants (Hansen & Jensenius, 2006) demonstrate how wearable technologies can incorporate sophisticated bodily gestures directly into musical performance systems.

The Radio Baton by Max Matthews

Waisvisz with first version of The Hands

Controllers and the emergence of digital musical instruments

The modern concept of the controller became clearly defined with the introduction of the Musical Instrument Digital Interface (MIDI) standard in the early 1980s. MIDI was developed to allow synthesizers from different manufacturers to communicate with one another and to enable performers to control multiple instruments simultaneously.

This development formalised the separation between gesture and sound generation.

Keyboard controllers quickly became widespread because they allowed performers to input equal-tempered pitch structures polyphonically using a familiar technique already shared by many musicians. As a result, keyboard interfaces became one of the most accessible and widely adopted controller formats in early digital synthesis environments.

Miranda and Wanderley define the controller within a digital musical instrument (DMI) as:

“The gestural controller (or control surface) is the device that provides the inputs to the DMI. It is where physical interactions between the performer and the instrument take place.”(Miranda & Wanderley, 2006, p.3)

Since then, the term gestural controller has become widely used within electroacoustic music literature to describe interfaces that mediate performer interaction with computational sound systems.

The Drum Pants (Hansen & Jensenius, 2006)

RCA synthesizer by Harry Olson in 1955 (courtesy of Tom Rhea)

Gesture as compositional material

In acoustic instrumental practice, gesture emerges through technique developed over long performance traditions. In interactive electroacoustic music, gesture itself can become a site of composition.

Composers working with interactive systems are able to shape:

• the scale of movement required to generate sound,

• the temporal responsiveness of interaction,

• the relationship between effort and sonic result,

• the visibility of performer action,

• and the degree of determinacy or unpredictability within interaction.

Gesture is therefore not only a means of realising musical ideas, but part of the compositional structure through which those ideas are articulated.

This shift encourages a reconceptualisation of the instrument—not as a fixed object, but as a dynamic relationship between body, interface, and computational process.

Gesture and the listening experience

Gesture also plays an important role in shaping audience perception. Listening in interactive performance contexts involves both sonic interpretation and visual inference. When gestural relationships are perceptually coherent, audiences can follow the unfolding structure of a performance through both sound and movement. When these relationships are obscured, performance may instead be experienced as technologically mediated or abstract.

Designing interactive electroacoustic works therefore involves composing not only sound, but also the perceptual conditions through which sound is understood.

For this reason, gesture continues to occupy a central position in my compositional and installation-based practice. Across performance systems, camera-based interaction, and responsive audiovisual environments, gesture provides a means of maintaining a meaningful relationship between bodily presence and computational sound processes.

Interactive electroacoustic music does not eliminate the performer’s body. Rather, it invites us to reconsider how bodily action can operate as a primary site of musical expression within technologically mediated environments.

Chadabe, J. (1997) Electric Sound: The Past and Promise of Electronic Music. Upper Saddle River, NJ: Prentice Hall.

Cadoz, C. and Wanderley, M. (2000) Gesture–Music. In: Wanderley, M. and Battier, M. (eds.) Trends in Gestural Control of Music. Paris: IRCAM – Centre Pompidou, pp. 71–94.

Drummond, J. (2009) Understanding interactive systems. Organised Sound, 14(2), pp. 124–133.

Godøy, R. and Leman, M. (eds.) (2010) Musical Gestures: Sound, Movement, and Meaning. New York: Routledge.

Hunt, A., Wanderley, M. and Paradis, M. (2003) The importance of parameter mapping in electronic instrument design. Journal of New Music Research, 32(4), pp. 429–440.

Jordà, S. (2005) Digital instruments and players: part I – efficiency and apprenticeship. In: Proceedings of the International Conference on New Interfaces for Musical Expression (NIME).

Leman, M. (2008) Embodied Music Cognition and Mediation Technology. Cambridge, MA: MIT Press.

Magnusson, T. (2009) Of epistemic tools: musical instruments as cognitive extensions. Organised Sound, 14(2), pp. 168–176.

Miranda, E. and Wanderley, M. (2006) New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. Middleton, WI: A-R Editions.

Rowe, R. (1993) Interactive Music Systems: Machine Listening and Composing. Cambridge, MA: MIT Press.

Tanaka, A. (2000) Musical performance practice on sensor-based instruments. In: Wanderley, M. and Battier, M. (eds.) Trends in Gestural Control of Music. Paris: IRCAM – Centre Pompidou, pp. 389–405.

Van Nort, D. (2010) Instrumental listening: sonic gesture as design principle. Organised Sound, 15(2), pp. 177–187.

Wanderley, M. (2001) Performing digital music: a question of control? In: Proceedings of the COST G-6 Conference on Digital Audio Effects (DAFx).

Wanderley, M. and Depalle, P. (2004) Gestural control of sound synthesis. Proceedings of the IEEE, 92(4), pp. 632–644.