Designing the Audiovisual Architecture of the Installation Music for 9

Abstract

This paper documents the technological design and implementation of Music for 9, a media installation presented as part of The Missing Link project. The work fragments a percussion performance into distributed visual and sonic components across multiple synchronised displays. Close-up images of the performer’s body are distributed across screens, each paired with an individual speaker that emits processed and partially dislocated sound. Through this spatial arrangement, the installation reconstructs rhythm and melody across the exhibition environment.

While the visual material remains constant throughout the work, the sonic layer gradually transitions from recognisable percussion sounds to an abstracted urban soundscape. This separation between image and sound encourages audiences to imagine a broader city environment beyond the visible performance.

This report analyses the technological infrastructure behind the installation, including robotic systems, audiovisual synchronisation, networked media playback, and custom display construction.

Project Team

Music for 9 was developed collaboratively by an interdisciplinary team working across media art, software engineering, and exhibition production. The project was co-curated by composers GayBird Leung and Vanissa Law under The Missing Link initiative. Technical development and installation were led by Wesker Shek, who designed the robotics control system and sensor interfaces. Fung Wing-lam (“Cat”) developed the audiovisual synchronisation system and the display infrastructure. Yan Lam served as project manager, coordinating logistics and installation processes, while Wilson Chau assisted with exhibition setup and operational testing. The project brought together artistic direction, software engineering, and exhibition production to realise a complex distributed media environment.

The Robot Dog

The installation incorporates a robotic quadruped platform (Fig. 1) to introduce a kinetic and performative element into the exhibition space. The robot used in this project is the Go1 EDU, manufactured by Unitree Robotics. Originally designed as a consumer robotics platform capable of performing companion-like behaviours, the device also provides a programmable environment for developers and artists.

Rather than using the manufacturer’s mobile application or handheld controller, the project team opted to access the robot through the Robot Operating System (ROS). This allowed the system to be integrated with the installation’s broader computational environment.

However, access to the robot’s hardware was limited. Many internal sensors—such as the camera, torch, and motion sensors—were locked within the manufacturer’s software ecosystem. As a result, the team developed an external sensing system. A Raspberry Pi computer was mounted on the robot using a custom 3D-printed structure. This external system allowed the team to run custom programs and attach additional sensors, including a compass used to determine the robot’s orientation.

FIgure 1. Robot dog Go1 EDU manufactured by Unitree Robotics.

There are some existing libraries available online for people to access the control of the dog without going through the official ways. However, some of the features, such as the built-in torch, camera, and motions sensors, are locked. Therefore, we have no way to know about the surroundings. The only controllable part is the motor group, which means I can only control the dog to walk, sit, or turn (mobility). Because the built-in features are not available, we will have built our own external sensor devices. Currently the programme for these is run on a Raspberry Pi, which is mounted on the back of the robot dog by a custom designed 3D printed part. A compass is used to locate the dog’s orientation.

Object detection by colour tracking

To track the position of the robot within the exhibition space, an Intel RealSense D435i depth camera was mounted within the installation environment. This camera system provides RGB imaging, stereo depth data, and inertial measurements that enable six degrees of freedom tracking.

Initial experiments employed the YOLO (You Only Look Once) object detection architecture. Despite its reputation as a powerful real-time detection algorithm, the system performed poorly under the lighting conditions of the exhibition venue. Two factors contributed to this limitation. First, the colour of the robot body was visually similar to the floor surface, making edge detection unreliable. Second, the exhibition space contained floor-to-ceiling windows (Fig. 2), resulting in significant variations in ambient lighting throughout the day.

Alternative methods, including QR-code tracking attached to the robot, were also tested. However, changing light conditions again compromised detection reliability. Ultimately, due to limited installation time, the team abandoned automated positional tracking. Instead, a physical leash was attached to the robot to restrict its movement within a safe range inside the enclosure.

Figure 2. Exhibition space surrounded by floor-to-ceiling windows.

In fact, object detection with optical cameras, even when paired with IR cameras, depends deeply on environment lighting. Therefore, it is very important to have sufficient time to test and debug at the venue. This is one of the very common challenges in the use of technology in art, that is, not having enough time to try things out at the venue.

Vanissa: I personally have experienced the same problems when trying to do colour tracking and object detection with RGB and IR cameras. Gesture detection hardware such as Kinect are usually more reliable because the human body is warm enough to emit detectable infrared radiation. I reckon the difficulties in tracking the dog might be due to the lack of body temperature of the dog.

Wesker: I think that might be one of the factors. After discussing with GayBird we tried do the tracking by QR codes. A camera is mounted on the inside of the cage, which will locate the QR code that is attached to the back of the dog. This did not go well because of extreme changes in ambient lighting at the venue. Because of the floor-to-ceiling glass window, luminosity of the room changes with the outdoor sunlight conditions. In some light conditions, the white on the QR code was recognised as grey, and therefore led to very unstable tracking. We tried to put different lighting around the cage to keep the ambience light level steady, so that the camera can operate under a relatively stable lighting condition, but we still failed. Because we only had a week to set up, we eventually gave up position tracking completely, and instead put a leash on the dog to limit its range of movement, to make sure it would not crash into the cage..

Programming the movement of the dog

The robot’s movements were programmed through ROS libraries developed by third-party developers. These libraries enabled both low-level motor control and high-level gesture control.

Low-level control allows individual motors to be manipulated directly, whereas high-level control combines multiple motors to generate behaviours such as walking, turning, or kneeling. Because the low-level control interface proved unstable and difficult to automate, the installation ultimately relied on high-level gesture commands.

Rather than being directly controlled by a performer, the robot was programmed to move randomly within the enclosure. This behaviour created a semi-autonomous presence that complemented the installation’s exploration of fragmented perception and mediated environments.

Power supply of the dog

Power management proved to be another challenge during development. According to the manufacturer’s documentation, the robot operates on 24-volt, 30-amp direct current, equivalent to approximately 720 watts.

Initial tests attempted to operate the robot continuously using external power adaptors. However, the robot repeatedly failed to stand after booting, suggesting insufficient power supply. Investigation suggested that the internal rechargeable battery may have deteriorated.

Because of this limitation, the robot ultimately operated on external portable batteries, allowing approximately one hour of continuous activity before requiring replacement or recharge.

Synchronisation system

A central feature of Music of 9 is its distributed audiovisual architecture. The installation includes nine video displays, each associated with an independent sound source.

Seven transparent displays are located in the main exhibition space, each paired with a speaker (Fig. 3). Two additional LED televisions are installed in a secondary room. Video playback across these displays is controlled by BrightSign digital signage media players, which synchronise through a UDP-based network protocol.

Figure 3. Monitors and speakers installed in the main exhibition space.

To reduce hardware costs, the team implemented a creative workaround. Rather than using nine individual media players, the system utilised four BrightSign units. A single 4K video signal was divided into multiple smaller outputs using HDMI splitters, enabling one media player to drive several displays simultaneously.

Figure 4.BrightSign media player unit.

Synchronisation of audio and video

Audio playback for the installation was controlled through Ableton Live, which acted as the master system within the audiovisual network.

Audio signals were distributed to the speakers throughout the exhibition space. At the same time, Ableton transmitted Open Sound Control (OSC) messages to the BrightSign players. These messages triggered specific video files stored on SD cards inside each player.

In this configuration, Ableton functions as the master controller, while the BrightSign devices operate as slave units awaiting playback commands.

Using BrightSign with computer

Unlike conventional BrightSign networks, where one player acts as the master unit, the system in Music of 9 used a computer-controlled architecture. This design allowed audio to remain independent from the BrightSign audio outputs, thereby preserving sound quality.

All devices—including computers and media players—were connected through a shared network router. Each unit was assigned a unique IP address so that OSC messages could address specific displays.

Programming had to be uploaded to each BrightSign device in advance via SD cards. During installation, the team repeatedly modified and re-uploaded this code to adjust video framing and synchronisation parameters.

Importance of the power up sequence

During early tests, synchronisation errors occurred when devices were switched on in inconsistent sequences. If software systems started before hardware components were fully operational, communication between devices sometimes failed.

To resolve this problem, the team developed a strict power-up procedure for exhibition staff. The correct order of activating hardware and software ensured stable synchronisation between the networked devices.

Sending live video through network

A live video feed from the robot’s camera was transmitted to a projection system within the installation. Because the camera and projector were located in separate rooms, running a long cable connection would have required an approximately 30-metre cable, which risked signal attenuation.

Instead, the team transmitted the video wirelessly through the local network. A computer connected to the robot camera streamed the feed to another computer in the projection room.

Video processing was implemented in TouchDesigner, where the live feed was masked with a blurred oval frame. This effect produced a dreamlike perspective resembling the robot’s point of view while partially obscuring audience faces visible in the original camera feed.

---

Appendix

About The Missing Link

Curated by composers Gaybird Leung and Vanissa Law, The Missing Link consists of three Arts Tech exhibitions, an immersive performance, and a series of open lectures on the current issues in the use of technology in arts. By adding two essential elements—timeliness and performance— into art forms that do not originally exist in time, the two composers wish to inspire new creations and to give more perspective to existing art forms. Through the creation of multimedia artworks the artists explore ways to use technology to connect different types of media to facilitate storytelling.

The Missing Link is a two-year project funded by the Arts Capacity Development Funding Scheme (ACDFS) from 2021-2023. On top of commissioning new art works, the technical goal of this project is to develop a media synchronisation system, which will then become open source to benefit the future development of the Arts Tech industry.

About Music for 9

Music of 9 was held at H Queen’s Hong Kong, from 27 April to 3 June 2023. It is a technological recreation of a youth centre in the 1990s that explores the paradigm shift of interpersonal communication over the past three decades.

Music for 9 is a media installation that fragments the video of a percussion performance, visually and aurally. Close-ups of body parts will be shown on synchronised displays, while the speakers that are paired with the displays will fill the space with impure sounds. The whole visual and audio spectrums are filled with rhythms and melodies built by re-integrated moving images and sound. Sound gradually changes from literally percussion music to sounds of the city. While the images on the screens remain unchanged, the sound that gradually breaks away from the image and took the audience to imaginations of the city soundscape.

Filled with recreational activities and new faces, youth centres were once a place to hang out, make friends and shape one’s social life. Two classic tabletop games, Novuss and Table Football, were re-created in such a way that can only be controlled through a computer interface.