In Sept 2023, I was part of Tod Machover’s Opera production of VALIS, a reimagination of an iconic Opera created by him in 1987. I was given the task to imagine the futuristic science fiction musical instrument (shown in this image) by Mini – a character symbolizing knowledge, the unknown in this – Philip K Dick’s last magnum opus part-biographical part insane novel from 1981.
I collaborated with fellow MIT student Nina, to create the gesture based instrument “The Jar”. Here is a short clip of the performance of the Jar on stage while I control the system parameters from the front of house during one of the premier performances of the VALIS 2023 opera in Boston.
As shown in the video, the position, tilt, acceleration and movements of the Jar control the sound being generated in real-time. All sounds are generated from AI models trained on a small dataset of science fiction sounds from the original 1987 opera recordings. Each night the show was performed led to a different kind of sound being produced from the system. Since the movement controls the sounds generated, the performer tilts and moves in space “looking” and listening for sounds generated from the system. When she finds interesting local spaces and gestures, she can touch the outside walls of the jar to store the position and retrigger sounds. This leads to a balance between exploratory performance and controlled performative gestures.
Here is a review of the performance in the Boston Globe – an-ai-opera-1987-reboots-new-generation
a more technical explanation: An IMU (inertial measurement unit) sensor sits inside the jar allowing for an accelerometer, gyroscope and magnetometer to give 9 dimensions of position, tilt and movement information. This is relayed wirelessly to a computer at the soundboard that is running a synthesis engine in MAX MSP. The movement is used to sample from a 127 dimensional autoencoder latent space trained on a small dataset of curated science fiction sounds. Almost like a theremin, continuous movement allows for a continuous sampling from the latent space synthesizing audio at 44.1KHz. The walls of the Jar instrument have capacitive sensing aluminium plates that can pass a trigger when touched with the hand. These are used to have predictive behavior by storing current position and movement dimensions and allowing for retriggering certain sonic behaviors. LED lights on the inside of the jar are used to funciton both as an indicator of the state of the system and as a means to communicate visually to the audience.
This was an exciting experience to explore gesture as a means of adding performative aspect to AI synthesis models – allowing for a more intricate vocabulary by the performer and also allowing the audience to understand the connection between performer and audio output through a more traditional “musical instrument” interaction.
Extra Notes: Here are some behind the scenes of using gesture in a theremin like forumation to explore sampling parameters of a latent space.
Manaswi Mishra 