In August 2024, I had the chance to work in Tod Machover’s symphony production with the Sejong Soloists on the Flow Symphony which had it’s world premier in the magnificent Seoul Arts Center, South Korea.
In this project I got to build small AI models from recordings of river sounds for live interaction with the incredible string ensemble – the Sejong Soloists. I was on stage receiving the live audio from the different musicians on stage allowing a real-time timbre transfer of the string performers to generate synthesized river sounds from the AI models. It was most exciting to use audio prompts and string interactions as a way to prompt the models. My favorite learning were the insights from the musicians on how they explored the sonic possibility space of the neural synthesis engine by using many extended techniques. As experts on their string instruments, they had a rich vocabulary to prompt the model with. Unlike foundation models where text prompting gives very little control over the results, here there was a very strong correlation between input audio and output synthesized leading to extremely nuanced control.
Here is some text describing the work – from the excerpt
FLOW Symphony is scored for string orchestra and electronics, and employs a custom-designed AI system to both enhance the performance and to create a constantly transforming online version that allows the piece to be experienced in many different forms. “Flow” referred specifically to the process and the movement of rivers, one of which—outside of Stowe, Vermont—Machover recorded extensively to create acoustic material for the piece. Along with a varied collection of specially recorded “river-like” sounds developed with the Sejong Soloists string players, Machover and his team at the MIT Media Lab created an AI model based entirely on Machover’s original sounds/music that could morph between natural sounds (like a running river) and musical ones (created by an ensemble, like a string orchestra), between simplicity and complexity, and between calm and intensity.
Designed and programmed by Manaswi Mishra, a PhD researcher in the Media Lab’s Opera of the Future group, FLOW AI serves two functions for this piece/project: first, it reacts to live ensemble playing in parts of the piece and adds unusual river/hybrid sonorities in appropriate but surprising ways; and second, by using a new AI Radio system developed by Mishra, it allows an online version of the composition to play out differently at each hearing, preserving the essential feel and “flow” of the music while allowing listeners to dial in changes to duration, complexity, and overall feel. The first AI Radio online version of FLOW Symphony can be experienced at https://flow.media.mit.edu/, and a final version will be launched in Spring 2025 in association with an international tour of the new composition.
Along with Tod Machover and Manaswi Mishra, MAS MS student Ana Schon also traveled to Seoul for rehearsals and the performance of FLOW Symphony, for which she designed and controlled the live sound mix between strings, electronics, and AI.
“FLOW Symphony is inspired by the ‘flowing’ nature of rivers, constant in the rush or trickle of water but also ever-changing through always-different droplets, in the slowly evolving interaction between water and rocks and riverbanks, and in our changing perception as we watch and listen to a river run by. My hope is that the flowing melodies, overlapping harmonies, pulsating rhythms, and twinkling textures of FLOW Symphony—all enhanced by AI—left listeners as captivated, refreshed, and intrigued as I was while listening to ‘my’ Vermont river.” – Tod Machover
FLOW Symphony was recorded by ARTE Korean TV, and will be broadcast and archived—available to listen to and view online—later this fall.
Listen to an excerpt from FLOW Symphony with string orchestra alone and with strings-and-river morphed through the new AI Radio system.
Manaswi Mishra 