How does a listener know immediately when she or he walks into a room with a live grand piano instead of a recorded one? One reason is the complex interactions between the piano and the space that surrounds it. While many works have combined piano with electronic sounds, few have attempted to fuse them by modeling the piano’s spatialization pattern dynamically with loudspeakers. This project seeks to explore this goal artistically in collaboration with the OM7/om-spat Tech Project team, through a new composition entitled Cosmologies. Few composers have used Artificial Intelligence (AI) techniques to assist in the creative process, yet AI and machine learning techniques present promising possibilities when applied to the problems of timbre and spatial audio. The intricate interdependency of timbre and space that brings a concert grand piano to life are difficult to reproduce electronically, but machine learning can help. The development of these computer tools will enable the production of Cosmologies for piano and electronics, in which the quicksilver filigree of live concatenated samples moving through the performance space complements the flickering image of the piano with its diffusion pattern magnified by close and contact microphones.

Cosmologies leads us into the secret inner life of a concert grand piano. By combining experiences and motivations, Aaron Einbond and OM7/om-spat Project explored artificial intelligence and machine learning, and interrogate the musical microcosm of the piano interior as it is exploded out to the space. Radiation patterns of acoustic instruments were analysed, allowing the computer to “learn” from their rich interactions of timbre and space. The electronic sounds were then diffused in an ambisonic loudspeaker system to situate the listener in a 3D sound space, while machine learning techniques were applied to choose sounds based on their timbral characteristic and map them spatially. The final result is a unique experience where the listener is situated inside the virtual instrument and stimulated to explore the intricate interdependencies of timbre and space.

Performance at the Centre Pompidou - IRCAM Live, March 2020 @Quentin Chevrier

From the research point of view, the residency was a unique opportunity to bring together the several branches of research and technology developed in the Tech Project lab — what is perceived as being the main and most relevant means to assess the quality and success of research and developments in the field of music technology. The Cosmologies residency was also one of the first fully realized projects involving the technology developed by the Tech partner – the new OM# environment and its connection to Spat (OM-Spat). The residency served both as a proof of concept, test case and outcome for this research project. It has pushed the research project one step further, giving the opportunity to deeply explore concepts and techniques, while contributing to the development of new tools – as it was the case of software packages for the software Max. Finally, Cosmologies has boosted the exploration of machine learning applied to computer assisted composition (and sound spatialization).

From the artistic perspective, the residency introduced several novel technological tools into artistic practice with potential to be explored by other researchers and artists. Cosmologies is the first application of machine learning to instrumental radiation patterns; the first work connecting concatenative synthesis with higher order ambisonics (HOA); and the first application of the EM32 Eigenmike in a live performance context.

Interview with Aaron Einbond at CENTQUATRE, Paris during the STARTS Residencies Days

Read the final report Discover more on the blog