MOTION CAPTURE + MUSIC

MOTION CAPTURE + MUSIC

/* Using Vicon Motion Capture Technology, Max MSP, and Logic Pro, I spent two semesters programming and creating an interactive sound art installation. Visitors wearing headphones could explore the soundscape by moving tracked objects, each assigned to an instrument. */

This project is the culmination of a two-semester senior thesis for the completion of a Bachelor of Arts in Computing and the Arts with a concentration in music from Yale University. It was hosted at the Yale Center for Collaborative Arts on March 28, 2O24, and featured in Yale News on April 24.

ABSTRACT

Over the summer of 2O23, researchers in the astrophysics community discovered proof of a new kind of gravitational wave creating a continuous hum heard across the universe. Likely created by supermassive black holes, the new evidence “‘shatters the perception of a static universe’” and points to future research in which “‘we’ll likely be able to pick out notes from the instruments playing in this orchestra’” [1]. Everything in the universe is making music, and we feel its vibrations. The goal of this project is to create an interactive sound art installation combining music and technology to create a much smaller model of this research, where motion capture technology tracking visitors’ movement through a space creates a collaborative piece of music.

Motion tracking technology designates each tracked object and person that enters the space as a new instrument, with musical lines mimicking their movement and stillness as they explore the exhibit.

My areas of interest both within Education Studies and as a musician on campus focus on accessibility to music and music education. I hope to challenge the notion that not everyone can be a musician by creating an enjoyable immersive experience that does not require a musical background or knowledge of music theory in order to contribute just as meaningfully as someone with extensive musical training.

THE CODE

All of the technical components of this project are split between Max MSP on five computers (one computer per listener) and the Vicon Motion Capture system built into the CCAM’s Leeds Studio. Max MSP is a visual programming language especially useful for musicians and performers. I used an external package called Ircam Spat5 for creating binaural soundscapes, expanding on it by importing motion tracking data sent over OSC messaging from Vicon to each laptop. Each object being tracked in Vicon is assigned to a sound source in Max, then audio files are played accordingly.

There is a second Max patch controlling the speaker audio of the room, programmed to control each of the 8 speakers individually. The program creates a subtle, complementary oscillation in gain (or volume) of the hum on each side of the room.

The visuals in the space projected on the walls and front of the room were also created with code in the generative art program, Hydra. A thesis project as well for its creator, Olivia Jack, Hydra is an online, in-browser coding language with functions inspired by analog modular synthesizers.

/* Ircam Spat 5 GUI for Max. Receives MoCap data via OSC, then assigns audio track to sound sources 1-7 (green circles) and listener (center). */

THE MUSIC

The 9 minute composition featured in the installation is a single, continuous piece of music written originally in Logic Pro and split to 9 different audio files: one for each tracked person or object in the room as well as the droning hum heard both in the headphones and over the speakers.

The low hum heard throughout the entire piece is a droning Db created on an airy pad in Logic. The entire piece is tonally centered around Db major. The piece is split into four distinct parts, each a reference to a musical idea, song, or memory that’s meant a lot to me in my time at Yale.

The first section is a nod to both “O Superman” by Laurie Anderson and “Abbey” by Mitski, two composers I look up to for their approaches to creating and use of technology. “Abbey” and the album containing it, Lush, was also a college composition project for the artist Mitski.

The second section is inspired by composer Gérard Grisey’s piece titled Partiels. Every sound you’ve ever heard has its own recipe—the ingredients being harmonic partials and how much of each of them you can hear. Grisey’s piece writes each section of the orchestra as a harmonic partial above the blaring trombones. In my piece, each person is a harmonic partial, with one note changing and returning each chord.

The two plucked basses of the third section start completely in sync, then become gradually out of phase with each other to electronically recreate the tape loop effect of Steve Reich’s “Come Out.” The two tracks have exactly the same notes and timing on first play, but one track is slightly shorter than the other and becomes more noticeable as the two are looped. The vibraphone and its moving line are a quote from Alabama Shakes’ “Sound & Color”, a song I found my first year and associate with my best friend. I added more drum kit parts to experience the space and object tracking with more percussive rather than tonal elements.

The fourth and final section is a more traditional orchestral piece. It’s a transposed excerpt from a longer work I started while studying abroad in Venice last summer. Each person and object in the room acts as an orchestral voice, and sounds from earlier on in the piece are called back. The audio files connecting the sections are recordings from my own phone: the first is of a video I took sitting on Hillhouse my first year, and the second is my family singing happy birthday to my dear Aunt Rhonda, who passed away alongside her daughter, also named Rhonda, this past year at Yale New Haven Hospital.

Creating a piece of music that tapped into all the areas I wanted to explore while being something I’m proud to share was by far the biggest challenge of this project, but every second of this piece has personal meaning.

The final composition visitors heard and its Logic Pro file:

// Logic Pro file color codes tracks (ex. Player 1 is bright pink)

THE VISUALS

// Moving stars created using Voronoi Noise. Coded in Hydra

// Moving spiral. Coded in Hydra

/* Thesis program printed on business cards. The cover background is a spectrogram of the Low Hum heard throughout the piece! */

SPECIAL THANKS

My Advisors

Professor Konrad Kaczmarek, Department of Music

Professor Scott Petersen, Department of Computer Science

Professor Ross Wightman, Yale Center for Collaborative Arts and Media

Yale CCAM

Doctor Matthew Suttor

Doctor Lauren Dubowski

Yale News

Lisa Prevost

Dan Renzetti

David Stanley

REFERENCE

[1] Sullivan, Will. “Gravitational Waves Create a Constant 'Hum' Across the Universe.” Smithsonian Magazine, 29 June 2O23, https://www.smithsonianmag.com/smart-news/gravitational-waves-create-a-constant-hum-across-the-universe-180982457/#:~:text=The%20universe%20is%20filled%20with,of%20giant%2C%20accelerating%20celestial%20bodies. Accessed 8 September 2O23.