Motion captured: ‘Symphonic’ installation draws on movement of its visitors

The universe is awash in sound.

So declared teams of scientists from around the world last summer. As reported in Smithsonian Magazine, the scientists said they’d discovered a new kind of gravitational wave that creates a constant, ambient hum. The findings suggested a “rolling, noisy universe alive with the cosmic symphony of gravitational waves.”

Those findings captivated Maggie Schnyer, now a Yale senior majoring in Computing and the Arts. After reading the article, Schnyer decided to create an interactive, mini universe of music of her own. Also a musician and assistant conductor of Yale’s Davenport Pops Orchestra, Schnyer set out to design an immersive installation that would use motion-capture technology to track visitors’ movements and create a collaborative piece of music.

Motion capture technology is used to record human movement and then translate that movement into a computerized form. It is commonly used in filmmaking, animation, video game design, and sports training.

Schnyer had seen motion capture tools used in combination with music, usually with dancers wearing suits fitted with sensors and producing sound in very meticulous ways. “I wanted to do something that was more accessible to anyone,” she said.

The project would also serve as her senior thesis. But first, Schnyer had a lot to learn.

Last semester she researched various motion capture music projects and started to familiarize herself with coding software for the audio and visual aspects of her project. She wanted to use projectors and lighting boards for the installation, something else she’d never done before.

“I was learning everything as I was going through the project,” Schnyer said, “and I received so much guidance from my advisors.”

Working with Schnyer were Konrad Kaczmarek, an associate professor in the Department of Music in Yale’s Faculty of Arts and Sciences, Scott Petersen, a senior lecturer in Computing and the Arts, and Ross Wightman, technical manager at the Yale Center for Collaborative Arts and Media (CCAM).

Her efforts culminated last month in an interactive motion-capture performance staged in CCAM’s Leeds Studio, which is equipped with a Vicon Motion Capture System.

Window shades blocking the light from outside twinkled with stars, a form of generative art Schnyer created with code. An ever-changing galaxy-like design, also one of her programs, glowed on a large screen at the front of the room.

A low hum could be heard — a nod to the universal hum. But Schnyer had also composed a soothing, nine-minute piece of music written in Logic Pro, Apple’s digital audio workstation, that could only be heard by guests wearing special sets of headphones affixed with sensors. The music was split into different tracks that were assigned to the four sets of headphones as well as other sensors attached to space-themed objects around the room. The motion-capturing cameras were attached to the ceiling.

As visitors moved about the room, different musical tracks would adjust the volume and panning in their headphones according to who and what they were close to.

“The motion-tracking technology designated each person that entered the space as a new instrument, with musical lines mimicking their movement and stillness as they explored the exhibit,” Schnyer said.

The project was a hit. Some 60 people (mainly students) signed up for one of the 15-minute slots to try out the headphones.

Schnyer was especially pleased at the end of the night when some friends who were laying on the floor in the studio told her how relaxing the music was and how much they wanted to hear it again.

“I assumed the biggest draw of my project would be the technological and movement aspects of it,” she said. “I was pleasantly surprised that the piece I had composed for the space was received well and contributed to, rather than distracted from, the meditative atmosphere I was aiming for.”

Schnyer, who is from Torrington, Connecticut, isn’t sure what’s next in her upcoming gap year, aside from her first trip to the Philippines, her mother’s native country. But she’s already thinking about other projects combining music and art with technology.

“It was so special to have had an opportunity to exercise full creative control over my own space in an installation,” she said, “and to be able to share it.”