There is a slogan at Dolby Laboratories that has long resonated with Yue Qiao: Hear the Whole Picture. In the age of high-definition and computer-generated imagery, visual input tends to dominate the processing experience. But audio is an equally important part of how users perceive everything from the score of a movie to the sound effects of a video game or even a conversation over Zoom.
No one knows this better than Yue. The beauty of sound has guided him throughout his entire life as both a performer and researcher. As a young child he studied piano and guitar, later joining a college rock band and the Princeton Laptop Orchestra. Growing up in China, he was first introduced to the field of audio engineering as an undergraduate at Peking University. Today, Yue is a second-year graduate student in Mechanical & Aerospace Engineering and a member of the 3D Audio and Applied Acoustics (3D3A) Laboratory at Princeton.
“When I was a sophomore majoring in physics I did mixing and recording for my band as a hobby. It aroused this strong interest within me to explore an audio technology career because the field has a strong foundation in electronics and physics,” he explains. “I was inspired by the idea that I could combine my interest in music with my skills in science.”
Spatial audio, or 3D sound, is an area of audio engineering that involves human’s spatial perception of sound. It takes the concept of surround sound one step further. The idea is to make the experience even more immersive and realistic by allowing the user to perceive sound in precise 3D locations—top, bottom, left, or right. For example, if someone is watching a movie with spatial audio and the actor drops a glass at the left side of the screen, the scattering sound would resonate from the left side of the viewer.
Yue’s particular research within the field involves creating personal sound zones using loudspeaker arrays. That means people in a particular zone could hear their preferred music or audio without interfering with someone next to them.
“In order to generate such sound zones, arrays of loudspeakers are often required, as well as microphones being placed at target locations to capture the sound field,” describes Yue. “Eventually, the audio filters are designed and implemented, and the whole process of problem solving involves multidisciplinary knowledge from signal processing, room acoustics, psychoacoustics, and optimization.”
With the advance of techniques like optimization and increasing demand of high-quality audio, Yue explains, the applications are numerous. Imagine a passenger and a driver being able to listen to different types of music in the same car without headphones. Visualize a museum where people in one open space can listen to different parts of an exhibit at their own pace. Think about a train station where travelers would hear different instructions based on what platform they stood on.
To create multiple sound zones, they optimize and test various algorithms in the 3D3A lab. Using loudspeakers and microphones arrays to narrow the sound distribution in the room, they tinker with these algorithms until they achieve the desired sound effect. Right now, Yue is involved primarily with developing the algorithms for the project. Eventually a hardware and software system will also have to be developed.
"Yue is that rare combination of the careful and skeptical experimentalist, and the creative and analytical theorist. In our weekly, two-hour research group meetings it is seldom that the group members do not learn something new from Yue,” says Edgar Choueiri, PhD, his advisor and Director of the 3D3A Lab.
Yue has already found practical applications for his work. He uses insights from the lab to enhance The Princeton Laptop Orchestra’s performances. PLork, as it is known around campus, is a 21st century take on orchestra—instead of instruments, the musicians use a laptop and custom designed hemispherical speakers that emulate the way traditional orchestral instruments cast their sound in space. The traditional conductor is replaced with wireless networking and video augmentation.
Before the pandemic Yue participated in several PLork performances. Some of their pieces are rehearsed while others are entirely improvisational.
“I have always played music as a way to express my feelings and emotions,” says Yue whose favorite bands include Radiohead and the Beatles. “PLork is an outlet for that enjoyment but it is also practical because I can apply my research outcomes. Since my research deals with how humans perceive sound, these devices or algorithms can be used in music composition to enhance the experience for our audiences.”
Yue’s interest in audio engineering began at Peking University. As a research assistant in the Speech and Hearing Research Center, he worked on sound field recording and reproduction using Higher Order Ambisonics. His research focused on reproducing high-fidelity sound and exploring its application in artistic creations.
In one project, he constructed a spherical microphone array with 32 distributed sensors using a physics-inspired algorithm. Yue implemented its overall recording, encoding and binaural playback system, which supported real-time head tracking. To evaluate the prototype, he used it to record group meetings and musical performances happening at Peking University and then reproduced the recordings using headphones.
“The microphone array could capture sound fields with a high spatial resolution,” he describes. “It became the foundation of future projects in our lab, such as sound source separation and localization.”
The importance of audio in an immersive experience is often underestimated. In the future, Yue hopes to continue playing music and finding ways to enhance the audio experience for the masses, maybe even back home someday.
“The same way Apple has reinvented headphones, I want to be a part of creating spatial audio effects that improve how people enjoy the sound around them, watch a movie, or listen to songs,” he says.
-- Carolyn Sayre