Year
Timeline
Category
Immersive
Interactive
Role
Tools
TouchDesigner
MediaPipe
VCV Rack
Audio Visage is an interactive music installation that explores the intersection of human expression and technology. Users control a procedurally generated soundtrack using only facial movements, while a large display shows a dark and glitchy webcam. As project lead and audio designer, I orchestrated the integration of facial motion capture with a custom modular synth system, creating an immersive experience that prompts reflection on our relationship with technology.
Our approach centered on creating an experience that was both engaging and unsettling. By limiting interaction to facial movements, we forced users to engage with technology in an unconventional way. The ominous aesthetic and glitch effects were deliberately chosen to evoke a dystopian sci-fi atmosphere, encouraging viewers to question the rapid advancement of technology. We mapped specific facial movements to musical parameters, creating an intuitive yet alien method of musical expression.
The technical implementation required seamless integration of multiple systems. We utilized MediaPipe's motion capture API through TouchDesigner to track facial movements, translating them into control signals. These signals were then routed via OSC protocol to VCV Rack, where I developed a custom modular synth patch. Each facial movement corresponded to specific musical parameters: mouth opening controlled drone intensity, head rotation affected filter frequencies, lateral movement panned audio, and blinks triggered glitch effects. Working closely with our visual team, we ensured the generative visuals responded to the same control signals, creating a cohesive audiovisual experience.
The project demonstrated our ability to integrate complex systems into a cohesive and fully functional interactive experience while maintaining a strong conceptual foundation. The technical implementation and thematic exploration of this project pushed our boundaries in real-time audio processing and motion capture. It also provided me with valuable experience in leading a multidisciplinary team through a complex creative technology project.
Core Visuals: JamieLynn Gallagher
Asset Design: Jennifer Saloutenko & Nyasia Revill
Audio: Brandon Riley