The technical implementation required seamless integration of multiple systems. We combined VCV Rack for the modular synth, MediaPipe for facial tracking, and TouchDesigner for visuals and integrating the whole project.

Learning a new program, VCV Rack 2, I developed a custom modular synth patch with 5 layers which included a drone, glitch effects, and randomly generated leads. The patch incorporated the signals from the facial tracking to control different elements.

We utilized MediaPipe's motion capture API through TouchDesigner to track facial movements, translating them into control signals. These signals were then routed via OSC protocol to VCV Rack. Each facial movement corresponded to specific musical parameters: mouth opening controlled drone intensity, head rotation affected filter frequencies, lateral movement panned audio, and blinks triggered glitch effects.