Musical Audio Gesture Recognition (AGR) is a Max-based workflow for turning rehearsal-time audio gestures into programmable performance actions. Rather than treating recognition as a fixed classifier inserted into a finished composition, AGR keeps gesture vocabulary design, dataset capture, feature representation, model training, and output routing editable inside the same patching environment used in rehearsal and performance. Built on top of FluCoMa, AGR provides an integrated capture–training–runtime pipeline that combines fixed-length padding without time warping, a local frequency-deviation representation derived from sinusoidal peaks, and deployment patches that expose recognized gestures as OSC, MIDI, or gesture-bound audio triggers.

We frame AGR as a programmable artistic workflow: performers and composers define what counts as a gesture, test recognition behavior against rehearsal examples, and revise mappings or thresholds as part of the compositional process. We characterize the workflow across four instrument-specific datasets (clarinet, flute, trumpet, trombone), using recognition behavior and post-trigger processing time as practical checks on rehearsal use rather than as benchmark endpoints. We also discuss deployments ranging from single-instrument prototypes to concert works and a 12-model large-ensemble production. In these settings, AGR is part of the performance patch: gesture labels, training examples, thresholds, and routing decisions can be revised in rehearsal while the concert setup remains continuous.

Hongshuo Fan is an interdisciplinary composer, new media artist, and creative programmer whose work bridges technology and the arts. He is currently an Assistant Professor of Music Technology at Texas A&M University. His practice focuses on creating immersive multimedia experiences that integrate acoustic instruments, live electronics, generative visuals, light, and body movement. Deeply influenced by the fusion of traditional culture and contemporary innovation, Hongshuo’s projects often leverage machine learning and artificial intelligence to push the boundaries of art and music. His portfolio spans interactive installations, audiovisual performance, and real-time media systems. Hongshuo’s work has been featured in international venues, including the International Computer Music Conference (ICMC) and the New Interfaces for Musical Expression (NIME) conference. His honors include the Asian-Oceania Regional Award of the International Computer Music Association and the Giga-Hertz Production Award.