Skip to content
D-CSIL

Project File

Audio & Voice Lab

Machine learning for sound: conditional drum generation, 6-stem source separation on MLX, XTTS voice cloning, and edge TTS.

Active ResearchGenerate it, separate it, say it — locally.

Drums

OBSERVED

DrumAI-Conditional trains conditional drum synthesis models, with a HibbieFormer drum variant (hibbieformer_drum_mlx.py) applying the lab's recurrent architecture to rhythm generation. DrumSep-AI runs Demucs-based drum-stem separation, and a rule-based MIDI Agent generates chord/bass/drum patterns as a menu-bar app.

Voice

OBSERVED

The voice-cloner app fine-tunes XTTS on recorded voices — three cloned profiles built so far from chunked, normalized 24kHz datasets with Whisper transcription. The Piper/piper-phonemize toolchain runs fast VITS-based TTS via ONNX on Raspberry Pi-class hardware in 28+ languages.

Separation & Generation

OBSERVED

BSRoFormer6StemMLX ports BandSplit-RoFormer 6-stem separation (vocals, drums, bass, piano, strings, other) natively to Apple Silicon. StableAudio MLX generates audio locally, with rendered outputs on disk.

Current Limitations

  • −No quantitative audio benchmarks (SDR, MOS) have been recorded yet — outputs are evaluated by ear so far.
  • −Voice cloning profiles are small-data fine-tunes; quality varies by speaker.

Next Steps

  • +Add objective separation metrics to the BSRoFormer port.
  • +Condition DrumAI on genre/tempo tags from the SD3 MIDI export corpus.