LAION · Chatterbox S3Gen voice conversion
English-first listening room: from vanilla to RL
New: the 40k arm continues the 24k Speaker-Mix LoRA after merging it into the Chatterbox flow decoder. It trains all 112.6M flow-generator parameters (the upstream tokenizer, CAMPPlus and HiFT remain frozen), in groups of 16 on four GPUs. Its within-group reward weights are 40% AudioBox Content Enjoyment from Humaneness Ears Medium, 30% negative source-top-two-emotion MSE and 30% target CAMPPlus cosine. The single warm-up/cosine schedule spans all 40,000 groups.
One source clip per emotion: English examples appear first, with German clips used where the English candidates did not pass the input-quality screen. Every row compares the untouched pretrained Chatterbox S3Gen, the 300-hour supervised model and three 24k-group RL models and their 40k full-flow continuation against the same source audio, fixed target voice and random seed. Open the full 160-case grid.
What changed in the vanilla column?
The vanilla Chatterbox column uses the original pretrained S3Gen decoder and its normal CAMPPlus speaker projection. It has no supervised or RL LoRA and no VoiceNet/EmoNet score projection. The other columns use the same acoustic target prefix and CAMPPlus target embedding, plus the source emotion/style and target static-score conditioning learned in this project.
The source supplies semantic speech tokens; fixed target A or B supplies the acoustic prefix and speaker embedding. This is S3Gen voice-conversion/decoder inference, not text-to-speech generation. All output players use the same source/target pair, ten flow steps and seed.
Fixed target voices
Target A is German; B is English. Both are the 5–6-second references from the original listening room.
Curated-set proxy metrics
These averages cover the 40 displayed sources × two targets. Higher Content Enjoyment (CE), Perceptual Quality (PQ) and target CAMPPlus cosine; lower top-two-emotion MSE are directionally preferable. This is a source-quality-selected, partly training-overlapping demonstration—not a held-out benchmark. CE/PQ/emotion come from Humaneness Ears Medium, which also supplied training rewards for the RL models.
| Model | CE ↑ | PQ ↑ | Emotion MSE ↓ | Speaker cosine ↑ |
|---|
UID-disjoint comparison · 93 other sources
These 93 sources were not used as known RL training UIDs and are separate from the 40 curated listening sources. All arms use the same 93 inputs, two targets and seeds. Speaker-level disjointness is not proven. CE, PQ and emotion are Humaneness Ears Medium proxy scores (the reward model), not independent human listening ratings. CAMPPlus is a proxy for target voice similarity.
| Model | CE ↑ | PQ ↑ | Emotion MSE ↓ | Speaker cosine ↑ |
|---|---|---|---|---|
| 300h supervised baseline | 5.643 | 7.524 | 0.260 | 0.754 |
| Speaker-Mix 24k LoRA | 5.930 | 7.790 | 0.289 | 0.730 |
| Full-flow 40k | 6.074 | 7.876 | 0.320 | 0.738 |
Selection method and limitations
For each emotion we first tried the two existing English sources (one real, one synthetic), without looking at any model prediction. A source needed predicted DNSMOS P808 of at least 3.20 and CAMPPlus minimum cosine of at least 0.40 across overlapping two-second windows. If neither English source passed, we tried the two German candidates. One emotion required a below-gate fallback to retain full 40-emotion coverage. Perceptual quality and known RL-source UID overlap broke close ties. The exact selected IDs and thresholds are recorded in CURATED.json; the below-gate case is marked.
Predicted P808 MOS is not a human MOS study, and within-clip CAMPPlus consistency is not speaker diarization. An expressive single speaker may score inconsistently, and a second speaker may still slip through. The input quality gate reduces obvious problems but cannot guarantee clean single-speaker audio. Known RL training-source UID overlap is marked on each row; speaker-level overlap was not tested.
Vanilla is the unmodified Chatterbox S3Gen checkpoint. The five tuned columns are evaluated at their previously selected checkpoints; the models, seeds and source/target audio match the full comparison. Input semantic tokens are re-tokenized from the original MP3s. No WER, independent emotion judge or human listening score is claimed here. Source-audio rights follow their original licenses; see INPUTS.json. Research preview only.