VoxMel TTS Technical Report
Abstract
Continuous autoregressive TTS relies on an acoustic target that enables stable next-latent modeling while preserving fine-grained acoustic details. While existing approaches employ learned compressed representations (e.g., continuous VAE latents) to render prediction tractable, they rely on complex neural encoders that often cap generation fidelity. Conversely, training-free targets like mel-spectrograms eliminate encoder overhead, yet their potential for autoregressive generation remains largely underexplored. In this work, we propose VoxMel, a tokenizer-free framework for autoregressive TTS that makes stable mel modeling tractable through a fixed, exactly invertible two-axis Haar transform. The transform first decomposes each mel spectrogram along frequency into local averages and a frequency residual, then splits the averaged component along time into Coarse-Mel and a temporal residual. A language-model backbone with a flow-matching head autoregressively generates Coarse-Mel. Non-autoregressive residual tiers then predict the temporal residual from Coarse-Mel, followed by the frequency residual conditioned on the reconstructed Frequency-Averaged Mel. Neither residual is fed back into the autoregressive history. Trained on 100K hours of Emilia, the supervised 0.6B model obtains 1.51\% WER and 0.738 speaker similarity (SIM) on test-en, and 1.29\% CER and 0.755 SIM on test-zh. GRPO improves these results to 1.15\% WER / 0.753 SIM and 1.17\% CER / 0.776 SIM, respectively. We also distill the guided flow head into two- and four-step students, requiring 2 or 4 head evaluations per coarse frame. The four-step student retains competitive SIM and nearly matches the supervised model in WER/CER.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
VoxMel introduces a tokenizer-free autoregressive TTS framework using a fixed Haar wavelet transform to separate coarse and residual mel components. This approach effectively mitigates error accumulation in autoregressive generation and achieves competitive performance with significantly lower inference costs through distillation, offering a robust alternative to learned-token systems.
Methodology
The paper proposes VoxMel, a tokenizer-free autoregressive TTS framework that applies a fixed, exactly invertible two-axis Haar wavelet transform to mel-spectrograms. The core innovation is the separation of the acoustic target into a "Coarse-Mel" component (generated autoregressively) and temporal/frequency residuals (generated non-autoregressively). By excluding the residuals from the autoregressive history, the method mitigates error accumulation and encourages reliance on text conditioning rather than acoustic feedback. The use of a flow-matching head for the autoregressive component and non-autoregressive tiers for residuals is a sound architectural choice that balances generation quality with computational efficiency. The application of GRPO (Group Relative Policy Optimization) with a frozen flow head is a notable contribution to post-training strategies for hybrid autoregressive-diffusion models.
Results & evaluation
The experiments are comprehensive, evaluating the model on LibriSpeech and Seed-TTS-eval benchmarks. The results show competitive performance, with the 0.6B model achieving 1.51% WER and 0.738 SIM on test-en, which is comparable to or better than many learned-token baselines like CosyVoice 2 and DiTAR. The ablation studies effectively demonstrate the benefits of the wavelet decomposition, showing increased text reliance and improved stability. The distillation experiments provide valuable insights into reducing inference cost (NFE) while maintaining quality. However, the comparison with state-of-the-art systems like Qwen3-TTS or MOSS-TTS (which use significantly more data/parameters) suggests that VoxMel's advantage is primarily in its tokenizer-free nature and efficiency rather than raw quality superiority over the largest models.
Reproducibility
The paper provides detailed architectural specifications, training hyperparameters, and evaluation protocols. The release of code and demo pages enhances reproducibility. The specific details of the Haar transform implementation and the flow-matching head configuration are clearly described. The use of standard datasets (Emilia, LibriSpeech, Seed-TTS-eval) allows for easy verification of results.
Limitations
The paper acknowledges limitations regarding the depth of the transform (only one level of Haar split is used) and the lack of evaluation on other wavelet families. The evaluation is limited to English and Chinese, and does not cover long-form synthesis, multi-speaker dialogue, or expressive control. The model size (0.6B) is smaller than some recent SOTA models (1.7B-8B), which may limit its performance ceiling compared to larger systems.
Broader impact
The work contributes to the trend of moving away from learned discrete tokens in TTS, offering a more interpretable and potentially more stable alternative. The technique of using fixed transforms to structure autoregressive generation could be applicable to other audio or signal generation tasks. The post-training strategy using GRPO with frozen heads is a useful technique for the community. VoxMel introduces a tokenizer-free autoregressive TTS framework using a fixed Haar wavelet transform to separate coarse and residual mel components. This approach effectively mitigates error accumulation in autoregressive generation and achieves competitive performance with significantly lower inference costs through distillation, offering a robust alternative to learned-token systems.
Author of this paper? Send a correction.