U-Codec: Neural Speech Codec under Extreme Temporal Compression for Fast High-fidelity Speech Generation

· Code ·

We propose U-Codec, an Ultra low frame-rate neural speech Codec that achieves high-fidelity reconstruction and fast speech generation at a frame-rate of 5Hz (5 frames per second). Extreme temporal compression typically leads to severe intelligibility and spectral detail loss. We overcome this by introducing a Transformer-based inter-frame long-term dependency module and systematically exploring residual vector quantization (RVQ) depth and codebook size to identify optimal configurations. Moreover, we apply U-Codec to a large language model (LLM)-based auto-regressive TTS model, which leverages global and local hierarchical architecture to effectively capture dependencies across multi-layer tokens. Experiments demonstrate that U-Codec improves LLM-based TTS inference speed by 3$\times$ over high-frame-rate codecs while maintaining similarity and naturalness. These results validate the feasibility of using 5Hz discrete tokens for fast and high-fidelity speech synthesis.

Fig.1 Preliminary experiments on varying the frame rate and RVQ depth of different codecs. Bubble size indicates the number of RVQ layers. Our method achieves competitive PESQ performance under ultra low frame rates (5–12.5Hz), demonstrating the effectiveness of the tokenizer with multi-layer RVQ. EnCodec, Mimi, and DAC operate at higher frame rates.

Fig.2 Architecture and training of our U-Codec at 5Hz. The encoder (left) consists of convolutional layers followed by a Transformer to capture long-term dependencies. Latent features are quantized through factorized residual vector quantization (FRVQ) and optimized for high-fidelity reconstruction. The decoder (right) mirrors the encoder to synthesize the output waveform.