ICodecTts<T>
Interfaces · AiDotNet.TextToSpeech.Interfaces
Interface for codec-based TTS models that use neural audio codecs with language model decoding.
Implements: ITtsModel<T>
Known implementations: Bark<T>, CSM<T>, ChatTTS<T>, CosyVoiceBase<T>, Dia<T>, FireRedTTS<T>, FishSpeechBase<T>, KaniTTSBase<T>, Llasa<T>, MaskGCT<T>, Moshi<T>, OrpheusTTS<T>, OuteTTS<T>, ParlerTTS<T>, SPEARTTS<T>, SeedTTS<T>, SoundStorm<T>, SparkTTS<T>, TortoiseTTS<T>, UniAudio<T>, VallEModelBase<T>, VoiceCraft<T>, WhisperSpeech<T>, XTTSv2<T>, Zonos<T>
How It Works
Codec-based TTS models use a neural audio codec (e.g., EnCodec, SoundStream, DAC) to represent audio as discrete tokens, then use a language model to predict those tokens from text: Text -> [LM] -> Audio Tokens -> [Codec Decoder] -> Waveform. Architectures include:
- AR + NAR: VALL-E (autoregressive first codebook + non-autoregressive rest)
- Flow matching: CosyVoice, Voicebox (conditional flow matching on codec tokens)
- LLM-based: Fish Speech, Llasa (fine-tuned LLM predicts codec tokens)
- Parallel: SoundStorm (MaskGIT-style parallel decoding)
Properties
| Property | Summary |
|---|---|
int CodebookSize { get; } |
Gets the codebook vocabulary size. |
int CodecFrameRate { get; } |
Gets the codec frame rate in Hz (tokens per second of audio). |
int NumCodebooks { get; } |
Gets the number of residual vector quantization codebooks. |
Methods
| Method | Summary |
|---|---|
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
Decodes codec tokens back to audio waveform. |
Tensor<T> EncodeToTokens(Tensor<T> audio) |
Encodes audio into discrete codec tokens. |