ICodecTts<T>

Interfaces · AiDotNet.TextToSpeech.Interfaces

Interface for codec-based TTS models that use neural audio codecs with language model decoding.

Implements: ITtsModel<T>

Known implementations: Bark<T>, CSM<T>, ChatTTS<T>, CosyVoiceBase<T>, Dia<T>, FireRedTTS<T>, FishSpeechBase<T>, KaniTTSBase<T>, Llasa<T>, MaskGCT<T>, Moshi<T>, OrpheusTTS<T>, OuteTTS<T>, ParlerTTS<T>, SPEARTTS<T>, SeedTTS<T>, SoundStorm<T>, SparkTTS<T>, TortoiseTTS<T>, UniAudio<T>, VallEModelBase<T>, VoiceCraft<T>, WhisperSpeech<T>, XTTSv2<T>, Zonos<T>

How It Works

Codec-based TTS models use a neural audio codec (e.g., EnCodec, SoundStream, DAC) to represent audio as discrete tokens, then use a language model to predict those tokens from text: Text -> [LM] -> Audio Tokens -> [Codec Decoder] -> Waveform. Architectures include:

  • AR + NAR: VALL-E (autoregressive first codebook + non-autoregressive rest)
  • Flow matching: CosyVoice, Voicebox (conditional flow matching on codec tokens)
  • LLM-based: Fish Speech, Llasa (fine-tuned LLM predicts codec tokens)
  • Parallel: SoundStorm (MaskGIT-style parallel decoding)

Properties

Property Summary
int CodebookSize { get; } Gets the codebook vocabulary size.
int CodecFrameRate { get; } Gets the codec frame rate in Hz (tokens per second of audio).
int NumCodebooks { get; } Gets the number of residual vector quantization codebooks.

Methods

Method Summary
Tensor<T> DecodeFromTokens(Tensor<T> tokens) Decodes codec tokens back to audio waveform.
Tensor<T> EncodeToTokens(Tensor<T> audio) Encodes audio into discrete codec tokens.