FireRedTTS<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

FireRedTTS (Guo et al. 2024): a foundation text-to-speech system — a decoder-only language model predicts a recording’s semantic tokens from text and a speaker embedding, a flow-matching Mel decoder turns the tokens into a 16 kHz mel in a prompt’s timbre, and BigVGAN-v2 makes the audio.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → FireRedTTS<T>

Implements: ICodecTts<T>

For Beginners

give FireRedTTS a few seconds of someone speaking and it reads new text in that voice.

How It Works

The parts (§3): the semantic-aware speech tokenizer (HuBERT features, a convolutional encoder and a product quantizer of 16,384 tokens at 40 ms, FireRedTtsSpeechTokenizer) and its acoustic encoder (an ECAPA-TDNN speaker embedding); the GPT (§3.2, the release’s XTTS-style model, FireRedTtsGpt); the flow-matching Mel decoder (§3.3.1, FireRedTtsFlow); and BigVGAN-v2 at 24 kHz (the release) or the paper’s 48 kHz (Vocoder).

Synthesis runs the release’s synthesize: the text split (text_split), each piece normalized and tokenized, seven token sequences sampled and the second shortest kept, the flow and the vocoder, the pieces joined. Instruction tuning (§4.2) is opt-in (InstructionTuning): an emotion and paralinguistic markup (SynthesizeInstruction).

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Generative, inputSize: 1, outputSize: 1);
var model = new FireRedTTS<float>(architecture, new FireRedTTSOptions());
model.LoadReferenceWeights("FireRedTTS");          // fireredtts_gpt.pt, fireredtts_token2wav.pt, fireredtts_speaker.bin
model.Voice = model.CreateVoice(promptAudio, promptSampleRate);
var audio = model.Synthesize("小红书,是中国大陆的网络购物和社交平台。");   // 24 kHz

Constructors

Constructor Summary
FireRedTTS(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
FireRedTTS(NeuralNetworkArchitecture<T> architecture, FireRedTTSOptions options) Creates a native, trainable FireRedTTS.
FireRedTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, FireRedTTSOptions options) Creates an ONNX-backed FireRedTTS for inference.

Properties

Property Summary
bool TrainsOnSampleObjective { get; }
int CodebookSize { get; }
int CodecFrameRate { get; }
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int MaxTextLength { get; }
int NumCodebooks { get; }

Methods

Method Summary
IReadOnlyList<string> SplitText(string text) The release’s text pieces (text_split; its dropped last piece kept unless ReleaseTextSplit).
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> DecodeFromTokens(Tensor<T> tokens)
Tensor<T> EncodeToTokens(Tensor<T> audio)
Tensor<T> Synthesize(string text)
Tensor<T> SynthesizeInstruction(string text, FireRedTtsEmotion emotion) Instruction-tuned speech (§4.2): the text with the paper’s paralinguistic markup (Table 1: tags such as [laugh] and [breath] in the text, and after a Chinese character ˆ for speaking with a laugh, @ for emphasis, P, C, R or `S…
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate) A voice from a recording of the speaker at sampleRate: its speaker embedding (16 kHz, extract_spk_embeddings) and the flow’s prompt mel (MelSpectrogramExtractor).
int[] EncodeSemanticTokens(Tensor<T> audio, int rate) The semantic tokens of a recording at rate (extract_speech_tokens).
int[] TokenizeText(string text, string language) The token ids of a text piece (normalized, language-tagged, tokenized; VoiceBpeTokenizer.encode).
void LoadReferenceWeights(string directory) Loads the release from its directory (Hugging Face fireredteam/FireRedTTS): the GPT (fireredtts_gpt.pt), the flow and BigVGAN (fireredtts_token2wav.pt; the flow’s frozen input embedding holds the speech tokenizer’s codebooks) and the…
void LoadSpeechTokenizer(string codecPath, string hubertPath) Loads the speech tokenizer: its encoder, quantizer and acoustic encoder from the FireRedTTS-1S release’s speech_tokenizer/codec.bin (the same codebooks as FireRedTTS-1’s flow), and HuBERT from a Transformers-format checkpoint of `TencentG…

Fields

Field Summary
int AnalysisRate The rate the tokenizer, the speaker encoder and the mels read (fixed by the paper and release).