FireRedTTS<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
FireRedTTS (Guo et al. 2024): a foundation text-to-speech system — a decoder-only language model predicts a recording’s semantic tokens from text and a speaker embedding, a flow-matching Mel decoder turns the tokens into a 16 kHz mel in a prompt’s timbre, and BigVGAN-v2 makes the audio.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → FireRedTTS<T>
Implements: ICodecTts<T>
For Beginners
give FireRedTTS a few seconds of someone speaking and it reads new text in that voice.
How It Works
The parts (§3): the semantic-aware speech tokenizer (HuBERT features, a convolutional encoder and a product
quantizer of 16,384 tokens at 40 ms, FireRedTtsSpeechTokenizer) and its acoustic encoder (an
ECAPA-TDNN speaker embedding); the GPT (§3.2, the release’s XTTS-style model, FireRedTtsGpt); the
flow-matching Mel decoder (§3.3.1, FireRedTtsFlow); and BigVGAN-v2 at 24 kHz (the release) or the
paper’s 48 kHz (Vocoder).
Synthesis runs the release’s synthesize: the text split (text_split), each piece normalized and
tokenized, seven token sequences sampled and the second shortest kept, the flow and the vocoder, the pieces joined.
Instruction tuning (§4.2) is opt-in (InstructionTuning): an emotion and paralinguistic
markup (SynthesizeInstruction).
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Generative, inputSize: 1, outputSize: 1);
var model = new FireRedTTS<float>(architecture, new FireRedTTSOptions());
model.LoadReferenceWeights("FireRedTTS"); // fireredtts_gpt.pt, fireredtts_token2wav.pt, fireredtts_speaker.bin
model.Voice = model.CreateVoice(promptAudio, promptSampleRate);
var audio = model.Synthesize("小红书,是中国大陆的网络购物和社交平台。"); // 24 kHz
Constructors
| Constructor | Summary |
|---|---|
FireRedTTS(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
FireRedTTS(NeuralNetworkArchitecture<T> architecture, FireRedTTSOptions options) |
Creates a native, trainable FireRedTTS. |
FireRedTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, FireRedTTSOptions options) |
Creates an ONNX-backed FireRedTTS for inference. |
Properties
| Property | Summary |
|---|---|
bool TrainsOnSampleObjective { get; } |
|
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
|
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<string> SplitText(string text) |
The release’s text pieces (text_split; its dropped last piece kept unless ReleaseTextSplit). |
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
|
Tensor<T> EncodeToTokens(Tensor<T> audio) |
|
Tensor<T> Synthesize(string text) |
|
Tensor<T> SynthesizeInstruction(string text, FireRedTtsEmotion emotion) |
Instruction-tuned speech (§4.2): the text with the paper’s paralinguistic markup (Table 1: tags such as [laugh] and [breath] in the text, and after a Chinese character ˆ for speaking with a laugh, @ for emphasis, P, C, R or `S… |
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate) |
A voice from a recording of the speaker at sampleRate: its speaker embedding (16 kHz, extract_spk_embeddings) and the flow’s prompt mel (MelSpectrogramExtractor). |
int[] EncodeSemanticTokens(Tensor<T> audio, int rate) |
The semantic tokens of a recording at rate (extract_speech_tokens). |
int[] TokenizeText(string text, string language) |
The token ids of a text piece (normalized, language-tagged, tokenized; VoiceBpeTokenizer.encode). |
void LoadReferenceWeights(string directory) |
Loads the release from its directory (Hugging Face fireredteam/FireRedTTS): the GPT (fireredtts_gpt.pt), the flow and BigVGAN (fireredtts_token2wav.pt; the flow’s frozen input embedding holds the speech tokenizer’s codebooks) and the… |
void LoadSpeechTokenizer(string codecPath, string hubertPath) |
Loads the speech tokenizer: its encoder, quantizer and acoustic encoder from the FireRedTTS-1S release’s speech_tokenizer/codec.bin (the same codebooks as FireRedTTS-1’s flow), and HuBERT from a Transformers-format checkpoint of `TencentG… |
Fields
| Field | Summary |
|---|---|
int AnalysisRate |
The rate the tokenizer, the speaker encoder and the mels read (fixed by the paper and release). |