FishSpeechBase<T>
Base Classes · AiDotNet.TextToSpeech.CodecBased
The Fish Speech models (Liao et al. 2024): a Dual-AR transformer that speaks Firefly-GAN’s codes after the text, and
the codec that turns them into 44.1 kHz audio. The releases differ in their tokenizer, in how a frame’s codes appear in
the sequence (1.5: the frame’s <|semantic:c0|> token; 1.4: one <|semantic|> token, codes plus
one), in the system prompt (1.5’s “Speak out the provided text.”) and in their text cleaning.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → FishSpeechBase<T>
Implements: ICodecTts<T>, IPaperOptimizerVariant
Known derived types: FishSpeechV15<T>, FishSpeech<T>
How It Works
Synthesize runs the release’s generate_long: the text cleaned and split into segments of at
most ChunkLength bytes, each generated after the references and the earlier
segments that fit the context, its codes decoded by Firefly-GAN and the audio concatenated. A voice from
CreateVoice clones the speaker of one or more transcribed recordings.
Without a tokenizer file (TokenizerPath or
LoadReferenceWeights’s directory) the model reads text as UTF-8 bytes, one token each, with the release’s
special tokens beside them, a vocabulary for training from scratch.
Properties
| Property | Summary |
|---|---|
bool TrainsOnSampleObjective { get; } |
|
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
|
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
|
string PaperOptimizerVariant { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<string> SplitText(string text) |
The segments Synthesize speaks in turn (split_text, after the Chinese number normalizer when NormalizeChineseText is set). |
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
T TrainPreference(TtsTrainingSample<T> chosen, TtsTrainingSample<T> rejected) |
One step of standard DPO on a preference pair for the same text: -log σ(β((log π(chosen) − log π_ref(chosen)) − (log π(rejected) − log π_ref(rejected)))) over each sequence’s scored tokens and codes, β being DpoBeta. |
T TrainPreferenceReferenceFree(TtsTrainingSample<T> chosen, TtsTrainingSample<T> rejected) |
One step of the release’s reference-free preference objective (git 8093258): the chosen sample’s cross-entropy plus -log σ(β(log π(chosen codes) − log π(rejected codes))) averaged over positions, β being ReferenceFreeDpoBeta. |
Tensor<T> DecodeAudio(int[] codes) |
The audio Firefly-GAN decodes from codes [codebooks, frames]. |
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
|
Tensor<T> EncodeToTokens(Tensor<T> audio) |
|
Tensor<T> Synthesize(string text) |
|
TtsVoice<T> CreateVoice(IEnumerable<ValueTuple<Tensor<T>, int, string>> references) |
A voice cloned from several transcribed recordings of the speaker (the release’s references: each one’s text and codes precede the text to speak). |
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate, string transcript) |
A voice cloned from a recording of the speaker at sampleRate and what it says. |
int[] EncodeAudio(Tensor<T> audio) |
Firefly-GAN’s codes [codebooks, frames] of audio at the model’s rate. |
int[] Tokenize(string text) |
The token ids of text as the user’s turn reads it (cleaned, then tokenized). |
string CleanText(string text) |
The release’s clean_text for this version. |
string Detokenize(IEnumerable<int> ids) |
Decodes token ids to text. |
void FreezePreferenceReference() |
Takes the current transformer as standard DPO’s frozen reference (Rafailov et al. |
void LoadCodec(string path) |
Loads Firefly-GAN from the release’s generator checkpoint. |
void LoadReferenceWeights(string directory) |
Loads a release’s directory (fishaudio/fish-speech-1.5 or -1.4): the Dual-AR transformer (model.pth), Firefly-GAN (CodecCheckpointName) and the tokenizer (1.5: tokenizer.tiktoken; 1.4: tokenizer.json), unless TokenizerPath nam… |
void LoadTransformer(string path) |
Loads the Dual-AR transformer from the release’s model.pth (or any checkpoint with its names). |
Fields
| Field | Summary |
|---|---|
string CodecCheckpointName |
The release’s codec checkpoint file name in the model directory. |