FishSpeechBase<T>

Base Classes · AiDotNet.TextToSpeech.CodecBased

The Fish Speech models (Liao et al. 2024): a Dual-AR transformer that speaks Firefly-GAN’s codes after the text, and the codec that turns them into 44.1 kHz audio. The releases differ in their tokenizer, in how a frame’s codes appear in the sequence (1.5: the frame’s <|semantic:c0|> token; 1.4: one <|semantic|> token, codes plus one), in the system prompt (1.5’s “Speak out the provided text.”) and in their text cleaning.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → FishSpeechBase<T>

Implements: ICodecTts<T>, IPaperOptimizerVariant

Known derived types: FishSpeechV15<T>, FishSpeech<T>

How It Works

Synthesize runs the release’s generate_long: the text cleaned and split into segments of at most ChunkLength bytes, each generated after the references and the earlier segments that fit the context, its codes decoded by Firefly-GAN and the audio concatenated. A voice from CreateVoice clones the speaker of one or more transcribed recordings.

Without a tokenizer file (TokenizerPath or LoadReferenceWeights’s directory) the model reads text as UTF-8 bytes, one token each, with the release’s special tokens beside them, a vocabulary for training from scratch.

Properties

Property Summary
bool TrainsOnSampleObjective { get; }
int CodebookSize { get; }
int CodecFrameRate { get; }
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int MaxTextLength { get; }
int NumCodebooks { get; }
string PaperOptimizerVariant { get; }

Methods

Method Summary
IReadOnlyList<string> SplitText(string text) The segments Synthesize speaks in turn (split_text, after the Chinese number normalizer when NormalizeChineseText is set).
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
T TrainPreference(TtsTrainingSample<T> chosen, TtsTrainingSample<T> rejected) One step of standard DPO on a preference pair for the same text: -log σ(β((log π(chosen) − log π_ref(chosen)) − (log π(rejected) − log π_ref(rejected)))) over each sequence’s scored tokens and codes, β being DpoBeta.
T TrainPreferenceReferenceFree(TtsTrainingSample<T> chosen, TtsTrainingSample<T> rejected) One step of the release’s reference-free preference objective (git 8093258): the chosen sample’s cross-entropy plus -log σ(β(log π(chosen codes) − log π(rejected codes))) averaged over positions, β being ReferenceFreeDpoBeta.
Tensor<T> DecodeAudio(int[] codes) The audio Firefly-GAN decodes from codes [codebooks, frames].
Tensor<T> DecodeFromTokens(Tensor<T> tokens)
Tensor<T> EncodeToTokens(Tensor<T> audio)
Tensor<T> Synthesize(string text)
TtsVoice<T> CreateVoice(IEnumerable<ValueTuple<Tensor<T>, int, string>> references) A voice cloned from several transcribed recordings of the speaker (the release’s references: each one’s text and codes precede the text to speak).
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate, string transcript) A voice cloned from a recording of the speaker at sampleRate and what it says.
int[] EncodeAudio(Tensor<T> audio) Firefly-GAN’s codes [codebooks, frames] of audio at the model’s rate.
int[] Tokenize(string text) The token ids of text as the user’s turn reads it (cleaned, then tokenized).
string CleanText(string text) The release’s clean_text for this version.
string Detokenize(IEnumerable<int> ids) Decodes token ids to text.
void FreezePreferenceReference() Takes the current transformer as standard DPO’s frozen reference (Rafailov et al.
void LoadCodec(string path) Loads Firefly-GAN from the release’s generator checkpoint.
void LoadReferenceWeights(string directory) Loads a release’s directory (fishaudio/fish-speech-1.5 or -1.4): the Dual-AR transformer (model.pth), Firefly-GAN (CodecCheckpointName) and the tokenizer (1.5: tokenizer.tiktoken; 1.4: tokenizer.json), unless TokenizerPath nam…
void LoadTransformer(string path) Loads the Dual-AR transformer from the release’s model.pth (or any checkpoint with its names).

Fields

Field Summary
string CodecCheckpointName The release’s codec checkpoint file name in the model directory.