FishSpeechV15<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
Fish Speech 1.5 (Liao et al. 2024, “Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis”): a Dual-AR transformer — a slow transformer over text and semantic tokens, a fast transformer over each frame’s eight codebooks — speaking Firefly-GAN’s grouped-FSQ codes, which Firefly-GAN turns into 44.1 kHz audio.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → FishSpeechBase<T> → FishSpeechV15<T>
For Beginners
Fish Speech reads text and writes speech one short frame at a time, like a language model writing words. Give it a few seconds of someone speaking (and what they said) and it speaks new text in that voice.
How It Works
The paper leaves the sizes and the sequence format to the release (fish-speech v1.5.1 and the
fishaudio/fish-speech-1.5 checkpoint), whose code fills the gaps: each frame is the token
<|semantic:c0|> with all eight codes’ embeddings added, prompts open with the system message “Speak out the
provided text.” when references are given, and the training sample is that conversation with the speech scored from
<|voice|> on. Training follows the paper’s recipe by default; DPO, which the paper names as its last stage
without details, is standard DPO (TrainPreference), and the release’s reference-free
variant is TrainPreferenceReferenceFree.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Generative, inputSize: 1, outputSize: 1);
var model = new FishSpeechV15<float>(architecture, new FishSpeechV15Options());
model.LoadReferenceWeights("fish-speech-1.5"); // model.pth, the Firefly-GAN generator, tokenizer.tiktoken
model.Voice = model.CreateVoice(reference, referenceRate, "What the reference says.");
var speech = model.Synthesize("Text to speak in that voice."); // 44.1 kHz
Constructors
| Constructor | Summary |
|---|---|
FishSpeechV15(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
FishSpeechV15(NeuralNetworkArchitecture<T> architecture, FishSpeechV15Options options) |
Creates a native, trainable Fish Speech 1.5. |
FishSpeechV15(NeuralNetworkArchitecture<T> architecture, string modelPath, FishSpeechV15Options options) |
Creates an ONNX-backed Fish Speech 1.5 for inference. |