FishSpeechV15<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

Fish Speech 1.5 (Liao et al. 2024, “Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis”): a Dual-AR transformer — a slow transformer over text and semantic tokens, a fast transformer over each frame’s eight codebooks — speaking Firefly-GAN’s grouped-FSQ codes, which Firefly-GAN turns into 44.1 kHz audio.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → FishSpeechBase<T> → FishSpeechV15<T>

For Beginners

Fish Speech reads text and writes speech one short frame at a time, like a language model writing words. Give it a few seconds of someone speaking (and what they said) and it speaks new text in that voice.

How It Works

The paper leaves the sizes and the sequence format to the release (fish-speech v1.5.1 and the fishaudio/fish-speech-1.5 checkpoint), whose code fills the gaps: each frame is the token <|semantic:c0|> with all eight codes’ embeddings added, prompts open with the system message “Speak out the provided text.” when references are given, and the training sample is that conversation with the speech scored from <|voice|> on. Training follows the paper’s recipe by default; DPO, which the paper names as its last stage without details, is standard DPO (TrainPreference), and the release’s reference-free variant is TrainPreferenceReferenceFree.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Generative, inputSize: 1, outputSize: 1);
var model = new FishSpeechV15<float>(architecture, new FishSpeechV15Options());
model.LoadReferenceWeights("fish-speech-1.5");      // model.pth, the Firefly-GAN generator, tokenizer.tiktoken
model.Voice = model.CreateVoice(reference, referenceRate, "What the reference says.");
var speech = model.Synthesize("Text to speak in that voice.");   // 44.1 kHz

Constructors

Constructor Summary
FishSpeechV15(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
FishSpeechV15(NeuralNetworkArchitecture<T> architecture, FishSpeechV15Options options) Creates a native, trainable Fish Speech 1.5.
FishSpeechV15(NeuralNetworkArchitecture<T> architecture, string modelPath, FishSpeechV15Options options) Creates an ONNX-backed Fish Speech 1.5 for inference.