FishSpeech<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
Fish Speech 1.4: the Dual-AR transformer and Firefly-GAN of FishSpeechV15 (Liao et al. 2024) in the
1.4 release’s format — a 32,000-token byte-level BPE vocabulary and one <|semantic|> token per frame
whose eight codes are stored plus one (zero pads text positions).
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → FishSpeechBase<T> → FishSpeech<T>
For Beginners
This is the earlier Fish Speech release. It speaks text in the voice of a short reference recording, one frame of audio codes at a time.
How It Works
The release (fish-speech v1.4.3 and the fishaudio/fish-speech-1.4 checkpoint) defines everything the
paper leaves open: the prompt is <|im_start|>user\n{text}<|im_end|><|im_start|>assistant\n with no
system message, the training sample scores the token row throughout and the codes from the first frame, text is cleaned
more aggressively (brackets, dashes and newlines dropped) and, as the 1.4 API does by default, Chinese numbers are
written out first. Training follows the release’s fine-tuning recipe by default.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Generative, inputSize: 1, outputSize: 1);
var model = new FishSpeech<float>(architecture, new FishSpeechOptions());
model.LoadReferenceWeights("fish-speech-1.4"); // model.pth, the Firefly-GAN generator, tokenizer.json
model.Voice = model.CreateVoice(reference, referenceRate, "What the reference says.");
var speech = model.Synthesize("Text to speak in that voice."); // 44.1 kHz
Constructors
| Constructor | Summary |
|---|---|
FishSpeech(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
FishSpeech(NeuralNetworkArchitecture<T> architecture, FishSpeechOptions options) |
Creates a native, trainable Fish Speech 1.4. |
FishSpeech(NeuralNetworkArchitecture<T> architecture, string modelPath, FishSpeechOptions options) |
Creates an ONNX-backed Fish Speech 1.4 for inference. |