FishSpeech<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

Fish Speech 1.4: the Dual-AR transformer and Firefly-GAN of FishSpeechV15 (Liao et al. 2024) in the 1.4 release’s format — a 32,000-token byte-level BPE vocabulary and one <|semantic|> token per frame whose eight codes are stored plus one (zero pads text positions).

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → FishSpeechBase<T> → FishSpeech<T>

For Beginners

This is the earlier Fish Speech release. It speaks text in the voice of a short reference recording, one frame of audio codes at a time.

How It Works

The release (fish-speech v1.4.3 and the fishaudio/fish-speech-1.4 checkpoint) defines everything the paper leaves open: the prompt is <|im_start|>user\n{text}<|im_end|><|im_start|>assistant\n with no system message, the training sample scores the token row throughout and the codes from the first frame, text is cleaned more aggressively (brackets, dashes and newlines dropped) and, as the 1.4 API does by default, Chinese numbers are written out first. Training follows the release’s fine-tuning recipe by default.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Generative, inputSize: 1, outputSize: 1);
var model = new FishSpeech<float>(architecture, new FishSpeechOptions());
model.LoadReferenceWeights("fish-speech-1.4");      // model.pth, the Firefly-GAN generator, tokenizer.json
model.Voice = model.CreateVoice(reference, referenceRate, "What the reference says.");
var speech = model.Synthesize("Text to speak in that voice.");   // 44.1 kHz

Constructors

Constructor Summary
FishSpeech(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
FishSpeech(NeuralNetworkArchitecture<T> architecture, FishSpeechOptions options) Creates a native, trainable Fish Speech 1.4.
FishSpeech(NeuralNetworkArchitecture<T> architecture, string modelPath, FishSpeechOptions options) Creates an ONNX-backed Fish Speech 1.4 for inference.