SeedTTS<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

Seed-TTS’s disclosed autoregressive pipeline: speech tokenizer, causal token LM, token diffusion, and acoustic vocoder.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → SeedTTS<T>

Implements: ICodecTts<T>

Known derived types: SeedTTSClone<T>

How It Works

This implementation follows Figure 1 and Section 2 of the technical report. The LM predicts speech tokens while masking text positions from its loss; diffusion restores continuous acoustic detail in a coarse-to-fine process; and a separately trainable neural vocoder produces the waveform. A reference recording supplies both the speech token prefix and the timbre condition used by the paper’s self-distillation extension.

ByteDance explicitly did not release model code or weights. The native graph therefore exposes all undisclosed dimensions as options and loads only clearly labelled AiDotNet-converted flat checkpoints; it does not claim that its configurable defaults are proprietary production settings.

Constructors

Constructor Summary
SeedTTS(NeuralNetworkArchitecture<T> architecture, SeedTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates the native four-block Seed-TTS graph.
SeedTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, SeedTTSOptions options) Creates the native four-block Seed-TTS graph.

Properties

Property Summary
SeedTTSTrainingComponent TrainingComponent { get; }
bool HasPaperPipeline { get; } Whether the native graph contains the four paper blocks rather than caller-supplied sequential layers.
bool TrainsOnSampleObjective { get; }
int CodebookSize { get; }
int CodecFrameRate { get; }
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int MaxTextLength { get; }
int NumCodebooks { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> DecodeFromTokens(Tensor<T> tokens)
Tensor<T> EncodeToTokens(Tensor<T> audio)
Tensor<T> ExtractTimbreEmbedding(Tensor<T> referenceAudio) Extracts the timbre reference integrated into the self-distilled diffusion model.
Tensor<T> Synthesize(string text) Synthesizes with the configured voice prompt when one is present, otherwise from the learned prior.
Tensor<T> SynthesizeWithPrompt(string text, string promptText, Tensor<T> referenceAudio) Performs the paper’s in-context continuation from a text/audio prompt.
void LoadConvertedWeights(string path) Copies {ConvertedParameterTensor} from source straight into a parameter vector.