SparkTTS<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

Spark-TTS: Qwen2.5 generates a single semantic speech-token stream which BiCodec renders with decoupled speaker tokens.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → SparkTTS<T>

Implements: ICodecTts<T>

How It Works

The native graph follows the official 0.5B release: XLSR hidden-state mixing, a Vocos semantic encoder and 8,192-way factorized VQ, an ECAPA/Perceiver/FSQ speaker tokenizer, Qwen2.5-0.5B, a d-vector-conditioned Vocos prenet, and the released four-stage DAC WaveGenerator. Voice cloning uses the official 32-global-token prompt grammar; controllable synthesis asks the LM to generate those global tokens from gender, pitch and speed labels.

Constructors

Constructor Summary
SparkTTS(NeuralNetworkArchitecture<T> architecture, SparkTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates the native official-release graph.
SparkTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, SparkTTSOptions options) Creates the native official-release graph.

Properties

Property Summary
bool HasPaperPipeline { get; } Whether this instance owns the official Qwen/BiCodec component graph.
bool TrainsOnSampleObjective { get; }
int CodebookSize { get; }
int CodecFrameRate { get; }
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int MaxTextLength { get; }
int NumCodebooks { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
T TrainAnnotatedSample(TtsTrainingSample<T> sample, bool female, SparkTTSControlLevel pitch, SparkTTSControlLevel speed, int pitchMel, int syllablesPerSecond) Trains both zero-shot and controllable samples for one annotated recording (paper Eq.
Tensor<T> DecodeFromTokens(Tensor<T> semanticTokens, Tensor<T> globalTokens) Decodes semantic speech tokens with an explicit 32-token speaker prompt.
Tensor<T> DecodeFromTokens(Tensor<T> tokens)
Tensor<T> EncodeGlobalTokens(Tensor<T> referenceAudio) Extracts the 32 released BiCodec global speaker-token ids.
Tensor<T> EncodeToTokens(Tensor<T> audio)
Tensor<T> Synthesize(string text) Synthesizes with a configured voice prompt, or with the paper’s neutral controllable-speaker path.
Tensor<T> SynthesizeControlled(string text, bool female, SparkTTSControlLevel pitch, SparkTTSControlLevel speed) Runs the official gender/pitch/speed controllable-speaker prompt and its global-token chain of thought.
Tensor<T> SynthesizeWithPrompt(string text, string promptText, Tensor<T> referenceAudio) Runs the released zero-shot voice-cloning prompt format.
void LoadAuthorWeights(string languageModelPath, string biCodecPath, string wav2VecPath) Loads all three original release components.
void LoadConvertedWeights(string path) Copies {ConvertedParameterTensor} from source straight into a parameter vector.
void LoadLanguageModelWeights(string path) Loads the original Hugging Face Qwen2.5 state dictionary, including its tied output embedding.
void LoadSpeakerTokenizerWeights(string path, string prefix) Loads the speaker_encoder subtree of the released BiCodec checkpoint, including ECAPA running statistics.