DiTToTTS<T>

Models & Types · AiDotNet.TextToSpeech.FlowDiffusion

DiTTo-TTS: scalable zero-shot text-to-speech with a diffusion Transformer in CLaM-TTS Mel-VAE space.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → DiTToTTS<T>

Implements: IEndToEndTts<T>

How It Works

This is the paper pipeline rather than the former generic layer stack: frozen SpeechT5 text features, a separately trained causal length predictor, 9-way global AdaLN DiT with v-prediction and classifier-free guidance, the frozen CLaM-TTS Mel-VAE/RVQ, and BigVGAN. The authors did not release code or checkpoints; native checkpoint loading therefore uses the explicit AiDotNet conversion names documented by LoadConvertedWeights.

Constructors

Constructor Summary
DiTToTTS(NeuralNetworkArchitecture<T> architecture, DiTToTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
DiTToTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, DiTToTTSOptions options)

Properties

Property Summary
DiTToTrainingStage CurrentStage { get; set; }
bool TrainsOnSampleObjective { get; }
int HiddenDim { get; }
int MaxTextLength { get; }
int NumFlowSteps { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> ComputeMel(Tensor<T> audio)
Tensor<T> Synthesize(string text)
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate, string transcript) Builds an optional zero-shot acoustic prompt.
int[] TokenizeText(string text)
void LoadCodecSemanticDecoder(string checkpoint) Loads the frozen ByT5 decoder used only by Mel-VAE++ semantic codec finetuning.
void LoadConvertedWeights(DiTToCheckpointPaths checkpoints) Loads the public SpeechT5/BigVGAN state and AiDotNet-converted DiTTo/CLaM tensors.