DiTToTTS<T>
Models & Types · AiDotNet.TextToSpeech.FlowDiffusion
DiTTo-TTS: scalable zero-shot text-to-speech with a diffusion Transformer in CLaM-TTS Mel-VAE space.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → DiTToTTS<T>
Implements: IEndToEndTts<T>
How It Works
This is the paper pipeline rather than the former generic layer stack: frozen SpeechT5 text features, a separately
trained causal length predictor, 9-way global AdaLN DiT with v-prediction and classifier-free guidance, the frozen
CLaM-TTS Mel-VAE/RVQ, and BigVGAN. The authors did not release code or checkpoints; native checkpoint loading therefore
uses the explicit AiDotNet conversion names documented by LoadConvertedWeights.
Constructors
| Constructor | Summary |
|---|---|
DiTToTTS(NeuralNetworkArchitecture<T> architecture, DiTToTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
DiTToTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, DiTToTTSOptions options) |
Properties
| Property | Summary |
|---|---|
DiTToTrainingStage CurrentStage { get; set; } |
|
bool TrainsOnSampleObjective { get; } |
|
int HiddenDim { get; } |
|
int MaxTextLength { get; } |
|
int NumFlowSteps { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> ComputeMel(Tensor<T> audio) |
|
Tensor<T> Synthesize(string text) |
|
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate, string transcript) |
Builds an optional zero-shot acoustic prompt. |
int[] TokenizeText(string text) |
|
void LoadCodecSemanticDecoder(string checkpoint) |
Loads the frozen ByT5 decoder used only by Mel-VAE++ semantic codec finetuning. |
void LoadConvertedWeights(DiTToCheckpointPaths checkpoints) |
Loads the public SpeechT5/BigVGAN state and AiDotNet-converted DiTTo/CLaM tensors. |