E3TTS<T>

Models & Types · AiDotNet.TextToSpeech.FlowDiffusion

E3 TTS: BERT-conditioned, raw-waveform diffusion without acoustic intermediates or explicit alignment.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → E3TTS<T>

Implements: IEndToEndTts<T>

How It Works

This implements Gao et al.’s published model rather than a generic flow stack: uncased BERT-base, the exact ten DBlock/UBlock configuration in Table 1, adaptive softmax convolution kernels at high waveform resolutions, FiLM and self/cross-attention at low resolutions, learned timestep loss variance, and the paper’s 1000-step DDPM schedule. The authors published a paper and samples but no code or weights, so non-BERT weights use an explicit AiDotNet conversion schema documented by LoadConvertedWeights.

Constructors

Constructor Summary
E3TTS(NeuralNetworkArchitecture<T> architecture, E3TTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
E3TTS(NeuralNetworkArchitecture<T> architecture, string modelPath, E3TTSOptions options)

Properties

Property Summary
bool TrainsOnSampleObjective { get; }
int HiddenDim { get; }
int MaxTextLength { get; }
int NumFlowSteps { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> Edit(string text, Tensor<T> waveform, int startSample, int endSample, int speakerId) Re-synthesizes only startSample..endSample while the DDPM trajectory keeps the samples outside that span fixed, matching the paper’s text-based speech editing procedure.
Tensor<T> Synthesize(string text)
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate, string transcript) Builds the waveform prompt used by the paper’s zero-shot prompt synthesis experiment.
double AlphaBar(int step) Equation 7’s cumulative inference noise schedule; step zero is the clean endpoint.
double TrainingAlphaBar(int step) Cumulative product of the linear training betas in Eq.
int[] TokenizeText(string text) Tokenizes plain text as BERT-base-uncased subwords, including [CLS] and [SEP].
void LoadBertWeights(string path) Loads a Hugging Face/Google bert-base-uncased checkpoint into E3’s text encoder.
void LoadCheckpoints(E3TTSCheckpointPaths checkpoints)
void LoadConvertedWeights(string path) Loads E3 U-Net weights converted to AiDotNet’s documented module names.