E3TTS<T>
Models & Types · AiDotNet.TextToSpeech.FlowDiffusion
E3 TTS: BERT-conditioned, raw-waveform diffusion without acoustic intermediates or explicit alignment.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → E3TTS<T>
Implements: IEndToEndTts<T>
How It Works
This implements Gao et al.’s published model rather than a generic flow stack: uncased BERT-base, the exact ten
DBlock/UBlock configuration in Table 1, adaptive softmax convolution kernels at high waveform resolutions, FiLM and
self/cross-attention at low resolutions, learned timestep loss variance, and the paper’s 1000-step DDPM schedule.
The authors published a paper and samples but no code or weights, so non-BERT weights use an explicit AiDotNet
conversion schema documented by LoadConvertedWeights.
Constructors
| Constructor | Summary |
|---|---|
E3TTS(NeuralNetworkArchitecture<T> architecture, E3TTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
E3TTS(NeuralNetworkArchitecture<T> architecture, string modelPath, E3TTSOptions options) |
Properties
| Property | Summary |
|---|---|
bool TrainsOnSampleObjective { get; } |
|
int HiddenDim { get; } |
|
int MaxTextLength { get; } |
|
int NumFlowSteps { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> Edit(string text, Tensor<T> waveform, int startSample, int endSample, int speakerId) |
Re-synthesizes only startSample..endSample while the DDPM trajectory keeps the samples outside that span fixed, matching the paper’s text-based speech editing procedure. |
Tensor<T> Synthesize(string text) |
|
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate, string transcript) |
Builds the waveform prompt used by the paper’s zero-shot prompt synthesis experiment. |
double AlphaBar(int step) |
Equation 7’s cumulative inference noise schedule; step zero is the clean endpoint. |
double TrainingAlphaBar(int step) |
Cumulative product of the linear training betas in Eq. |
int[] TokenizeText(string text) |
Tokenizes plain text as BERT-base-uncased subwords, including [CLS] and [SEP]. |
void LoadBertWeights(string path) |
Loads a Hugging Face/Google bert-base-uncased checkpoint into E3’s text encoder. |
void LoadCheckpoints(E3TTSCheckpointPaths checkpoints) |
|
void LoadConvertedWeights(string path) |
Loads E3 U-Net weights converted to AiDotNet’s documented module names. |