VallEModelBase<T>
Base Classes · AiDotNet.TextToSpeech.CodecBased
What the models of the VALL-E family share: EnCodec, the AR and NAR codec language models (VallECore),
a phoneme vocabulary, training each model on its own, and prompted synthesis.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VallEModelBase<T>
Implements: ICodecTts<T>
Known derived types: VALLE2<T>, VALLEX<T>, VALLE<T>
How It Works
A model supplies its phoneme table and front end, how its NAR model is prompted in training, and (for a multilingual
model) its languages; this base owns the codec, the vocabulary [<pad>, <bos>, <eos>] + sorted(table) (the reference’s TextTokenCollater), the optimizers, synthesis and the weights.
Properties
| Property | Summary |
|---|---|
EnCodec<T> Codec { get; } |
The codec whose codes the model predicts. |
VallETrainingStage CurrentStage { get; set; } |
The network a training step updates. |
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
|
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
|
Tensor<T> EncodeToTokens(Tensor<T> audio) |
|
TtsVoice<T> CreateVoice(Tensor<T> recording, string transcript) |
A voice for Voice: a prompt recording at the model’s sample rate (about 3 seconds) and its transcript. |
int[] EncodePhonemes(IEnumerable<string> symbols) |
The token id of each phoneme symbol, without the <bos> … <eos> frame. |
void LoadReferenceWeights(string path) |
Loads a reference state dictionary (safetensors, or a .pt of the checkpoint’s model entry). |
void Train(Tensor<T> input, Tensor<T> expectedOutput) |