VallEModelBase<T>

Base Classes · AiDotNet.TextToSpeech.CodecBased

What the models of the VALL-E family share: EnCodec, the AR and NAR codec language models (VallECore), a phoneme vocabulary, training each model on its own, and prompted synthesis.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VallEModelBase<T>

Implements: ICodecTts<T>

Known derived types: VALLE2<T>, VALLEX<T>, VALLE<T>

How It Works

A model supplies its phoneme table and front end, how its NAR model is prompted in training, and (for a multilingual model) its languages; this base owns the codec, the vocabulary [<pad>, <bos>, <eos>] + sorted(table) (the reference’s TextTokenCollater), the optimizers, synthesis and the weights.

Properties

Property Summary
EnCodec<T> Codec { get; } The codec whose codes the model predicts.
VallETrainingStage CurrentStage { get; set; } The network a training step updates.
int CodebookSize { get; }
int CodecFrameRate { get; }
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int MaxTextLength { get; }
int NumCodebooks { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> DecodeFromTokens(Tensor<T> tokens)
Tensor<T> EncodeToTokens(Tensor<T> audio)
TtsVoice<T> CreateVoice(Tensor<T> recording, string transcript) A voice for Voice: a prompt recording at the model’s sample rate (about 3 seconds) and its transcript.
int[] EncodePhonemes(IEnumerable<string> symbols) The token id of each phoneme symbol, without the <bos> … <eos> frame.
void LoadReferenceWeights(string path) Loads a reference state dictionary (safetensors, or a .pt of the checkpoint’s model entry).
void Train(Tensor<T> input, Tensor<T> expectedOutput)