VALLE2<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
VALL-E 2: VALL-E with grouped code modeling and repetition-aware sampling, which predicts several first-codebook codes per autoregressive step and avoids the decoding loops of plain sampling.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VallEModelBase<T> → VALLE2<T>
For Beginners
VALL-E 2 writes speech tokens a few at a time instead of one by one, and when it notices itself repeating it switches to a more random choice, so it does not get stuck saying the same sound.
How It Works
Reference: “VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers” (Chen et al.,
2024). Microsoft released neither code nor a reproduction; what the paper leaves open is chosen in
VALLE2Options and the decision log.
Model (§3.2–3.3): the AR model reads the text, <eos> <bos> and one embedding per group of
G first-codebook codes, and predicts the next group’s G codes; the NAR model reads the text, an acoustic condition
(all eight codebooks), the lower codebooks of the target and a code-ID token, and predicts codebook j. Both use learned
positions and VALL-E’s Transformer layers (VallE2Core).
Training: the AR model on whole utterances (clipped at the start to whole groups, §3.1); the NAR model with the utterance’s first frames as its acoustic condition, at most half of it, otherwise a random 3–30 seconds (§4.1.1).
Synthesis (§3.4): the prompt’s transcript and the text, the prompt’s first codebook as the AR prefix (clipped to whole groups), repetition-aware sampling (Algorithm 1), the NAR greedy over the whole text with the prompt’s eight codebooks as condition, and Vocos’s EnCodec model decoding the new codes.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional,
NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var valle2 = new VALLE2<float>(architecture, new VALLE2Options());
valle2.Voice = valle2.CreateVoice(promptAudio24kHz, "the prompt's transcript");
var audio = valle2.Synthesize("Hello there.");
Constructors
| Constructor | Summary |
|---|---|
VALLE2(NeuralNetworkArchitecture<T> architecture, VALLE2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a native, trainable VALL-E 2. |
VALLE2(NeuralNetworkArchitecture<T> architecture, string modelPath, VALLE2Options options) |
Creates a native, trainable VALL-E 2. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
void LoadDecoderWeights(string path) |
Loads Vocos’s released EnCodec model (pytorch_model.bin of charactr/vocos-encodec-24khz, as .pt or safetensors). |