VALLE<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

VALL-E: a neural codec language model for zero-shot text-to-speech, which continues a 3-second recording of an unseen speaker by predicting EnCodec codes from phonemes.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VallEModelBase<T> → VALLE<T>

For Beginners

VALL-E treats speech as text-like tokens. Given a few seconds of someone’s voice and a sentence, it writes the tokens of that person saying the sentence, and a codec turns them into audio.

How It Works

Reference: “Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers” (Wang et al., 2023). Microsoft released no code; what the paper leaves open follows the reproduction lifeiteng/vall-e, against which this model is tested (see VALLEOptions).

Model (§4): EnCodec turns 24 kHz speech into eight codebooks of codes at 75 frames a second. An autoregressive Transformer predicts the first codebook from the phonemes and the codes before it; a non-autoregressive Transformer, told the stage through adaptive layer norm, predicts each later codebook from the phonemes, the codebooks below it and an acoustic prompt — in training a random 3-second segment of the same utterance (§5.1). Each is trained on its own (CurrentStage).

Synthesis (§4.3, “VALL-E”): the prompt’s transcript precedes the text, the prompt’s first-codebook codes start the AR decoding (sampling until the end token, or 16 codes per phoneme token), the NAR fills the other codebooks greedily after the prompt (its text without the prompt’s transcript, as the reference does), and EnCodec decodes the new frames. A voice without a transcript is an empty enrolled transcript.

Training data: Tokens are the phoneme ids (EncodePhonemes), CodecTokens the EnCodec codes [frames, 8] (or Audio at 24 kHz, which the model encodes). The paper crops each utterance to a random 10–20 seconds together with its aligned phonemes; that needs the alignment, so it is the caller’s.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional,
    NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var valle = new VALLE<float>(architecture, new VALLEOptions());
valle.Voice = valle.CreateVoice(promptAudio24kHz, "the prompt's transcript");
var audio = valle.Synthesize("Hello there.");

Constructors

Constructor Summary
VALLE(NeuralNetworkArchitecture<T> architecture, VALLEOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a native, trainable VALL-E.
VALLE(NeuralNetworkArchitecture<T> architecture, string modelPath, VALLEOptions options) Creates a native, trainable VALL-E.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()