VALLEX<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

VALL-E X: VALL-E trained on English and Mandarin with a language ID, which speaks a language in the voice of a prompt recorded in another.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VallEModelBase<T> → VALLEX<T>

Known derived types: VALLEXClone<T>

For Beginners

Record a few seconds of yourself in English, and VALL-E X can speak Chinese in your voice — or the other way round.

How It Works

Reference: “Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling” (Zhang et al., 2023). Microsoft released no code; the runnable reproduction is Plachtaa/VALL-E-X, whose Mandarin front end MandarinG2P reproduces. The networks are VALL-E’s (VallEModelBase).

Languages (§3.3): a language embedding added to the AR model’s acoustic-token embeddings — the prompt’s codes in the prompt’s language, the generated ones in the text’s — guides the speaking style. LanguagePlacement selects the reference’s placement instead (the phoneme embeddings of both models).

Training (§3.2, §5.2): the AR model as VALL-E’s; the NAR model, at a uniformly drawn stage, reads the phonemes, an acoustic prompt from another sentence of the same speaker (all codebooks, Eq. 2) and the lower codebooks. Each model is optimized on its own.

Synthesis (§3.4): the prompt’s transcript and the text precede the prompt’s first-codebook codes in the AR model (Eq. 3, sampled to the end token); the NAR reads the text alone and the prompt’s codes (Eq. 4, greedy); EnCodec decodes. Text is phonemized by script: Chinese characters through MandarinG2P, the rest through EnglishG2P, so mixed sentences work.

Training data: Tokens (phoneme ids, EncodePhonemes), CodecTokens (or the recording), PromptCodecTokens (another sentence of the same speaker) and LanguageId (VallEXLanguage).

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional,
    NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var vallex = new VALLEX<float>(architecture, new VALLEXOptions());
vallex.Voice = vallex.CreateVoice(englishPrompt24kHz, "the prompt's transcript");
var audio = vallex.Synthesize("你好,世界。");

Constructors

Constructor Summary
VALLEX(NeuralNetworkArchitecture<T> architecture, VALLEXOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a native, trainable VALL-E X.
VALLEX(NeuralNetworkArchitecture<T> architecture, string modelPath, VALLEXOptions options) Creates a native, trainable VALL-E X.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
int[] EncodePhonemes(IEnumerable<string> symbols)