VALLEX<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
VALL-E X: VALL-E trained on English and Mandarin with a language ID, which speaks a language in the voice of a prompt recorded in another.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VallEModelBase<T> → VALLEX<T>
Known derived types: VALLEXClone<T>
For Beginners
Record a few seconds of yourself in English, and VALL-E X can speak Chinese in your voice — or the other way round.
How It Works
Reference: “Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling” (Zhang et al.,
2023). Microsoft released no code; the runnable reproduction is Plachtaa/VALL-E-X, whose Mandarin front end
MandarinG2P reproduces. The networks are VALL-E’s (VallEModelBase).
Languages (§3.3): a language embedding added to the AR model’s acoustic-token embeddings — the prompt’s codes
in the prompt’s language, the generated ones in the text’s — guides the speaking style.
LanguagePlacement selects the reference’s placement instead (the phoneme embeddings of
both models).
Training (§3.2, §5.2): the AR model as VALL-E’s; the NAR model, at a uniformly drawn stage, reads the phonemes, an acoustic prompt from another sentence of the same speaker (all codebooks, Eq. 2) and the lower codebooks. Each model is optimized on its own.
Synthesis (§3.4): the prompt’s transcript and the text precede the prompt’s first-codebook codes in the AR
model (Eq. 3, sampled to the end token); the NAR reads the text alone and the prompt’s codes (Eq. 4, greedy);
EnCodec decodes. Text is phonemized by script: Chinese characters through MandarinG2P, the rest
through EnglishG2P, so mixed sentences work.
Training data: Tokens (phoneme ids,
EncodePhonemes), CodecTokens (or the recording),
PromptCodecTokens (another sentence of the same speaker) and
LanguageId (VallEXLanguage).
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional,
NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var vallex = new VALLEX<float>(architecture, new VALLEXOptions());
vallex.Voice = vallex.CreateVoice(englishPrompt24kHz, "the prompt's transcript");
var audio = vallex.Synthesize("你好,世界。");
Constructors
| Constructor | Summary |
|---|---|
VALLEX(NeuralNetworkArchitecture<T> architecture, VALLEXOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a native, trainable VALL-E X. |
VALLEX(NeuralNetworkArchitecture<T> architecture, string modelPath, VALLEXOptions options) |
Creates a native, trainable VALL-E X. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
int[] EncodePhonemes(IEnumerable<string> symbols) |