Vevo<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
Vevo (Zhang et al., ICLR 2025): controllable zero-shot voice imitation by self-supervised disentanglement — HuBERT’s features quantized into content tokens (32 codes, timbre and style squeezed out) and content-style tokens (8,192 codes, timbre squeezed out); an autoregressive transformer that turns content tokens (Vevo-Voice, Vevo-Style) or phones (Vevo-TTS) into content-style tokens in a reference’s style; and a flow-matching transformer that turns content-style tokens into a mel in a reference’s timbre (Vevo-Timbre), voiced by Vocos.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → Vevo<T>
For Beginners
Vevo separates what is said, how it is said (accent, emotion) and who says it. Give it text and a recording to read the text in that voice, or give it a recording to change its voice, its speaking style, or only its timbre.
How It Works
The paper leaves the sizes to the release (Amphion models/vc/vevo, the amphion/Vevo checkpoints),
whose code fills the gaps. Synthesize is Vevo-TTS: the voice’s recording gives the style (with its
transcript’s phones as the prompt) and, unless TimbreReference is set, the timbre.
ConvertVoice, ConvertStyle and ConvertTimbre are the voice-conversion
tasks.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new Vevo<float>(architecture, new VevoOptions());
model.LoadReferenceWeights(new VevoCheckpoints { Directory = "Vevo", Hubert = "hubert-large-ll60k/pytorch_model.bin" });
model.Voice = model.CreateVoice(reference, referenceRate, "What the reference says.", "en");
var speech = model.Synthesize("Text to speak in that voice."); // 24 kHz
var converted = model.ConvertVoice(source, sourceRate, reference, referenceRate);
Constructors
| Constructor | Summary |
|---|---|
Vevo(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
Vevo(NeuralNetworkArchitecture<T> architecture, VevoOptions options) |
Creates a native, trainable Vevo. |
Vevo(NeuralNetworkArchitecture<T> architecture, string modelPath, VevoOptions options) |
Creates an ONNX-backed Vevo for inference. |
Properties
| Property | Summary |
|---|---|
Tensor<T> TimbreReference { get; set; } |
A recording whose timbre synthesis takes (at SampleRate); null for the voice’s own. |
bool TrainsOnSampleObjective { get; } |
|
int MaxTextLength { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> ConvertStyle(Tensor<T> source, int sourceRate, Tensor<T> styleReference, int styleRate) |
Vevo-Style: source in styleReference’s style (accent, emotion), keeping its own timbre; the result is at 24 kHz. |
Tensor<T> ConvertTimbre(Tensor<T> source, int sourceRate, Tensor<T> timbreReference, int timbreRate) |
Vevo-Timbre: source in timbreReference’s timbre, keeping its content and style (the flow alone, inference_fm); the result is at 24 kHz. |
Tensor<T> ConvertVoice(Tensor<T> source, int sourceRate, Tensor<T> reference, int referenceRate) |
Vevo-Voice: source spoken in reference’s voice — its timbre and its style (accent, emotion) — both at their own rates; the result is at 24 kHz. |
Tensor<T> Synthesize(string text) |
|
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate, string transcript, string language) |
A voice from a recording of the speaker at sampleRate and what it says: the recording gives the style (and, unless TimbreReference is set, the timbre); its transcript’s phones in language prompt the text-to-content-style transformer. |
ValueTuple<string, int[]> Phonemize(string text, string language) |
Amphion’s g2p_: the phone string and the phone ids of text in language. |
void LoadReferenceWeights(VevoCheckpoints checkpoints) |
Loads the release: HuBERT-Large (its first HubertLayer layers), the content-style tokenizer with HuBERT’s feature statistics, the autoregressive transformers, the content tokenizer (when voice conversion is on), the flow-matching transfor… |