Vevo<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

Vevo (Zhang et al., ICLR 2025): controllable zero-shot voice imitation by self-supervised disentanglement — HuBERT’s features quantized into content tokens (32 codes, timbre and style squeezed out) and content-style tokens (8,192 codes, timbre squeezed out); an autoregressive transformer that turns content tokens (Vevo-Voice, Vevo-Style) or phones (Vevo-TTS) into content-style tokens in a reference’s style; and a flow-matching transformer that turns content-style tokens into a mel in a reference’s timbre (Vevo-Timbre), voiced by Vocos.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → Vevo<T>

For Beginners

Vevo separates what is said, how it is said (accent, emotion) and who says it. Give it text and a recording to read the text in that voice, or give it a recording to change its voice, its speaking style, or only its timbre.

How It Works

The paper leaves the sizes to the release (Amphion models/vc/vevo, the amphion/Vevo checkpoints), whose code fills the gaps. Synthesize is Vevo-TTS: the voice’s recording gives the style (with its transcript’s phones as the prompt) and, unless TimbreReference is set, the timbre. ConvertVoice, ConvertStyle and ConvertTimbre are the voice-conversion tasks.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new Vevo<float>(architecture, new VevoOptions());
model.LoadReferenceWeights(new VevoCheckpoints { Directory = "Vevo", Hubert = "hubert-large-ll60k/pytorch_model.bin" });
model.Voice = model.CreateVoice(reference, referenceRate, "What the reference says.", "en");
var speech = model.Synthesize("Text to speak in that voice.");          // 24 kHz
var converted = model.ConvertVoice(source, sourceRate, reference, referenceRate);

Constructors

Constructor Summary
Vevo(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
Vevo(NeuralNetworkArchitecture<T> architecture, VevoOptions options) Creates a native, trainable Vevo.
Vevo(NeuralNetworkArchitecture<T> architecture, string modelPath, VevoOptions options) Creates an ONNX-backed Vevo for inference.

Properties

Property Summary
Tensor<T> TimbreReference { get; set; } A recording whose timbre synthesis takes (at SampleRate); null for the voice’s own.
bool TrainsOnSampleObjective { get; }
int MaxTextLength { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> ConvertStyle(Tensor<T> source, int sourceRate, Tensor<T> styleReference, int styleRate) Vevo-Style: source in styleReference’s style (accent, emotion), keeping its own timbre; the result is at 24 kHz.
Tensor<T> ConvertTimbre(Tensor<T> source, int sourceRate, Tensor<T> timbreReference, int timbreRate) Vevo-Timbre: source in timbreReference’s timbre, keeping its content and style (the flow alone, inference_fm); the result is at 24 kHz.
Tensor<T> ConvertVoice(Tensor<T> source, int sourceRate, Tensor<T> reference, int referenceRate) Vevo-Voice: source spoken in reference’s voice — its timbre and its style (accent, emotion) — both at their own rates; the result is at 24 kHz.
Tensor<T> Synthesize(string text)
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate, string transcript, string language) A voice from a recording of the speaker at sampleRate and what it says: the recording gives the style (and, unless TimbreReference is set, the timbre); its transcript’s phones in language prompt the text-to-content-style transformer.
ValueTuple<string, int[]> Phonemize(string text, string language) Amphion’s g2p_: the phone string and the phone ids of text in language.
void LoadReferenceWeights(VevoCheckpoints checkpoints) Loads the release: HuBERT-Large (its first HubertLayer layers), the content-style tokenizer with HuBERT’s feature statistics, the autoregressive transformers, the content tokenizer (when voice conversion is on), the flow-matching transfor…