OpenVoiceV2<T>

Models & Types · AiDotNet.TextToSpeech.VoiceCloning

OpenVoiceV2: OpenVoice V2: Improved Instant Voice Cloning.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → OpenVoiceV2<T>

Implements: IEndToEndTts<T>, IVoiceCloner<T>

For Beginners

OpenVoiceV2: OpenVoice V2: Improved Instant Voice Cloning.. This model converts text input into speech audio output.

How It Works

References:

  • Paper: “OpenVoice V2: Improved Instant Voice Cloning” (Qin et al., 2024)

Example

// Create an OpenVoice V2 model for improved instant voice cloning
// with enhanced tone color conversion and multilingual support
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.Regression,
    inputHeight: 200, inputWidth: 1, inputDepth: 1, outputSize: 80);
            
// ONNX inference mode with pre-trained model
var model = new OpenVoiceV2<double>(architecture, "openvoicev2.onnx");
            
// Training mode with native layers
var trainModel = new OpenVoiceV2<double>(architecture, new OpenVoiceV2Options());

Constructors

Constructor Summary
OpenVoiceV2(NeuralNetworkArchitecture<T> architecture, OpenVoiceV2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
OpenVoiceV2(NeuralNetworkArchitecture<T> architecture, string modelPath, OpenVoiceV2Options options)

Properties

Property Summary
ITtsModel<T> BaseSynthesizer { get; set; } The separate native MeloTTS base voice used by the V2 release.
bool TrainsOnSampleObjective { get; }
double MinReferenceDuration { get; }
int HiddenDim { get; }
int MaxTextLength { get; }
int NumFlowSteps { get; }
int SpeakerEmbeddingDim { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> ConvertVoice(Tensor<T> audio, Tensor<T> sourceEmbedding, Tensor<T> targetEmbedding, double tau, Tensor<T> noise) Converts mono 22050 Hz source audio using source and target speaker embeddings.
Tensor<T> ExtractSpeakerEmbedding(Tensor<T> referenceAudio)
Tensor<T> Synthesize(string text) Synthesizes speech from text.
Tensor<T> SynthesizeWithVoice(string text, Tensor<T> referenceAudio)
void LoadConverterCheckpoint(string path) Loads the original V2 converter checkpoint, including its reference encoder.