OpenVoice<T>

Models & Types · AiDotNet.TextToSpeech.VoiceCloning

OpenVoice: versatile instant voice cloning with decoupled tone color conversion.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → OpenVoice<T>

Implements: IEndToEndTts<T>, IVoiceCloner<T>

For Beginners

OpenVoice: versatile instant voice cloning with decoupled tone color conversion.. This model converts text input into speech audio output.

How It Works

References:

  • Paper: “OpenVoice: Versatile Instant Voice Cloning” (Qin et al., 2023)

Example

// Create an OpenVoice model for versatile instant voice cloning
// with decoupled tone color conversion for flexible synthesis
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.Regression,
    inputHeight: 200, inputWidth: 1, inputDepth: 1, outputSize: 80);
            
// ONNX inference mode with pre-trained model
var model = new OpenVoice<double>(architecture, "openvoice.onnx");
            
// Training mode with native layers
var trainModel = new OpenVoice<double>(architecture, new OpenVoiceOptions());

Constructors

Constructor Summary
OpenVoice(NeuralNetworkArchitecture<T> architecture, OpenVoiceOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
OpenVoice(NeuralNetworkArchitecture<T> architecture, string modelPath, OpenVoiceOptions options)

Properties

Property Summary
ITtsModel<T> BaseSynthesizer { get; set; } The separate base TTS of paper section 2.2; defaults to the native VITS implementation.
bool TrainsOnSampleObjective { get; }
double MinReferenceDuration { get; }
int HiddenDim { get; }
int MaxTextLength { get; }
int NumFlowSteps { get; }
int SpeakerEmbeddingDim { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> ConvertVoice(Tensor<T> audio, Tensor<T> sourceEmbedding, Tensor<T> targetEmbedding, double tau, Tensor<T> noise) Converts mono 22050 Hz source audio using the source and target speaker embeddings.
Tensor<T> ExtractSpeakerEmbedding(Tensor<T> referenceAudio)
Tensor<T> Synthesize(string text)
Tensor<T> SynthesizeWithVoice(string text, Tensor<T> referenceAudio)
void LoadConverterCheckpoint(string path) Loads the original converter checkpoint, including its learned reference encoder.