GPTSoVITS<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

GPT-SoVITS (RVC-Boss, 2024): few-shot voice cloning in which a GPT writes 25 Hz semantic tokens for the text, continuing a reference recording’s, and a synthesizer speaks them in the reference voice: SoVITS’s VITS-style GAN for v1 to v2ProPlus, conditional flow matching to mels and a vocoder for v3, v4 and V5.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → GPTSoVITS<T>

For Beginners

Record a few seconds of someone speaking, write down what they said, and GPT-SoVITS reads new text in that voice. It works in Chinese, English, Japanese, Korean and Cantonese.

How It Works

The pipeline is the release’s inference_webui.get_tts_wav: the target text cut into sentences, each read by the language front ends into phones (and, for Chinese, chinese-roberta’s features), the reference’s chinese-hubert features quantized into semantic tokens, the GPT (Text2SemanticDecoder) sampling new tokens after the prompt’s phones and tokens, and the synthesizer decoding them with the reference’s style (plus, for v2Pro, an ERes2NetV2 speaker embedding). Sentences are joined by silence.

There is no paper; the release (RVC-Boss/GPT-SoVITS) is the reference, each GptSoVitsVersion with its configuration. Training follows its three scripts: s1 for the GPT, s2 for the GAN synthesizer and the flow-matching synthesizer’s LoRA (CurrentStage).

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var options = new GPTSoVITSOptions { TextLanguage = GptSoVitsTextLanguage.English, PromptLanguage = GptSoVitsTextLanguage.English };
var model = new GPTSoVITS<float>(architecture, options);
model.LoadReferenceWeights(new GptSoVitsCheckpoints { Directory = "GPT-SoVITS/pretrained_models" });
model.Voice = model.CreateVoice(referenceAudio, 32000, "What the reference recording says.");
var audio = model.Synthesize("Text to speak in that voice.");   // 32 kHz for v2ProPlus

Constructors

Constructor Summary
GPTSoVITS(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
GPTSoVITS(NeuralNetworkArchitecture<T> architecture, GPTSoVITSOptions options) Creates a native, trainable GPT-SoVITS.
GPTSoVITS(NeuralNetworkArchitecture<T> architecture, string modelPath, GPTSoVITSOptions options) Creates an ONNX-backed GPT-SoVITS for inference.

Properties

Property Summary
bool TrainsOnSampleObjective { get; }
int MaxTextLength { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> Synthesize(string text)
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate, string promptText) A voice from a recording of the speaker at sampleRate and what it says (promptText, in PromptLanguage), as get_tts_wav reads its reference: resampled to 16 kHz (resampy kaiser_best) with 0.3 s of silence appended for chinese-hubert’…
void LoadReferenceWeights(GptSoVitsCheckpoints checkpoints) Loads the release’s weights for Version: the GPT, the synthesizer (merging a LoRA fine-tune into, or onto the adapters over, its base), chinese-hubert, chinese-roberta, ERes2NetV2 for v2Pro and the vocoder for v3 to V5.