CosyVoice<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

CosyVoice: multilingual TTS with supervised semantic tokens and conditional flow matching.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → CosyVoiceBase<T> → CosyVoice<T>

Known derived types: CosyVoiceClone<T>

For Beginners

CosyVoice is a multilingual text-to-speech model that can clone voices without any training on the target speaker (zero-shot). It works by first converting text into “semantic tokens” using a language model, then using conditional flow matching to generate high-quality mel-spectrograms from those tokens.

How It Works

References:

  • Paper: “CosyVoice: A Scalable Multilingual Zero-Shot Text-to-Speech Synthesizer Based on Supervised Semantic Tokens” (Du et al., 2024)
  • Code: FunAudioLLM/CosyVoice (Apache-2.0), CosyVoice-300M’s cosyvoice.yaml.

Text is tokenized by Whisper’s multilingual tiktoken encoding. A language model (a Conformer text encoder and a causal Transformer) generates S3 speech tokens after [sos, x-vector, text, task_id], by repetition-aware sampling. A flow model encodes the tokens with a Conformer, stretches them to mel frames and samples an 80-band mel by conditional flow matching with classifier-free guidance, conditioned on the x-vector and a prompt’s mel. HiFT turns the mel into 22.05 kHz audio. The S3 tokenizer (speech_tokenizer_v1) and CAM++ provide a prompt’s speech tokens and x-vector; they are frozen, as in the release.

Voices (CreateVoice): zero-shot, from a recording and its transcript (the LM continues the prompt’s text and tokens); cross-lingual, from a recording alone (only the flow reads the prompt); or a speaker embedding alone (sft). Without a voice the x-vector is zero.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional, taskType: NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new CosyVoice<float>(architecture, new CosyVoiceOptions());
model.LoadPretrainedWeights("pretrained_models/CosyVoice-300M");
model.Voice = model.CreateVoice(promptAudio, 16000, "The transcript of the prompt.");
var audio = model.Synthesize("Hello from CosyVoice.");

Constructors

Constructor Summary
CosyVoice(NeuralNetworkArchitecture<T> architecture, CosyVoiceOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Initializes a new instance of the CosyVoice class in native training/inference mode.
CosyVoice(NeuralNetworkArchitecture<T> architecture, string modelPath, CosyVoiceOptions options) Initializes a new instance of the CosyVoice class in native training/inference mode.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
int[] Tokenize(string text) The text token ids of text: Whisper’s multilingual tiktoken encoding with special tokens written in the text allowed (encode(text, allowed_special='all')).
string Detokenize(IEnumerable<int> ids)
void LoadPretrainedWeights(string modelDirectory) Loads the released networks from a CosyVoice-300M directory, each a state dict converted to safetensors: llm.safetensors, flow.safetensors and hift.safetensors (from llm.pt, flow.pt, hift.pt), speech_tokenizer_v1.safetensors (…