CosyVoice<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
CosyVoice: multilingual TTS with supervised semantic tokens and conditional flow matching.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → CosyVoiceBase<T> → CosyVoice<T>
Known derived types: CosyVoiceClone<T>
For Beginners
CosyVoice is a multilingual text-to-speech model that can clone voices without any training on the target speaker (zero-shot). It works by first converting text into “semantic tokens” using a language model, then using conditional flow matching to generate high-quality mel-spectrograms from those tokens.
How It Works
References:
- Paper: “CosyVoice: A Scalable Multilingual Zero-Shot Text-to-Speech Synthesizer Based on Supervised Semantic Tokens” (Du et al., 2024)
- Code: FunAudioLLM/CosyVoice (Apache-2.0), CosyVoice-300M’s
cosyvoice.yaml.
Text is tokenized by Whisper’s multilingual tiktoken encoding. A language model (a Conformer text encoder and
a causal Transformer) generates S3 speech tokens after [sos, x-vector, text, task_id], by repetition-aware
sampling. A flow model encodes the tokens with a Conformer, stretches them to mel frames and samples an 80-band mel
by conditional flow matching with classifier-free guidance, conditioned on the x-vector and a prompt’s mel. HiFT turns
the mel into 22.05 kHz audio. The S3 tokenizer (speech_tokenizer_v1) and CAM++ provide a prompt’s speech
tokens and x-vector; they are frozen, as in the release.
Voices (CreateVoice): zero-shot, from a recording and its
transcript (the LM continues the prompt’s text and tokens); cross-lingual, from a recording alone (only the flow reads
the prompt); or a speaker embedding alone (sft). Without a voice the x-vector is zero.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional, taskType: NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new CosyVoice<float>(architecture, new CosyVoiceOptions());
model.LoadPretrainedWeights("pretrained_models/CosyVoice-300M");
model.Voice = model.CreateVoice(promptAudio, 16000, "The transcript of the prompt.");
var audio = model.Synthesize("Hello from CosyVoice.");
Constructors
| Constructor | Summary |
|---|---|
CosyVoice(NeuralNetworkArchitecture<T> architecture, CosyVoiceOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Initializes a new instance of the CosyVoice class in native training/inference mode. |
CosyVoice(NeuralNetworkArchitecture<T> architecture, string modelPath, CosyVoiceOptions options) |
Initializes a new instance of the CosyVoice class in native training/inference mode. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
int[] Tokenize(string text) |
The text token ids of text: Whisper’s multilingual tiktoken encoding with special tokens written in the text allowed (encode(text, allowed_special='all')). |
string Detokenize(IEnumerable<int> ids) |
|
void LoadPretrainedWeights(string modelDirectory) |
Loads the released networks from a CosyVoice-300M directory, each a state dict converted to safetensors: llm.safetensors, flow.safetensors and hift.safetensors (from llm.pt, flow.pt, hift.pt), speech_tokenizer_v1.safetensors (… |