CosyVoice2<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
CosyVoice 2: streaming TTS with a pretrained LLM, finite-scalar-quantized speech tokens and chunk-aware causal flow matching.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → CosyVoiceBase<T> → CosyVoice2<T>
For Beginners
CosyVoice 2 reads text with a small general-purpose language model that has learned to “speak” in speech tokens, then turns those tokens into sound. It can imitate a voice from a few seconds of recording.
How It Works
References:
- Paper: “CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models” (Du et al., 2024)
- Code: FunAudioLLM/CosyVoice (Apache-2.0), CosyVoice2-0.5B’s
cosyvoice2.yaml.
Text is tokenized by Qwen2.5’s byte-level BPE with CosyVoice 2’s added tokens (CosyVoice2).
The language model, Qwen2.5-0.5B (Qwen2LM), reads [sos, text, task_id] through its own token table and
generates the 25 Hz S3Tokenizer v2’s FSQ tokens by repetition-aware sampling; it reads no x-vector. The causal flow
(CausalMaskedDiffWithXvec) encodes the tokens with an upsampling Conformer giving two mel frames per token and
samples an 80-band mel with a causal U-Net estimator, conditioned on the x-vector and a prompt’s mel; in streaming
both attend within chunks. HiFT turns the mel into 24 kHz audio (upsampling 8·5·3 with SineGen2’s excitation). The S3
tokenizer v2 and CAM++ provide a prompt’s tokens and x-vector, frozen as in the release.
Voices (CreateVoice): zero-shot with a transcript,
cross-lingual without, or a speaker embedding alone.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional, taskType: NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new CosyVoice2<float>(architecture, new CosyVoice2Options());
model.LoadPretrainedWeights("pretrained_models/CosyVoice2-0.5B");
model.Voice = model.CreateVoice(promptAudio, 16000, "The transcript of the prompt.");
var audio = model.Synthesize("Hello from CosyVoice 2.");
Constructors
| Constructor | Summary |
|---|---|
CosyVoice2(NeuralNetworkArchitecture<T> architecture, CosyVoice2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Initializes a new instance of the CosyVoice2 class in native training/inference mode. |
CosyVoice2(NeuralNetworkArchitecture<T> architecture, string modelPath, CosyVoice2Options options) |
Initializes a new instance of the CosyVoice2 class in native training/inference mode. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
int[] Tokenize(string text) |
The text token ids of text: Qwen2.5’s tokenizer with CosyVoice 2’s added tokens (get_qwen_tokenizer(skip_special_tokens=True)), special tokens written in the text allowed. |
string Detokenize(IEnumerable<int> ids) |
|
void LoadPretrainedWeights(string modelDirectory) |
Loads the released networks from a CosyVoice2-0.5B directory, each a state dict converted to safetensors: llm.safetensors, flow.safetensors and hift.safetensors (from llm.pt, flow.pt, hift.pt), speech_tokenizer_v2.safetensors… |