CosyVoice2<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

CosyVoice 2: streaming TTS with a pretrained LLM, finite-scalar-quantized speech tokens and chunk-aware causal flow matching.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → CosyVoiceBase<T> → CosyVoice2<T>

For Beginners

CosyVoice 2 reads text with a small general-purpose language model that has learned to “speak” in speech tokens, then turns those tokens into sound. It can imitate a voice from a few seconds of recording.

How It Works

References:

  • Paper: “CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models” (Du et al., 2024)
  • Code: FunAudioLLM/CosyVoice (Apache-2.0), CosyVoice2-0.5B’s cosyvoice2.yaml.

Text is tokenized by Qwen2.5’s byte-level BPE with CosyVoice 2’s added tokens (CosyVoice2). The language model, Qwen2.5-0.5B (Qwen2LM), reads [sos, text, task_id] through its own token table and generates the 25 Hz S3Tokenizer v2’s FSQ tokens by repetition-aware sampling; it reads no x-vector. The causal flow (CausalMaskedDiffWithXvec) encodes the tokens with an upsampling Conformer giving two mel frames per token and samples an 80-band mel with a causal U-Net estimator, conditioned on the x-vector and a prompt’s mel; in streaming both attend within chunks. HiFT turns the mel into 24 kHz audio (upsampling 8·5·3 with SineGen2’s excitation). The S3 tokenizer v2 and CAM++ provide a prompt’s tokens and x-vector, frozen as in the release.

Voices (CreateVoice): zero-shot with a transcript, cross-lingual without, or a speaker embedding alone.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional, taskType: NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new CosyVoice2<float>(architecture, new CosyVoice2Options());
model.LoadPretrainedWeights("pretrained_models/CosyVoice2-0.5B");
model.Voice = model.CreateVoice(promptAudio, 16000, "The transcript of the prompt.");
var audio = model.Synthesize("Hello from CosyVoice 2.");

Constructors

Constructor Summary
CosyVoice2(NeuralNetworkArchitecture<T> architecture, CosyVoice2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Initializes a new instance of the CosyVoice2 class in native training/inference mode.
CosyVoice2(NeuralNetworkArchitecture<T> architecture, string modelPath, CosyVoice2Options options) Initializes a new instance of the CosyVoice2 class in native training/inference mode.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
int[] Tokenize(string text) The text token ids of text: Qwen2.5’s tokenizer with CosyVoice 2’s added tokens (get_qwen_tokenizer(skip_special_tokens=True)), special tokens written in the text allowed.
string Detokenize(IEnumerable<int> ids)
void LoadPretrainedWeights(string modelDirectory) Loads the released networks from a CosyVoice2-0.5B directory, each a state dict converted to safetensors: llm.safetensors, flow.safetensors and hift.safetensors (from llm.pt, flow.pt, hift.pt), speech_tokenizer_v2.safetensors…