CosyVoiceBase<T>

Base Classes · AiDotNet.TextToSpeech.CodecBased

What the CosyVoice models share (FunAudioLLM/CosyVoice, Apache-2.0): a language model speaking S3 tokens, a flow sampling an 80-band mel from them by conditional flow matching with classifier-free guidance, HiFT turning the mel into audio, and the frozen S3 tokenizer and CAM++ reading a prompt’s tokens and x-vector; the front end (CosyVoiceTextFrontEnd) and voices (zero-shot, cross-lingual, a speaker embedding); and training in three stages. Each model supplies its text tokenizer and language model.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → CosyVoiceBase<T>

Implements: ICodecTts<T>

Known derived types: Chatterbox<T>, CosyVoice2<T>, CosyVoice3<T>, CosyVoice<T>

Properties

Property Summary
CosyVoiceTrainingStage CurrentStage { get; set; } The network a training step trains; the recipe trains the language model, the flow, then the vocoder.
int CodebookSize { get; }
int CodecFrameRate { get; }
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int MaxTextLength { get; }
int NumCodebooks { get; }

Methods

Method Summary
IReadOnlyList<string> SplitText(string text) The sentences the model synthesizes text as (text_normalize(text, split=True) without an external normalizer): see CosyVoiceTextFrontEnd.
LayerInputDomain GetInputDomain(int[] inputShape)
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> DecodeFromTokens(Tensor<T> tokens) Decodes S3 speech tokens into audio through the flow and HiFT, in Voice.
Tensor<T> EncodeToTokens(Tensor<T> audio) Encodes audio at the model’s sample rate into S3 speech tokens [frames].
Tensor<T> MelSpectrogram(Tensor<T> audio) The flow’s mel [frames, MelChannels] of a recording at the model’s sample rate (_extract_speech_feat: Matcha’s mel_spectrogram with the model’s FFT, hop and window, 0–8 kHz, not centred).
Tensor<T> MelToAudio(Tensor<T> mel, Random random) The waveform [samples] of a mel [1, MelChannels, frames] (HiFTGenerator.inference).
Tensor<T> SpeakerEmbedding(Tensor<T> recording, int sampleRate) The CAM++ x-vector [SpeakerEmbeddingDim] of a recording (_extract_spk_embedding): resampled to 16 kHz, 80-band Kaldi filterbank energies minus their utterance mean, CAM++ in evaluation mode.
Tensor<T> Synthesize(string text)
Tensor<T> TokensToMel(IReadOnlyList<int> tokens, Random random) The mel [1, MelChannels, frames] of speech tokens (the flow’s inference): the voice’s prompt tokens and these encoded together, the prompt’s mel as the condition, Euler sampling from Gaussian noise, and the prompt’s frames removed.
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate, string transcript) A voice for Voice from a recording of the speaker [samples] at sampleRate: its S3 tokens, its mel (resampled to the model’s rate) and its CAM++ x-vector.
TtsVoice<T> CreateVoice(Tensor<T> speakerEmbedding) A voice of a speaker embedding alone (frontend_sft: a speaker of spk2info).
int[] GenerateSpeechTokens(IReadOnlyList<int> text, Random random) Generates speech tokens for text ids under Voice (llm.inference): prompted by the voice’s transcript and tokens in zero-shot, by its x-vector alone otherwise.
int[] SpeechTokens(Tensor<T> recording, int sampleRate) The S3 speech tokens of a recording [samples] at sampleRate (_extract_speech_token): resampled to 16 kHz, Whisper’s 128-band log-mel, the release’s S3 tokenizer.
int[] Tokenize(string text) The text token ids of text, special tokens written in it allowed.
string Detokenize(IEnumerable<int> ids) The text of token ids.
void SetTrainingMode(bool isTraining)