CosyVoiceBase<T>
Base Classes · AiDotNet.TextToSpeech.CodecBased
What the CosyVoice models share (FunAudioLLM/CosyVoice, Apache-2.0): a language model speaking S3 tokens, a flow
sampling an 80-band mel from them by conditional flow matching with classifier-free guidance, HiFT turning the mel into
audio, and the frozen S3 tokenizer and CAM++ reading a prompt’s tokens and x-vector; the front end
(CosyVoiceTextFrontEnd) and voices (zero-shot, cross-lingual, a speaker embedding); and training in three
stages. Each model supplies its text tokenizer and language model.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → CosyVoiceBase<T>
Implements: ICodecTts<T>
Known derived types: Chatterbox<T>, CosyVoice2<T>, CosyVoice3<T>, CosyVoice<T>
Properties
| Property | Summary |
|---|---|
CosyVoiceTrainingStage CurrentStage { get; set; } |
The network a training step trains; the recipe trains the language model, the flow, then the vocoder. |
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
|
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<string> SplitText(string text) |
The sentences the model synthesizes text as (text_normalize(text, split=True) without an external normalizer): see CosyVoiceTextFrontEnd. |
LayerInputDomain GetInputDomain(int[] inputShape) |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
Decodes S3 speech tokens into audio through the flow and HiFT, in Voice. |
Tensor<T> EncodeToTokens(Tensor<T> audio) |
Encodes audio at the model’s sample rate into S3 speech tokens [frames]. |
Tensor<T> MelSpectrogram(Tensor<T> audio) |
The flow’s mel [frames, MelChannels] of a recording at the model’s sample rate (_extract_speech_feat: Matcha’s mel_spectrogram with the model’s FFT, hop and window, 0–8 kHz, not centred). |
Tensor<T> MelToAudio(Tensor<T> mel, Random random) |
The waveform [samples] of a mel [1, MelChannels, frames] (HiFTGenerator.inference). |
Tensor<T> SpeakerEmbedding(Tensor<T> recording, int sampleRate) |
The CAM++ x-vector [SpeakerEmbeddingDim] of a recording (_extract_spk_embedding): resampled to 16 kHz, 80-band Kaldi filterbank energies minus their utterance mean, CAM++ in evaluation mode. |
Tensor<T> Synthesize(string text) |
|
Tensor<T> TokensToMel(IReadOnlyList<int> tokens, Random random) |
The mel [1, MelChannels, frames] of speech tokens (the flow’s inference): the voice’s prompt tokens and these encoded together, the prompt’s mel as the condition, Euler sampling from Gaussian noise, and the prompt’s frames removed. |
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate, string transcript) |
A voice for Voice from a recording of the speaker [samples] at sampleRate: its S3 tokens, its mel (resampled to the model’s rate) and its CAM++ x-vector. |
TtsVoice<T> CreateVoice(Tensor<T> speakerEmbedding) |
A voice of a speaker embedding alone (frontend_sft: a speaker of spk2info). |
int[] GenerateSpeechTokens(IReadOnlyList<int> text, Random random) |
Generates speech tokens for text ids under Voice (llm.inference): prompted by the voice’s transcript and tokens in zero-shot, by its x-vector alone otherwise. |
int[] SpeechTokens(Tensor<T> recording, int sampleRate) |
The S3 speech tokens of a recording [samples] at sampleRate (_extract_speech_token): resampled to 16 kHz, Whisper’s 128-band log-mel, the release’s S3 tokenizer. |
int[] Tokenize(string text) |
The text token ids of text, special tokens written in it allowed. |
string Detokenize(IEnumerable<int> ids) |
The text of token ids. |
void SetTrainingMode(bool isTraining) |