CosyVoice3<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

Fun-CosyVoice 3: Qwen2 speech-token generation, S3-v3 prompt tokens, causal DiT flow matching and CausalHiFT.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → CosyVoiceBase<T> → CosyVoice3<T>

How It Works

This implements the released Fun-CosyVoice3-0.5B-2512 topology. PyTorch state dictionaries are loaded after conversion to safetensors; the frozen prompt tokenizer remains the release’s speech_tokenizer_v3.onnx.

Constructors

Constructor Summary
CosyVoice3(NeuralNetworkArchitecture<T> architecture, CosyVoice3Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates the native released networks for training or checkpoint-backed inference.
CosyVoice3(NeuralNetworkArchitecture<T> architecture, string modelPath, CosyVoice3Options options) Creates the native released networks for training or checkpoint-backed inference.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateDifferentiableReward(TtsTrainingSample<T> sample, CosyVoice3<T> referencePolicy, ICosyVoiceTokenReward<T> reward, CosyVoiceDiffROOptions options) Evaluates DiffRO with reproducible noise without consuming the training noise stream.
T TrainDifferentiableReward(TtsTrainingSample<T> sample, CosyVoice3<T> referencePolicy, ICosyVoiceTokenReward<T> reward, CosyVoiceDiffROOptions options, Random random) Runs one DiffRO policy step, equations 3-7 of the CosyVoice 3 paper.
Tensor<T> TokensToMel(IReadOnlyList<int> tokens, Random random)
int[] SpeechTokens(Tensor<T> recording, int sampleRate)
int[] Tokenize(string text)
string Detokenize(IEnumerable<int> ids)
void LoadPretrainedWeights(string modelDirectory) Loads original llm/flow/hift.pt and frozen ONNX encoders, or converted safetensors components.