CosyVoiceToken2TextReward<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

A trainable token-to-text sequence model that supplies a differentiable ASR reward for DiffRO.

Implements: ICosyVoiceTokenReward<T>

How It Works

This concrete native reward uses a T5 encoder/decoder and a separate speech embedding table. CosyVoice’s paper does not release its reward architecture or weights, so these configurable choices are not author defaults. Train or load the reward before policy optimization. Text targets are explicit, including EOS when desired.

Constructors

Constructor Summary
CosyVoiceToken2TextReward(int speechVocabulary, int textVocabulary, int modelDim, int feedForwardDim, int heads, int encoderLayers, int decoderLayers, int decoderStartTokenId, int initializationSeed) Creates a native reward model with caller-configured dimensions and vocabulary IDs.

Properties

Property Summary
IReadOnlyList<LayerBase<T>> Layers { get; }
IReadOnlyList<Tensor<T>> Parameters { get; } The reward’s trainable parameter tensors; these are not part of the policy’s parameter ownership.
int SpeechVocabularySize { get; }
int TextVocabularySize { get; } The number of text-token categories in the reward’s decoder.

Methods

Method Summary
T Train(IReadOnlyList<int> speechTokens, IReadOnlyList<int> textTokens, T learningRate) Trains the ASR-like reward on hard speech IDs and paired text using one actual SGD step.
Tensor<T> LogReward(Tensor<T> softSpeechTokens, IReadOnlyList<int> textTokens)
void LoadWeights(Func<string, int[], double[]> read) Loads the reward’s weights from source, each tensor copied straight into its layer.