WhisperSpeechTrainingStage
Enums · AiDotNet.TextToSpeech.MultiModal
What a WhisperSpeech training step trains.
Fields
| Field | Summary |
|---|---|
WhisperSpeechTrainingStage SemanticToAcoustic |
The semantic-to-acoustic model (S2A): each EnCodec quantizer’s next-token cross-entropy under the delay pattern, conditioned on the recording’s speaker embedding. |
WhisperSpeechTrainingStage SemanticTokenizer |
The semantic tokenizer (the Whisper VQ bottleneck): the frozen Whisper decoder’s cross-entropy on the transcript from the quantized states, plus KL divergence to Whisper’s own logits, plus the commitment loss. |
WhisperSpeechTrainingStage TextToSemantic |
The text-to-semantic model (T2S): next-token cross-entropy over the recording’s semantic tokens, plus 0.1 × the causal text encoder’s next-byte cross-entropy. |
int value__ |