WhisperSpeechTrainingStage

Enums · AiDotNet.TextToSpeech.MultiModal

What a WhisperSpeech training step trains.

Fields

Field Summary
WhisperSpeechTrainingStage SemanticToAcoustic The semantic-to-acoustic model (S2A): each EnCodec quantizer’s next-token cross-entropy under the delay pattern, conditioned on the recording’s speaker embedding.
WhisperSpeechTrainingStage SemanticTokenizer The semantic tokenizer (the Whisper VQ bottleneck): the frozen Whisper decoder’s cross-entropy on the transcript from the quantized states, plus KL divergence to Whisper’s own logits, plus the commitment loss.
WhisperSpeechTrainingStage TextToSemantic The text-to-semantic model (T2S): next-token cross-entropy over the recording’s semantic tokens, plus 0.1 × the causal text encoder’s next-byte cross-entropy.
int value__