WhisperSpeechOptions
Options & Configuration · AiDotNet.TextToSpeech.MultiModal
Options for WhisperSpeech.
Inheritance: ModelOptions → WhisperSpeechOptions
For Beginners
WhisperSpeech turns text into “semantic tokens” (what Whisper hears), those into EnCodec audio tokens in a chosen voice, and the audio tokens into sound.
How It Works
The defaults are what the release’s Pipeline() loads (collabora/WhisperSpeech): t2s-small-en+pl and
s2a-q4-small-en+pl, whose semantic tokens are whisper-vq-stoks-medium-en+pl’s (Whisper medium’s encoder, quantized at 25
Hz to 512 codes), Vocos’s EnCodec 24 kHz decoder over four EnCodec codebooks (3 kbps), and speechbrain’s ECAPA speaker
embedding. ForCheckpoints configures the other released checkpoints. Training follows the release’s
trainer (AdamW β (0.9, 0.95) with per-module rates and weight decay, the one-cycle cosine schedule with its momentum
cycle, gradient-norm clipping) at each model’s tunables.
Constructors
| Constructor | Summary |
|---|---|
WhisperSpeechOptions() |
Creates the release’s default pipeline. |
Properties
| Property | Summary |
|---|---|
EnCodecOptions Codec { get; set; } |
The codec whose codes the S2A model writes (EnCodec 24 kHz at 3 kbps: four codebooks), frozen. |
OnnxModelOptions OnnxOptions { get; set; } |
Runtime configuration for an ONNX export. |
WhisperSpeechTrainingStage Stage { get; set; } |
What training steps train (the T2S model by default). |
bool S2ACausalEncoder { get; set; } |
The S2A semantic encoder is causal and adds its next-token loss (false). |
bool S2AConditioned { get; set; } |
The conditioned S2A variant (language, SNR and C50 embeddings with the speaker’s). |
bool S2AHiddenToEmbedding { get; set; } |
The S2A model keeps the hidden-to-embedding projection (force_hidden_to_emb; true for the q4 checkpoints, false from v1.1 on). |
bool T2SCausalEncoder { get; set; } |
The T2S text encoder is causal and adds its 0.1-weighted next-byte loss (true). |
bool T2SCharactersPerSecondInput { get; set; } |
The T2S decoder is conditioned on the characters-per-second bin (true). |
double CharactersPerSecond { get; set; } |
The speaking rate in characters a second (15) that the T2S model is conditioned on. |
double S2AEmbeddingsLrScale { get; set; } |
The S2A embeddings’ learning-rate multiple (10). |
double S2AEmbeddingsStd { get; set; } |
|
double S2AEncoderDepthRatio { get; set; } |
The share of the S2A blocks in the semantic encoder (0.25). |
double S2AInitStd { get; set; } |
The S2A initialization scale (9) and embedding deviation (0.2). |
double S2ALearningRate { get; set; } |
The S2A peak learning rate (3e-3), gradient-norm clip (2), weight decay (1e-3) and warmup (2000). |
double S2AMaxGradientNorm { get; set; } |
|
double S2AOutputMult { get; set; } |
The S2A logit multiplier (5.6) and query multiplier (0.3). |
double S2AQueryMult { get; set; } |
|
double S2AWeightDecay { get; set; } |
|
double T2SEmbeddingsLrScale { get; set; } |
The T2S embeddings’ (5) and embedding projectors’ (2.5) learning-rate multiples. |
double T2SEmbeddingsStd { get; set; } |
|
double T2SEncoderDepthRatio { get; set; } |
The share of the T2S blocks in the text encoder (0.25). |
double T2SInitStd { get; set; } |
The T2S initialization scale (1) and embedding deviation (0.01). |
double T2SLearningRate { get; set; } |
The T2S peak learning rate (1.5e-3), gradient-norm clip (0.2), weight decay (0.1) and warmup (4000). |
double T2SMaxGradientNorm { get; set; } |
|
double T2SOutputMult { get; set; } |
The T2S logit multiplier (0.35) and query multiplier (1). |
double T2SProjectorLrScale { get; set; } |
|
double T2SQueryMult { get; set; } |
|
double T2SWeightDecay { get; set; } |
|
double Temperature { get; set; } |
The sampling temperature of both models (0.7). |
double VqCodebookDecay { get; set; } |
The codebook’s moving-average decay (0.9) and dead-code threshold (0: the release trains without replacement). |
double VqDeadCodeThreshold { get; set; } |
|
double VqEmbeddingsStd { get; set; } |
|
double VqInitStd { get; set; } |
The tokenizer’s initialization scale (1.5), position deviation (0.045) and query multiplier (2). |
double VqLearningRate { get; set; } |
The tokenizer’s peak learning rate (9e-4), gradient-norm clip (2), weight decay (1e-3) and warmup (850). |
double VqMaxGradientNorm { get; set; } |
|
double VqQueryMult { get; set; } |
|
double VqWeightDecay { get; set; } |
|
int HopSize { get; set; } |
Samples per acoustic frame, equal to the product of the codec’s strides. |
int MaxTextLength { get; set; } |
The T2S text context in UTF-8 bytes, including its start position. |
int S2ADepth { get; set; } |
The S2A model’s depth (12), heads (12), head width (64) and feed-forward multiple (4). |
int S2AFfnMult { get; set; } |
|
int S2AHeadWidth { get; set; } |
|
int S2AHeads { get; set; } |
|
int S2AWarmupSteps { get; set; } |
|
int SampleRate { get; set; } |
The audio rate, which must match the selected EnCodec configuration. |
int SemanticCodes { get; set; } |
The semantic codes (512); the T2S and S2A tables add the padding code. |
int SemanticWidth { get; set; } |
The semantic codebook’s width (64), which the T2S and S2A models read frozen. |
int SpeakerAttentionChannels { get; set; } |
|
int SpeakerEmbeddingSize { get; set; } |
|
int SpeakerSeChannels { get; set; } |
|
int T2SDepth { get; set; } |
The T2S model’s depth (12: 6 encoder and 18 decoder blocks at the 0.25 ratio). |
int T2SFfnMult { get; set; } |
The T2S feed-forward multiple (4). |
int T2SHeadWidth { get; set; } |
The T2S model’s head width (64). |
int T2SHeads { get; set; } |
The T2S model’s heads (12); with T2SHeadWidth its width. |
int T2SPaddingTokenOffset { get; set; } |
The T2S padding and start token relative to the semantic table (−1: the last code; 0: past it). |
int T2SWarmupSteps { get; set; } |
|
int TotalSteps { get; set; } |
The steps of the one-cycle schedule (8000, the release trainer’s default iterations). |
int VocosDim { get; set; } |
Vocos’s backbone width (384), ConvNeXt width (1152) and blocks (8). |
int VocosIntermediateDim { get; set; } |
|
int VocosLayers { get; set; } |
|
int VqDepth { get; set; } |
The tokenizer’s output blocks (1) and heads (16; their width is WhisperWidth over them). |
int VqDownsample { get; set; } |
Whisper frames per semantic token (2, averaged). |
int VqHeads { get; set; } |
|
int VqWarmupSteps { get; set; } |
|
int WhisperAudioContext { get; set; } |
Whisper’s audio positions (1500, a 30-second window); the semantic tokens of a window are this over VqDownsample, and the S2A context three times that. |
int WhisperHeads { get; set; } |
|
int WhisperLayers { get; set; } |
|
int WhisperMels { get; set; } |
|
int WhisperTextContext { get; set; } |
|
int WhisperVocabulary { get; set; } |
|
int WhisperWidth { get; set; } |
The Whisper model the tokenizer quantizes (medium: 1024 wide, 16 heads, 24 encoder and 24 decoder layers, 1500 audio and 448 text positions, 80 mels, the 51,865-token multilingual vocabulary). |
int? TopK { get; set; } |
Sampling over the most likely k tokens only (null: all, the release’s default). |
int? VocosBandwidthId { get; set; } |
The Vocos bandwidth class decoding uses (null: the one of the codebook count, as the release picks it). |
int[] SpeakerChannels { get; set; } |
The ECAPA-TDNN speaker extractor’s stage widths (1024, 1024, 1024, 1024, 3072), attention and squeeze-excitation widths (128) and embedding size (192, the S2A speaker width). |
string Language { get; set; } |
The language spoken (a Whisper language code: “en”, “pl”, “de”, …). |
string ModelPath { get; set; } |
The optional ONNX export path. |
Methods
| Method | Summary |
|---|---|
EnCodecOptions CreateReleaseCodec() |
EnCodec’s 24 kHz model at 3 kbps (four codebooks), the acoustic tokens the release’s S2A models. |
WhisperSpeechOptions ForCheckpoints(string textToSemantic, string semanticToAcoustic, string semanticTokenizer) |
The configuration of released checkpoints (names as in the WhisperSpeech/WhisperSpeech repository, with or without .model): a T2S model, an S2A model and the semantic tokenizer their tokens come from (the en+pl models read whisper-vq-stok… |