AVHuBERTOptions
Options & Configuration · AiDotNet.SpeechRecognition.Robust
Options for AV-HuBERT: audio-visual masked multimodal cluster prediction, with a CTC projection.
Inheritance: ModelOptions → AVHuBERTOptions
For Beginners
These options set the size of AV-HuBERT’s encoder, how often pre-training hides parts of the audio and the lip video, and how often it drops a whole modality.
How It Works
The defaults are the BASE model of Shi et al. 2022 (section 4, appendix B) with the released pre-training configuration where the paper is silent. For LARGE set EncoderDim 1024, NumEncoderLayers 24, NumAttentionHeads 16 and FeedForwardDim 4096.
Constructors
| Constructor | Summary |
|---|---|
AVHuBERTOptions() |
Initializes a new instance with default values. |
AVHuBERTOptions(AVHuBERTOptions other) |
Initializes a new instance by copying from another instance. |
Properties
| Property | Summary |
|---|---|
OnnxModelOptions OnnxOptions { get; set; } |
ONNX runtime options. |
double AudioDropout { get; set; } |
When a modality is dropped, the probability p_a that it is the audio (appendix B.4: 0.5). |
double AudioMaskProbability { get; set; } |
Proportion of audio frames that start a masked span in pre-training (appendix B.4: p = 8 per 10 frames). |
double DropoutRate { get; set; } |
Dropout in the Transformer and on the fused input (appendix B.3: 0.1). |
double FeaturePenaltyWeight { get; set; } |
Weight of the mean squared fused features in the pre-training loss (the released configuration’s loss_weights: 10). |
double LayerDrop { get; set; } |
Probability of skipping each Transformer layer in training (0.05 in the released pre-training configuration; appendix B.3 states 0.1). |
double LogitTemperature { get; set; } |
Temperature of the cosine cluster logits (fairseq HuBERT’s logit_temp: 0.1). |
double ModalityDropout { get; set; } |
Probability p_m complement: with probability ModalityDropout one modality is dropped for the whole sequence (appendix B.4: p_m = 0.5). |
double VideoMaskProbability { get; set; } |
Proportion of video frames that start a masked span in pre-training (0.3 in the released configuration; appendix B.4 states 6 per 5 frames as p/l). |
int AudioMaskLength { get; set; } |
Masked audio span length in frames (appendix B.4: 10). |
int EncoderDim { get; set; } |
Transformer width (section 4: BASE 768, LARGE 1024). |
int FeedForwardDim { get; set; } |
Feed-forward width (section 4: BASE 3072, LARGE 4096). |
int FilterbankBins { get; set; } |
Log filterbank bins per 10 ms frame; four frames are stacked into one 104-dimensional 25 Hz frame (the released audio features). |
int FinalDim { get; set; } |
Width of the projection compared with the cluster embeddings (256 in the released configuration). |
int LipSize { get; set; } |
Side of the grayscale mouth crop in pixels (appendix B.2: 88 x 88). |
int MaxAudioLengthSeconds { get; set; } |
Longest audio a transcription covers, in seconds; longer input is truncated. |
int MaxTextLength { get; set; } |
Most tokens a transcription decodes. |
int NumAttentionHeads { get; set; } |
Attention heads (section 4: BASE 12, LARGE 16). |
int NumClusters { get; set; } |
Cluster targets of the masked prediction (appendix B.4: 2000 in the fifth iteration). |
int NumEncoderLayers { get; set; } |
Transformer layers (section 4: BASE 12, LARGE 24). |
int SampleRate { get; set; } |
Audio sample rate in Hz. |
int VideoMaskLength { get; set; } |
Masked video span length in frames (appendix B.4: 5). |
int VocabSize { get; set; } |
CTC vocabulary size, blank (token 0) included (appendix B.4: 46 phone units for CTC); tokens above 0 decode as Unicode code points. |
string Language { get; set; } |
Language reported for transcriptions. |
string ModelPath { get; set; } |
Path to an exported ONNX model, when one is used. |