AVHuBERTOptions

Options & Configuration · AiDotNet.SpeechRecognition.Robust

Options for AV-HuBERT: audio-visual masked multimodal cluster prediction, with a CTC projection.

Inheritance: ModelOptions → AVHuBERTOptions

For Beginners

These options set the size of AV-HuBERT’s encoder, how often pre-training hides parts of the audio and the lip video, and how often it drops a whole modality.

How It Works

The defaults are the BASE model of Shi et al. 2022 (section 4, appendix B) with the released pre-training configuration where the paper is silent. For LARGE set EncoderDim 1024, NumEncoderLayers 24, NumAttentionHeads 16 and FeedForwardDim 4096.

Constructors

Constructor Summary
AVHuBERTOptions() Initializes a new instance with default values.
AVHuBERTOptions(AVHuBERTOptions other) Initializes a new instance by copying from another instance.

Properties

Property Summary
OnnxModelOptions OnnxOptions { get; set; } ONNX runtime options.
double AudioDropout { get; set; } When a modality is dropped, the probability p_a that it is the audio (appendix B.4: 0.5).
double AudioMaskProbability { get; set; } Proportion of audio frames that start a masked span in pre-training (appendix B.4: p = 8 per 10 frames).
double DropoutRate { get; set; } Dropout in the Transformer and on the fused input (appendix B.3: 0.1).
double FeaturePenaltyWeight { get; set; } Weight of the mean squared fused features in the pre-training loss (the released configuration’s loss_weights: 10).
double LayerDrop { get; set; } Probability of skipping each Transformer layer in training (0.05 in the released pre-training configuration; appendix B.3 states 0.1).
double LogitTemperature { get; set; } Temperature of the cosine cluster logits (fairseq HuBERT’s logit_temp: 0.1).
double ModalityDropout { get; set; } Probability p_m complement: with probability ModalityDropout one modality is dropped for the whole sequence (appendix B.4: p_m = 0.5).
double VideoMaskProbability { get; set; } Proportion of video frames that start a masked span in pre-training (0.3 in the released configuration; appendix B.4 states 6 per 5 frames as p/l).
int AudioMaskLength { get; set; } Masked audio span length in frames (appendix B.4: 10).
int EncoderDim { get; set; } Transformer width (section 4: BASE 768, LARGE 1024).
int FeedForwardDim { get; set; } Feed-forward width (section 4: BASE 3072, LARGE 4096).
int FilterbankBins { get; set; } Log filterbank bins per 10 ms frame; four frames are stacked into one 104-dimensional 25 Hz frame (the released audio features).
int FinalDim { get; set; } Width of the projection compared with the cluster embeddings (256 in the released configuration).
int LipSize { get; set; } Side of the grayscale mouth crop in pixels (appendix B.2: 88 x 88).
int MaxAudioLengthSeconds { get; set; } Longest audio a transcription covers, in seconds; longer input is truncated.
int MaxTextLength { get; set; } Most tokens a transcription decodes.
int NumAttentionHeads { get; set; } Attention heads (section 4: BASE 12, LARGE 16).
int NumClusters { get; set; } Cluster targets of the masked prediction (appendix B.4: 2000 in the fifth iteration).
int NumEncoderLayers { get; set; } Transformer layers (section 4: BASE 12, LARGE 24).
int SampleRate { get; set; } Audio sample rate in Hz.
int VideoMaskLength { get; set; } Masked video span length in frames (appendix B.4: 5).
int VocabSize { get; set; } CTC vocabulary size, blank (token 0) included (appendix B.4: 46 phone units for CTC); tokens above 0 decode as Unicode code points.
string Language { get; set; } Language reported for transcriptions.
string ModelPath { get; set; } Path to an exported ONNX model, when one is used.