AVHuBERT<T>

Models & Types · AiDotNet.SpeechRecognition.Robust

AV-HuBERT: a Transformer over fused audio and lip-video features, pre-trained by masked multimodal cluster prediction, with a CTC projection for recognition from audio, lips or both.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AVHuBERT<T>

Implements: ISpeechRecognizer<T>

For Beginners

AV-HuBERT watches the speaker’s mouth and listens at the same time. During pre-training it hides stretches of the audio and the video and learns to predict cluster labels of what was hidden; sometimes it gets only the audio or only the video, so afterwards it can recognize speech from either, which makes it robust to noise and able to lip-read.

How It Works

References:

  • Paper: “Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction” (Shi et al., Meta, ICLR 2022)

The network is the BASE model (section 4, appendix B.3): a linear audio encoder over 104-dimensional stacked log filterbanks, a modified ResNet-18 (5x7x7 3-D convolution front, PReLU) over 88x88 grayscale mouth crops, both at 25 Hz, fused by concatenation with modality dropout (Equation 2), and the pre-norm wav2vec 2.0 Transformer (12 layers, 768 wide). Pretrain runs the masked prediction of section 3 against cluster targets the caller supplies (the paper’s k-means over MFCC, then over the previous iteration’s features); recognition puts a CTC projection on the Transformer (section 3.3). Transcribe uses the audio alone, TranscribeLips the video alone and TranscribeAudioVisual both.

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputHeight: 16000, inputWidth: 1, inputDepth: 1, outputSize: 46);
var model = new AVHuBERT<double>(architecture);
var result = model.TranscribeAudioVisual(audio, lips);   // lips: [frames, 88, 88] at 25 frames a second

Constructors

Constructor Summary
AVHuBERT(NeuralNetworkArchitecture<T> architecture, AVHuBERTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
AVHuBERT(NeuralNetworkArchitecture<T> architecture, string modelPath, AVHuBERTOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
T Pretrain(Tensor<T> audio, Tensor<T> lips, int[] clusterTargets) One masked multimodal cluster prediction step (section 3) on one utterance.
T PretrainingLoss(Tensor<T> audio, Tensor<T> lips, int[] clusterTargets, Random random) The masked multimodal cluster prediction loss of one utterance (Shi et al.
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the encoder, projection and CTC layer give per-frame distributions, decoded greedily.
TranscriptionResult<T> TranscribeAudioVisual(Tensor<T> audio, Tensor<T> lips, string language, bool? includeTimestamps) Transcribes from the audio and the lip video together.
TranscriptionResult<T> TranscribeLips(Tensor<T> lips, string language, bool? includeTimestamps) Transcribes from the lip video alone (lip reading).
string DetectLanguage(Tensor<T> audio)