AVHuBERT<T>
Models & Types · AiDotNet.SpeechRecognition.Robust
AV-HuBERT: a Transformer over fused audio and lip-video features, pre-trained by masked multimodal cluster prediction, with a CTC projection for recognition from audio, lips or both.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AVHuBERT<T>
Implements: ISpeechRecognizer<T>
For Beginners
AV-HuBERT watches the speaker’s mouth and listens at the same time. During pre-training it hides stretches of the audio and the video and learns to predict cluster labels of what was hidden; sometimes it gets only the audio or only the video, so afterwards it can recognize speech from either, which makes it robust to noise and able to lip-read.
How It Works
References:
- Paper: “Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction” (Shi et al., Meta, ICLR 2022)
The network is the BASE model (section 4, appendix B.3): a linear audio encoder over 104-dimensional stacked log
filterbanks, a modified ResNet-18 (5x7x7 3-D convolution front, PReLU) over 88x88 grayscale mouth crops, both at 25 Hz,
fused by concatenation with modality dropout (Equation 2), and the pre-norm wav2vec 2.0 Transformer (12 layers, 768
wide). Pretrain runs the masked prediction of section 3 against cluster targets the caller supplies (the
paper’s k-means over MFCC, then over the previous iteration’s features); recognition puts a CTC projection on the
Transformer (section 3.3). Transcribe uses the audio alone, TranscribeLips the video alone
and TranscribeAudioVisual both.
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputHeight: 16000, inputWidth: 1, inputDepth: 1, outputSize: 46);
var model = new AVHuBERT<double>(architecture);
var result = model.TranscribeAudioVisual(audio, lips); // lips: [frames, 88, 88] at 25 frames a second
Constructors
| Constructor | Summary |
|---|---|
AVHuBERT(NeuralNetworkArchitecture<T> architecture, AVHuBERTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
AVHuBERT(NeuralNetworkArchitecture<T> architecture, string modelPath, AVHuBERTOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
T Pretrain(Tensor<T> audio, Tensor<T> lips, int[] clusterTargets) |
One masked multimodal cluster prediction step (section 3) on one utterance. |
T PretrainingLoss(Tensor<T> audio, Tensor<T> lips, int[] clusterTargets, Random random) |
The masked multimodal cluster prediction loss of one utterance (Shi et al. |
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the encoder, projection and CTC layer give per-frame distributions, decoded greedily. |
TranscriptionResult<T> TranscribeAudioVisual(Tensor<T> audio, Tensor<T> lips, string language, bool? includeTimestamps) |
Transcribes from the audio and the lip video together. |
TranscriptionResult<T> TranscribeLips(Tensor<T> lips, string language, bool? includeTimestamps) |
Transcribes from the lip video alone (lip reading). |
string DetectLanguage(Tensor<T> audio) |