VideoMAE<T>

Models & Types · AiDotNet.Video.ActionRecognition

Video Masked Autoencoder (VideoMAE) for video understanding and action recognition.

Inheritance: NeuralNetworkBase<T> → VideoMAE<T>

For Beginners

VideoMAE is a self-supervised learning model for video understanding. It learns powerful video representations by masking random patches in video frames and training the model to reconstruct the missing content. This learned representation can then be used for various tasks:

  • Action recognition (identifying what’s happening in a video)
  • Video classification
  • Temporal reasoning
  • Video captioning

The key insight is that learning to reconstruct masked video teaches the model about motion, appearance, and temporal patterns in videos.

How It Works

Technical Details:

  • Vision Transformer (ViT) architecture with temporal extension
  • Tube masking strategy for spatiotemporal masking
  • High masking ratio (75-90%) for efficient training
  • Joint space-time attention mechanism

Pretraining: call PretrainMAE repeatedly on unlabeled clips; each call is one optimizer step on the masked-patch reconstruction loss (per-patch normalised pixel targets by default, see NormalizeTarget). Then fine-tune the classifier with Train. The encoder here is a convolutional stand-in for the paper’s ViT: masked patches are kept at zero through every encoder block (the sparse-convolution form of dropping them), and the decoder is a small convolutional stack with a per-patch pixel head.

Reference: Tong et al., “VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training” NeurIPS 2022.

Example

// Create a VideoMAE model for action recognition on Kinetics-400
var videoMAE = new VideoMAE<double>();
            
// Or configure with a custom architecture and parameters. Declare the input as
// FourDimensional [frames, channels, height, width] so GetInputShape() describes a clip.
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.FourDimensional,
    taskType: NeuralNetworkTaskType.MultiClassClassification,
    inputFrames: 16, inputHeight: 224, inputWidth: 224, inputDepth: 3,
    outputSize: 400);
var model = new VideoMAE<double>(architecture, options: new VideoMAEOptions { NumClasses = 400, NumFrames = 16 });

Constructors

Constructor Summary
VideoMAE() Initializes a new instance of VideoMAE with default architecture (16 frames of 224x224, 400 classes).
VideoMAE(NeuralNetworkArchitecture<T> architecture, VideoMAEOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Initializes a new instance of the VideoMAE class in native (trainable) mode.
VideoMAE(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, VideoMAEOptions options) Initializes a new instance of the VideoMAE class in ONNX (inference-only) mode.

Properties

Property Summary
bool SupportsTraining { get; } Gets whether training is supported.

Methods

Method Summary
List<ValueTuple<int, double>> GetTopKPredictions(Tensor<T> video, int k) Gets the top-k predicted actions for a video.
ModelMetadata<T> GetModelMetadata()
T PretrainMAE(Tensor<T> video) Performs one masked-autoencoder pretraining step on a video clip: masks it, reconstructs the masked tubelet patches, and updates the encoder, decoder and reconstruction head to reduce the reconstruction error.
Tensor<T> ClassifyAction(Tensor<T> video) Classifies actions in a video clip.
Tensor<T> ExtractFeatures(Tensor<T> video) Extracts video features for downstream tasks.
Tensor<T> ForwardForTraining(Tensor<T> input)