TimeSformer<T>

Models & Types · AiDotNet.Video.ActionRecognition

TimeSformer: Is Space-Time Attention All You Need for Video Understanding?

Inheritance: NeuralNetworkBase<T> → TimeSformer<T>

For Beginners

TimeSformer is a transformer-based model for video classification that applies attention across both space and time dimensions. Unlike CNNs that use 3D convolutions, TimeSformer uses pure self-attention to understand video content.

Key capabilities:

  • Video action recognition (classify what action is happening)
  • Temporal reasoning (understand events across time)
  • Scene understanding (understand spatial context)

The model uses “divided space-time attention” where:

  1. First, attention is applied across time (same spatial location, different frames)
  2. Then, attention is applied across space (same frame, different locations)

Example usage (native mode for training):

Example usage (ONNX mode for inference only):

How It Works

Technical Details:

  • Divided space-time attention for efficiency
  • Patch embedding similar to ViT
  • Learnable positional embeddings for space and time
  • Classification token for final prediction

Reference: “Is Space-Time Attention All You Need for Video Understanding?” https://arxiv.org/abs/2102.05095

Constructors

Constructor Summary
TimeSformer() Initializes a new instance with default architecture settings (8-frame 224x224 RGB clips, 400 classes).
TimeSformer(NeuralNetworkArchitecture<T> architecture, TimeSformerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a TimeSformer model using native layers for training and inference.
TimeSformer(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, TimeSformerOptions options) Creates a TimeSformer model using a pretrained ONNX model for inference.

Properties

Property Summary
bool SupportsTraining { get; } Gets whether training is supported (only in native mode).

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
List<ValueTuple<int, double>> GetTopKPredictions(Tensor<T> videoFrames, int topK) Gets the top-K predicted action classes with probabilities.
ModelMetadata<T> GetModelMetadata()
Tensor<T> Classify(Tensor<T> videoFrames) Classifies video frames into action categories.
Tensor<T> ExtractFeatures(Tensor<T> videoFrames) Extracts video features before the classification head.
Tensor<T> ForwardForTraining(Tensor<T> input)