TimeSformer<T>
Models & Types · AiDotNet.Video.ActionRecognition
TimeSformer: Is Space-Time Attention All You Need for Video Understanding?
Inheritance: NeuralNetworkBase<T> → TimeSformer<T>
For Beginners
TimeSformer is a transformer-based model for video classification that applies attention across both space and time dimensions. Unlike CNNs that use 3D convolutions, TimeSformer uses pure self-attention to understand video content.
Key capabilities:
- Video action recognition (classify what action is happening)
- Temporal reasoning (understand events across time)
- Scene understanding (understand spatial context)
The model uses “divided space-time attention” where:
- First, attention is applied across time (same spatial location, different frames)
- Then, attention is applied across space (same frame, different locations)
Example usage (native mode for training):
Example usage (ONNX mode for inference only):
How It Works
Technical Details:
- Divided space-time attention for efficiency
- Patch embedding similar to ViT
- Learnable positional embeddings for space and time
- Classification token for final prediction
Reference: “Is Space-Time Attention All You Need for Video Understanding?”
https://arxiv.org/abs/2102.05095
Constructors
| Constructor | Summary |
|---|---|
TimeSformer() |
Initializes a new instance with default architecture settings (8-frame 224x224 RGB clips, 400 classes). |
TimeSformer(NeuralNetworkArchitecture<T> architecture, TimeSformerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a TimeSformer model using native layers for training and inference. |
TimeSformer(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, TimeSformerOptions options) |
Creates a TimeSformer model using a pretrained ONNX model for inference. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Gets whether training is supported (only in native mode). |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
List<ValueTuple<int, double>> GetTopKPredictions(Tensor<T> videoFrames, int topK) |
Gets the top-K predicted action classes with probabilities. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Classify(Tensor<T> videoFrames) |
Classifies video frames into action categories. |
Tensor<T> ExtractFeatures(Tensor<T> videoFrames) |
Extracts video features before the classification head. |
Tensor<T> ForwardForTraining(Tensor<T> input) |