STTN<T>

Models & Types · AiDotNet.Video.Inpainting

STTN spatial-temporal transformer network for video inpainting with multi-scale attention.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoInpaintingBase<T> → STTN<T>

For Beginners

STTN (Spatial-Temporal Transformer Network) performs video inpainting using transformers that jointly attend to spatial and temporal dimensions to fill holes consistently across frames.

How It Works

References:

  • Paper: “Learning Joint Spatial-Temporal Transformations for Video Inpainting” (Zeng et al., ECCV 2020)

STTN uses multi-scale spatial-temporal transformers that jointly search for and attend to relevant patches across both space and time dimensions. Multi-head attention at multiple feature scales enables both fine-grained texture transfer and large-scale structure completion.

Example

// Create an STTN model for spatial-temporal video inpainting
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.Generative,
    inputHeight: 256, inputWidth: 256, inputDepth: 3);
var options = new STTNOptions();
var sttn = new STTN<double>(architecture, options);
            
// Or load a pre-trained ONNX model for inference
var sttnOnnx = new STTN<double>(architecture, "sttn_model.onnx");

Constructors

Constructor Summary
STTN(NeuralNetworkArchitecture<T> architecture, STTNOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a STTN model for native training and inference.
STTN(NeuralNetworkArchitecture<T> architecture, string modelPath, STTNOptions options) Creates a STTN model for native training and inference.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Inpaint(Tensor<T> frames, Tensor<T> masks)