STTN<T>
Models & Types · AiDotNet.Video.Inpainting
STTN spatial-temporal transformer network for video inpainting with multi-scale attention.
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoInpaintingBase<T> → STTN<T>
For Beginners
STTN (Spatial-Temporal Transformer Network) performs video inpainting using transformers that jointly attend to spatial and temporal dimensions to fill holes consistently across frames.
How It Works
References:
- Paper: “Learning Joint Spatial-Temporal Transformations for Video Inpainting” (Zeng et al., ECCV 2020)
STTN uses multi-scale spatial-temporal transformers that jointly search for and attend to relevant patches across both space and time dimensions. Multi-head attention at multiple feature scales enables both fine-grained texture transfer and large-scale structure completion.
Example
// Create an STTN model for spatial-temporal video inpainting
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.Generative,
inputHeight: 256, inputWidth: 256, inputDepth: 3);
var options = new STTNOptions();
var sttn = new STTN<double>(architecture, options);
// Or load a pre-trained ONNX model for inference
var sttnOnnx = new STTN<double>(architecture, "sttn_model.onnx");
Constructors
| Constructor | Summary |
|---|---|
STTN(NeuralNetworkArchitecture<T> architecture, STTNOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a STTN model for native training and inference. |
STTN(NeuralNetworkArchitecture<T> architecture, string modelPath, STTNOptions options) |
Creates a STTN model for native training and inference. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Inpaint(Tensor<T> frames, Tensor<T> masks) |