FuseFormer<T>
Models & Types · AiDotNet.Video.Inpainting
FuseFormer transformer-based video inpainting with fine-grained spatial-temporal fusion.
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoInpaintingBase<T> → FuseFormer<T>
For Beginners
FuseFormer uses transformer attention to fuse information from multiple frames for video inpainting. It fills missing regions by attending to relevant visible content across the entire video.
How It Works
References:
- Paper: “FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting” (Liu et al., ICCV 2021)
FuseFormer applies soft split and soft composition operations within a transformer encoder to fuse fine-grained spatial-temporal features from overlapping patches, attending to both local texture details and global structure across frames for high-quality inpainting.
Example
// Create a FuseFormer model for transformer-based video inpainting
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.Generative,
inputHeight: 256, inputWidth: 256, inputDepth: 3);
var options = new FuseFormerOptions();
var fuseFormer = new FuseFormer<double>(architecture, options);
// Or load a pre-trained ONNX model for inference
var fuseFormerOnnx = new FuseFormer<double>(architecture, "fuseformer_model.onnx");
Constructors
| Constructor | Summary |
|---|---|
FuseFormer(NeuralNetworkArchitecture<T> architecture, FuseFormerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a FuseFormer model for native training and inference. |
FuseFormer(NeuralNetworkArchitecture<T> architecture, string modelPath, FuseFormerOptions options) |
Creates a FuseFormer model for native training and inference. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Inpaint(Tensor<T> frames, Tensor<T> masks) |