Figan<T>

Models & Types · AiDotNet.Video.FrameInterpolation

FIGAN: frame interpolation with multi-scale deep loss functions and a generative adversarial network.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → Figan<T>

For Beginners

Given two consecutive video frames, this invents the frame in between. It first guesses how things moved — roughly at low resolution, then sharpening — then drags both frames towards the middle and decides pixel by pixel which one to trust. A second network acts as a critic, judging whether the invented frame looks real, which pushes the result to be sharp rather than blurry.

How It Works

van Amersfoort, Shi, Acosta, Massa, Totz, Wang and Caballero, “Frame Interpolation with Multi-Scale Deep Loss Functions and Generative Adversarial Networks” (arXiv:1711.06045).

REPLACES TDPNet. That class cited arXiv:2404.05765 as “TDPNet: Temporal Difference Prediction Network for Video Frame Interpolation” by Pengcheng Lei and Fei Gao. That identifier is “A Novel Bi-LSTM And Transformer Architecture For Generating Tabla Music” — Indian percussion music generation. The title, authors, year and subject were all invented, and no TDPNet paper appears to exist. Its documented mechanisms (“temporal difference prediction”, “difference-aware attention”, a DifferenceThreshold) had no published basis either.

The paper’s three contributions, and where each lives:

The generator is built from the paper’s module template (Table 1): six 3x3 convolutions, N_i -> 32, four 32 -> 32 ReLU layers, then 32 -> N_o. The discriminator is eight blocks of convolution + batch normalisation + leaky ReLU with strides alternating 2 and 1 and the filter count doubling at each stride-2 block.

Example

var arch = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.Regression,
    inputHeight: 128, inputWidth: 128, inputDepth: 3);
            
var model = new Figan<double>(arch, new FiganOptions { NumScales = 3 });
var frame0 = Tensor<double>.CreateRandom(1, 3, 128, 128);
var frame1 = Tensor<double>.CreateRandom(1, 3, 128, 128);
var middle = model.Interpolate(frame0, frame1);

Constructors

Constructor Summary
Figan(NeuralNetworkArchitecture<T> architecture, FiganOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a FIGAN model in native training mode.
Figan(NeuralNetworkArchitecture<T> architecture, string modelPath, FiganOptions options) Creates a FIGAN model in native training mode.

Properties

Property Summary
FiganMultiScaleFlow<T> Flow { get; } Gets the multi-scale flow estimator and synthesiser.
IReadOnlyList<ILayer<T>> DiscriminatorLayers { get; } Gets the discriminator layers, or an empty list in ONNX mode.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t)