Mcnet<T>

Models & Types · AiDotNet.Video.Prediction

MCnet: future-frame prediction by decomposing a video into separate motion and content pathways.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → Mcnet<T>

For Beginners

Given several frames of video, this predicts what comes next. It reads “how things are moving” from the differences between consecutive frames and “what things look like” from the most recent frame, then recombines the two to draw the next frame.

How It Works

Villegas, Yang, Hong, Lin and Lee, “Decomposing Motion and Content for Natural Video Sequence Prediction” (ICLR 2017, arXiv:1706.08033).

REPLACES DRVI, AND CHANGES THE TASK. That class cited arXiv:2401.05765 as “DRVI: Disentangled Representations for Video Interpolation” by Rong Du and Zhiwei Xiong. That identifier is “A new computationally efficient algorithm to solve Feature Selection for Functional Data Classification in high-dimensional spaces” — functional-data feature selection, unrelated to video. Title, authors, year and subject were all invented.

More importantly the TASK was wrong. MCnet is video PREDICTION — it observes x_1..x_t and predicts x_hat_{t+1} onward — whereas DRVI lived in Video/FrameInterpolation and implemented Interpolate(frame0, frame1, t), synthesising a frame BETWEEN two known frames. The interpolation contract cannot express prediction: there is no second frame to interpolate towards. Hence the move to Video/Prediction and the change of base class.

The asymmetry is the point: motion reads CHANGES between frames, content reads one frame. A static scene differences to zero regardless of what it depicts, so the motion pathway has nothing about appearance to latch onto — see McnetDecomposition.

The objective adds a GRADIENT DIFFERENCE term to the pixel loss (see McnetLoss), which is what keeps predictions sharp: minimising pixel error alone yields a blurry average over plausible futures.

Example

var observedFrames = Tensor<double>.CreateRandom(1, 3, 32, 32);
var arch = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.Regression,
    inputHeight: 64, inputWidth: 64, inputDepth: 3);
            
var model = new Mcnet<double>(arch, new McnetOptions { NumInputFrames = 4 });
var next = model.PredictNextFrame(observedFrames);   // [time, H, W, C] -> [H, W, C]

Constructors

Constructor Summary
Mcnet(NeuralNetworkArchitecture<T> architecture, McnetOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an MCnet model in native training mode.
Mcnet(NeuralNetworkArchitecture<T> architecture, string modelPath, McnetOptions options) Creates an MCnet model in native training mode.

Properties

Property Summary
McnetDecomposition<T> Decomposition { get; } Gets the motion/content decomposition helper.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> PredictNextFrame(Tensor<T> observedFrames) Predicts the next frame from a sequence of observed frames.
double ComputeLoss(Vector<T> predicted, Vector<T> target, int height, int width, double? discriminatorOnGenerated) The paper’s objective for one prediction: alpha * L_img + beta * L_GAN.