Mcnet<T>
Models & Types · AiDotNet.Video.Prediction
MCnet: future-frame prediction by decomposing a video into separate motion and content pathways.
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → Mcnet<T>
For Beginners
Given several frames of video, this predicts what comes next. It reads “how things are moving” from the differences between consecutive frames and “what things look like” from the most recent frame, then recombines the two to draw the next frame.
How It Works
Villegas, Yang, Hong, Lin and Lee, “Decomposing Motion and Content for Natural Video Sequence Prediction” (ICLR 2017, arXiv:1706.08033).
REPLACES DRVI, AND CHANGES THE TASK. That class cited arXiv:2401.05765 as “DRVI: Disentangled Representations for Video Interpolation” by Rong Du and Zhiwei Xiong. That identifier is “A new computationally efficient algorithm to solve Feature Selection for Functional Data Classification in high-dimensional spaces” — functional-data feature selection, unrelated to video. Title, authors, year and subject were all invented.
More importantly the TASK was wrong. MCnet is video PREDICTION — it observes x_1..x_t and
predicts x_hat_{t+1} onward — whereas DRVI lived in Video/FrameInterpolation and
implemented Interpolate(frame0, frame1, t), synthesising a frame BETWEEN two known frames.
The interpolation contract cannot express prediction: there is no second frame to interpolate
towards. Hence the move to Video/Prediction and the change of base class.
The asymmetry is the point: motion reads CHANGES between frames, content reads one frame. A static
scene differences to zero regardless of what it depicts, so the motion pathway has nothing about
appearance to latch onto — see McnetDecomposition.
The objective adds a GRADIENT DIFFERENCE term to the pixel loss (see McnetLoss),
which is what keeps predictions sharp: minimising pixel error alone yields a blurry average over
plausible futures.
Example
var observedFrames = Tensor<double>.CreateRandom(1, 3, 32, 32);
var arch = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.Regression,
inputHeight: 64, inputWidth: 64, inputDepth: 3);
var model = new Mcnet<double>(arch, new McnetOptions { NumInputFrames = 4 });
var next = model.PredictNextFrame(observedFrames); // [time, H, W, C] -> [H, W, C]
Constructors
| Constructor | Summary |
|---|---|
Mcnet(NeuralNetworkArchitecture<T> architecture, McnetOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an MCnet model in native training mode. |
Mcnet(NeuralNetworkArchitecture<T> architecture, string modelPath, McnetOptions options) |
Creates an MCnet model in native training mode. |
Properties
| Property | Summary |
|---|---|
McnetDecomposition<T> Decomposition { get; } |
Gets the motion/content decomposition helper. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> PredictNextFrame(Tensor<T> observedFrames) |
Predicts the next frame from a sequence of observed frames. |
double ComputeLoss(Vector<T> predicted, Vector<T> target, int height, int width, double? discriminatorOnGenerated) |
The paper’s objective for one prediction: alpha * L_img + beta * L_GAN. |