McnetDecomposition<T>

Models & Types · AiDotNet.Video.Prediction

MCnet’s motion/content decomposition: the asymmetric split that gives the model its name.

For Beginners

Two things matter when predicting the next video frame: what the scene looks like, and how it is moving. This separates them — movement is read from the CHANGES between consecutive frames, appearance from a single frame — so the model does not have to relearn what a chair looks like every time the camera pans.

How It Works

Villegas et al., arXiv:1706.08033.

The asymmetry IS the decomposition, and it is easy to flatten. The motion pathway consumes IMAGE DIFFERENCES x_t - x_{t-1} recurrently; the content pathway consumes a SINGLE frame, x_t, with no recurrence. Feeding both pathways the same frames — or feeding the motion pathway raw frames rather than differences — produces two encoders that learn overlapping representations, which is precisely the entanglement the paper removes.

This type owns the differencing, the combination and the per-scale residual pairing. The learned parts (the convolutional LSTM, the CNNs) live in the owning model’s layer collection so their weights are real parameters visible to the optimizer and the gradient tape.

Constructors

Constructor Summary
McnetDecomposition(int scales) Creates the decomposition helper.

Properties

Property Summary
int Scales { get; } Gets the number of scales at which residual features are communicated.

Methods

Method Summary
Tensor<T> Combine(Tensor<T> motion, Tensor<T> content) [d_t, s_t]: concatenates motion and content features along the channel axis, ready for g_comb.
Tensor<T> ContentInput(Tensor<T> frames) The content pathway’s input: the LAST observed frame only.
Tensor<T> MotionInput(Tensor<T> frames) The motion pathway’s input: the difference sequence x_t - x_{t-1}.
Tensor<T> ResidualPair(Tensor<T> contentAtScale, Tensor<T> motionAtScale, int scale) Pairs the two pathways’ features at a given scale for r_t^l = f_res([s_t^l, d_t^l])^l.