TemporalProcessorModule<T>
Layers · AiDotNet.NeuralNetworks.Layers
Temporal Processor Module (TPM) from Stream-DiffVSR (Shiu et al., 2025): aligns and fuses the current frame’s features with the previous frame’s warped features to enforce temporal coherence.
Inheritance: LayerBase<T> → TemporalProcessorModule<T>
Implements: IShapeContract
For Beginners
when a video is upscaled frame by frame, each frame can look good on its own while the sequence still shimmers, because the model makes slightly different choices each time. This module gives the decoder a memory: it takes the previous frame’s features (already motion- aligned to the current frame) and blends them into the current features, so detail stays put instead of flickering.
How It Works
Paper. Section “Temporal Processor Module (TPM)”: these modules are integrated “after each spatial convolutional layer in the decoder to explicitly inject temporal coherence”, they “utilize latent features from the current frame and warped features from the previous frame”, and per Fig. 2 the module “aligns and fuses these features via interpolation, convolution, and weighted fusion”. The decoder applies them at several resolutions as a “multi-scale fusion strategy”.
This implementation follows that description literally, in three stages:
- Interpolation — the previous features are bilinearly resampled to the current features’
spatial size. This is what lets one module type sit at every decoder resolution: the previous frame’s features arrive at whatever scale they were produced at.
- Convolution — the resampled previous features are projected to the current channel count,
then the concatenation of current and aligned-previous features is convolved to produce a temporal correction.
- Weighted fusion — a learned per-position, per-channel gate (a 1x1 convolution followed by
a sigmoid) decides how much of that correction to apply, and the result is added to the current features. The gate matters: how much to trust the previous frame is content-dependent, since at an occlusion or a scene cut the warped features are simply wrong, and a fixed blend weight would drag stale content into the new frame.
First frame. An online model has no previous output for the first frame. Passing
null as the previous features returns the current features unchanged, which is the correct
degenerate case — spatial-only reconstruction — rather than fabricating a zero “previous frame”
whose features would actively pull the first frame towards black.
Constructors
| Constructor | Summary |
|---|---|
TemporalProcessorModule(int kernelSize) |
Creates a temporal processor module. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
Tensor<T> Forward(Tensor<T> current, Tensor<T> warpedPrevious) |
Fuses current features with the previous frame’s warped features. |
void ClearGradients() |
|
void ResetState() |
|
void UpdateParameters(T learningRate) |