GroupedSpatialTemporalShift<T>

Models & Types · AiDotNet.Video.Denoising

The grouped spatial-temporal shift module: Shift-Net’s alignment mechanism, which replaces optical flow, deformable convolution and cross-frame attention with pure tensor shifting.

For Beginners

To clean up a video frame it helps to look at neighbouring frames, but things move between them. Instead of computing where everything went, this simply slides copies of the neighbour’s features in 25 different directions and lets the following convolution pick whichever slide happens to line up. Sliding is free; searching is not.

How It Works

Li, Wang, Zhang, Wang, Loy, “A Simple Baseline for Video Restoration with Grouped Spatial-temporal Shift” (CVPR 2023, arXiv:2206.10810).

The scheme, in the paper’s terms:

Shift is FREE, which is the entire argument. Moving a feature slice costs no multiply-adds, so alignment becomes a memory operation and the model reaches the accuracy of attention-based restorers at a fraction of the cost. Anything here that quietly became a convolution or a weighted blend would defeat the point of the paper.

Both halves matter and they are different. The temporal half moves information BETWEEN frames; the spatial half moves it WITHIN a frame across 25 displacements, which is what supplies the large effective receptive field that stands in for explicit correspondence search. Implementing only the temporal shift — the obvious reading of “temporal shift” — leaves the module unable to align anything that moved sideways.

Constructors

Constructor Summary
GroupedSpatialTemporalShift()

Properties

Property Summary
IReadOnlyList<ValueTuple<int, int>> Offsets { get; } The M displacement pairs, in row-major order over Displacements.
int SliceCount { get; } The paper’s slice count M, the number of (dx, dy) displacement pairs.

Methods

Method Summary
Tensor<T> BackwardTemporalShift(Tensor<T> current, Tensor<T> next) A backward temporal shift (BTS): frame i keeps its own a group and receives the spatially-shifted b group of frame i + 1.
Tensor<T> ForwardTemporalShift(Tensor<T> current, Tensor<T> previous) A forward temporal shift (FTS): frame i keeps its own a group and receives the spatially-shifted b group of frame i - 1.
Tensor<T> GroupedSpatialShift(Tensor<T> features) The grouped SPATIAL shift: splits a feature map into SliceCount channel slices and shifts each by its own displacement.
Tensor<T> Shift(Tensor<T> features, int dx, int dy) Shifts a feature map spatially by dx, dy.
ValueTuple<Tensor<T>, Tensor<T>> SplitGroups(Tensor<T> features) Splits a feature map EQUALLY along the channel axis into the a and b groups.
double Loss(Vector<T> predicted, Vector<T> target) The paper’s loss: L = (1/T) * sum_i ||H_i - O_i||_1.

Fields

Field Summary
int BaseShiftLength The paper’s base shift length s.
int[] Displacements The displacement values the paper’s implementation uses per axis: {-9, -5, 0, 5, 9}.