FlowWarpHelper
Helpers & Utilities · AiDotNet.Video
Backward-warps an image by an optical-flow field, using tape-recorded engine operations.
For Beginners
“warping” moves every pixel of an image along a motion field so it lines up with a different frame. If you know how things moved between frame 1 and frame 2, warping frame 1 by that motion gives you a prediction of frame 2. Video models use this constantly: it is how the previous frame’s detail gets reused for the current frame instead of being re-invented (which is what causes flicker).
How It Works
Why this exists. Several models each carried their own copy of this operation written as a
nested per-pixel loop that wrote into a freshly allocated tensor. That has two consequences. First,
raw buffer writes are not recorded operations, so no gradient can flow back through a warp — any
loss term that compares warped content is then training nothing, and any model whose architecture
depends on warping is effectively a per-frame model. Second, a scalar loop over
batch x height x width x channels is orders of magnitude slower than the batched kernel underneath
GridSample.
Method. Flow is converted to a normalized sampling grid and handed to GridSample,
which is the same formulation as PyTorch’s grid_sample: for output pixel (x, y) the
source location is (x + flow_x, y + flow_y), mapped to [-1, 1]. The pixel coordinates
themselves are constants, so all gradient flows through the flow field and the source image —
exactly as required for a flow-based loss or a warped-conditioning path.
Methods
| Method | Summary |
|---|---|
Tensor<T> Warp(IEngine engine, Tensor<T> image, Tensor<T> flow) |
Backward-warps image by flow. |