FlowWarpHelper

Helpers & Utilities · AiDotNet.Video

Backward-warps an image by an optical-flow field, using tape-recorded engine operations.

For Beginners

“warping” moves every pixel of an image along a motion field so it lines up with a different frame. If you know how things moved between frame 1 and frame 2, warping frame 1 by that motion gives you a prediction of frame 2. Video models use this constantly: it is how the previous frame’s detail gets reused for the current frame instead of being re-invented (which is what causes flicker).

How It Works

Why this exists. Several models each carried their own copy of this operation written as a nested per-pixel loop that wrote into a freshly allocated tensor. That has two consequences. First, raw buffer writes are not recorded operations, so no gradient can flow back through a warp — any loss term that compares warped content is then training nothing, and any model whose architecture depends on warping is effectively a per-frame model. Second, a scalar loop over batch x height x width x channels is orders of magnitude slower than the batched kernel underneath GridSample.

Method. Flow is converted to a normalized sampling grid and handed to GridSample, which is the same formulation as PyTorch’s grid_sample: for output pixel (x, y) the source location is (x + flow_x, y + flow_y), mapped to [-1, 1]. The pixel coordinates themselves are constants, so all gradient flows through the flow field and the source image — exactly as required for a flow-based loss or a warped-conditioning path.

Methods

Method Summary
Tensor<T> Warp(IEngine engine, Tensor<T> image, Tensor<T> flow) Backward-warps image by flow.