UniVSTMaskPropagation<T>

Models & Types · AiDotNet.Diffusion.StyleTransfer

UniVST’s point-matching mask propagation (Song et al., arXiv:2410.20084, TPAMI 2025). Propagates a first-frame mask across a video using DDIM-inversion feature maps.

For Beginners

Given a mask drawn on the first frame, this figures out where that same region sits in every later frame, by finding which points look most alike.

How It Works

This is the component that lets UniVST drop the external tracking model other localized-editing pipelines depend on: the diffusion model’s own inversion features already encode correspondence, so a nearest-neighbour vote over them propagates the region directly.

Each point in frame i is matched against points in the ANCHOR frames — the first frame plus the previous MaskAnchorHistory frames — by cosine similarity in feature space. The k best matches vote, and the majority label becomes the point’s label. Pinning the first frame is what keeps the mask from drifting; a purely local chain accumulates error frame over frame.

Constructors

Constructor Summary
UniVSTMaskPropagation(int neighbors, int anchorHistory, double downsampleRate, int? seed) Creates a mask propagator.

Properties

Property Summary
double DownsampleRate { get; } Gets the anchor-point downsampling rate.
int AnchorHistory { get; } Gets how many preceding frames anchor the vote, in addition to the first frame.
int Neighbors { get; } Gets k, the number of nearest neighbours that vote on each point.

Methods

Method Summary
IReadOnlyList<Tensor<T>> Propagate(IReadOnlyList<Tensor<T>> features, Tensor<T> firstFrameMask) Propagates firstFrameMask across every frame.
IReadOnlyList<int> SelectAnchorFrames(int frame) Returns the anchor frame indices for frame frame: the first frame plus up to AnchorHistory immediately preceding frames, without duplicates.