UniVSTModel<T>

Models & Types · AiDotNet.Diffusion.StyleTransfer

UniVST: training-free LOCALIZED video style transfer over a frozen latent diffusion model (Song, Lin, Zhan, Yan, Cao and Ji, arXiv:2410.20084, TPAMI 2025).

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → UniVSTModel<T>

For Beginners

Give it a video, a mask on the first frame, and a style image. It restyles just that region across the whole video, tracking the region itself and smoothing the result so it does not flicker.

How It Works

UniVST restyles only a chosen region of a video and leaves the rest untouched, which is what separates it from diffusion methods that transfer style across the whole frame. It trains nothing: all three of its mechanisms steer a pretrained Stable Diffusion v1.5.

The three mechanisms live in their own classes, each independently testable: MaskPropagation (point-matching propagation over DDIM-inversion features, which removes the external tracking model other localized-editing pipelines need), Stylization (AdaIN at both the latent and attention level), and Smoothing (sliding-window flow-warped smoothing fed back as refined noise).

There is no loss and no training step. The inherited training surface exists because the base class models a diffusion pipeline, but UniVST has no objective of its own: attempting to train it would be training the underlying Stable Diffusion weights, not UniVST.

Example

var model = new UniVSTModel<double>();
            
// one feature tensor per video frame, and the mask you drew on the first
var inversionFeatures = new List<Tensor<double>>
{
    new Tensor<double>([1, 4, 64, 64]),
    new Tensor<double>([1, 4, 64, 64])
};
var firstFrameMask = new Tensor<double>([1, 1, 64, 64]);
            
var masks = model.MaskPropagation.Propagate(inversionFeatures, firstFrameMask);

Constructors

Constructor Summary
UniVSTModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, UniVSTOptions univstOptions, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> predictor, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) Creates a UniVST model.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
UniVSTAdaInStylization<T> Stylization { get; } Gets the AdaIN stylization helper (latent and attention level).
UniVSTConsistentSmoothing<T> Smoothing { get; } Gets the sliding-window consistent smoother.
UniVSTMaskPropagation<T> MaskPropagation { get; } Gets the point-matching mask propagator.
UniVSTOptions UniVSTOptions { get; } Gets the UniVST-specific settings (schedules and mask hyperparameters).
UniVSTQkvTransform<T> QkvTransform { get; } Gets the Q/K/V transform that carries the attention-level stylization.
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> ApplyLatentStylization(Tensor<T> editLatent, Tensor<T> contentLatent, Tensor<T> styleLatent, Tensor<T> mask, double timestepFraction) Applies the latent-level half of the stylization for one denoising step: AdaIN inside its window, then mask gating so only the tracked region is affected.
double TimestepFraction(int step) Maps a DDIM step index to the fraction of T that UniVST’s schedules are expressed in.
void PrepareAttentionStep(double timestepFraction, Tensor<T> contentQuery, Tensor<T> styleKey, Tensor<T> styleValue) Prepares QkvTransform for a denoising step by pointing it at this step’s content and style projections.