UniVSTModel<T>
Models & Types · AiDotNet.Diffusion.StyleTransfer
UniVST: training-free LOCALIZED video style transfer over a frozen latent diffusion model (Song, Lin, Zhan, Yan, Cao and Ji, arXiv:2410.20084, TPAMI 2025).
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → UniVSTModel<T>
For Beginners
Give it a video, a mask on the first frame, and a style image. It restyles just that region across the whole video, tracking the region itself and smoothing the result so it does not flicker.
How It Works
UniVST restyles only a chosen region of a video and leaves the rest untouched, which is what separates it from diffusion methods that transfer style across the whole frame. It trains nothing: all three of its mechanisms steer a pretrained Stable Diffusion v1.5.
The three mechanisms live in their own classes, each independently testable:
MaskPropagation (point-matching propagation over DDIM-inversion features, which
removes the external tracking model other localized-editing pipelines need),
Stylization (AdaIN at both the latent and attention level), and
Smoothing (sliding-window flow-warped smoothing fed back as refined noise).
There is no loss and no training step. The inherited training surface exists because the base class models a diffusion pipeline, but UniVST has no objective of its own: attempting to train it would be training the underlying Stable Diffusion weights, not UniVST.
Example
var model = new UniVSTModel<double>();
// one feature tensor per video frame, and the mask you drew on the first
var inversionFeatures = new List<Tensor<double>>
{
new Tensor<double>([1, 4, 64, 64]),
new Tensor<double>([1, 4, 64, 64])
};
var firstFrameMask = new Tensor<double>([1, 1, 64, 64]);
var masks = model.MaskPropagation.Propagate(inversionFeatures, firstFrameMask);
Constructors
| Constructor | Summary |
|---|---|
UniVSTModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, UniVSTOptions univstOptions, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> predictor, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) |
Creates a UniVST model. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
UniVSTAdaInStylization<T> Stylization { get; } |
Gets the AdaIN stylization helper (latent and attention level). |
UniVSTConsistentSmoothing<T> Smoothing { get; } |
Gets the sliding-window consistent smoother. |
UniVSTMaskPropagation<T> MaskPropagation { get; } |
Gets the point-matching mask propagator. |
UniVSTOptions UniVSTOptions { get; } |
Gets the UniVST-specific settings (schedules and mask hyperparameters). |
UniVSTQkvTransform<T> QkvTransform { get; } |
Gets the Q/K/V transform that carries the attention-level stylization. |
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ApplyLatentStylization(Tensor<T> editLatent, Tensor<T> contentLatent, Tensor<T> styleLatent, Tensor<T> mask, double timestepFraction) |
Applies the latent-level half of the stylization for one denoising step: AdaIN inside its window, then mask gating so only the tracked region is affected. |
double TimestepFraction(int step) |
Maps a DDIM step index to the fraction of T that UniVST’s schedules are expressed in. |
void PrepareAttentionStep(double timestepFraction, Tensor<T> contentQuery, Tensor<T> styleKey, Tensor<T> styleValue) |
Prepares QkvTransform for a denoising step by pointing it at this step’s content and style projections. |