InstructVid2VidModel<T>

Models & Types · AiDotNet.Diffusion.Video.VideoEditing

InstructVid2Vid natural language instruction-based video editing.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → InstructVid2VidModel<T>

For Beginners

InstructVid2Vid allows natural language video editing - describe what you want changed (e.g., make it snowy) and the model applies that edit across all frames while preserving motion and structure.

How It Works

References:

  • Paper: “InstructVid2Vid: Controllable Video Editing with Natural Language Instructions” (ICME, 2024)

InstructVid2Vid enables video editing through natural language instructions, similar to InstructPix2Pix but extended to video. The model learns to follow editing instructions while maintaining temporal consistency across frames. Supports diverse edits including style transfer, object manipulation, and scene modification.

Technical specifications:

  • Architecture: Instruction-Conditioned UNet + Temporal Consistency
  • Latent channels: 4
  • Default: 24 frames at 8 FPS
  • Supports I2V: No | T2V: Yes | V2V: Yes

Example

var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 50 };
var input = Tensor<float>.CreateRandom(new[] { 1, 4, 24, 64, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new InstructVid2VidModel<float>(options: options))
    .Build(trainX, trainY);
var edited = result.Predict(input);

Constructors

Constructor Summary
InstructVid2VidModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, VideoUNetPredictor<T> predictor, TemporalVAE<T> temporalVAE, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) Initializes a new instance of InstructVid2VidModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> TemporalVAE { get; }
IVAEModel<T> VAE { get; }
bool SupportsImageToVideo { get; }
bool SupportsTextToVideo { get; }
bool SupportsVideoToVideo { get; }
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()