InstructVid2VidModel<T>
Models & Types · AiDotNet.Diffusion.Video.VideoEditing
InstructVid2Vid natural language instruction-based video editing.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → InstructVid2VidModel<T>
For Beginners
InstructVid2Vid allows natural language video editing - describe what you want changed (e.g., make it snowy) and the model applies that edit across all frames while preserving motion and structure.
How It Works
References:
- Paper: “InstructVid2Vid: Controllable Video Editing with Natural Language Instructions” (ICME, 2024)
InstructVid2Vid enables video editing through natural language instructions, similar to InstructPix2Pix but extended to video. The model learns to follow editing instructions while maintaining temporal consistency across frames. Supports diverse edits including style transfer, object manipulation, and scene modification.
Technical specifications:
- Architecture: Instruction-Conditioned UNet + Temporal Consistency
- Latent channels: 4
- Default: 24 frames at 8 FPS
- Supports I2V: No | T2V: Yes | V2V: Yes
Example
var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 50 };
var input = Tensor<float>.CreateRandom(new[] { 1, 4, 24, 64, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new InstructVid2VidModel<float>(options: options))
.Build(trainX, trainY);
var edited = result.Predict(input);
Constructors
| Constructor | Summary |
|---|---|
InstructVid2VidModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, VideoUNetPredictor<T> predictor, TemporalVAE<T> temporalVAE, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) |
Initializes a new instance of InstructVid2VidModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> TemporalVAE { get; } |
|
IVAEModel<T> VAE { get; } |
|
bool SupportsImageToVideo { get; } |
|
bool SupportsTextToVideo { get; } |
|
bool SupportsVideoToVideo { get; } |
|
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |