FreeNoiseVideoModel<T>

Models & Types · AiDotNet.Diffusion.Video.LongVideo

FreeNoise extended video generation through noise rescheduling.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → FreeNoiseVideoModel<T>

For Beginners

FreeNoise extends short video generators to produce longer sequences by intelligently rescheduling the noise patterns. This simple technique enables existing models to generate longer videos without retraining.

How It Works

References:

  • Paper: “FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling” (2024)

FreeNoise enables longer video generation without additional training by rescheduling the noise initialization. Instead of independent noise for each frame, it uses temporally correlated noise with a sliding window scheme. This produces longer, more coherent videos from any short-video diffusion model.

Technical specifications:

  • Architecture: Noise Rescheduling + Sliding Window + Temporal Correlation
  • Latent channels: 4
  • Default: 64 frames at 8 FPS
  • Supports I2V: Yes | T2V: Yes | V2V: No

Example

var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 50 };
var noise = Tensor<float>.CreateRandom(new[] { 1, 4, 64, 64, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new FreeNoiseVideoModel<float>(options: options))
    .Build(trainX, trainY);
var video = result.Predict(noise);

Constructors

Constructor Summary
FreeNoiseVideoModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, VideoUNetPredictor<T> predictor, TemporalVAE<T> temporalVAE, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS) Initializes a new instance of FreeNoiseVideoModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> TemporalVAE { get; }
IVAEModel<T> VAE { get; }
bool SupportsImageToVideo { get; }
bool SupportsTextToVideo { get; }
bool SupportsVideoToVideo { get; }
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()