IARTOptions

Options & Configuration · AiDotNet.Video.Options

Configuration options for IART, the implicit-resampling alignment recurrent video super-resolution transformer.

Inheritance: ModelOptions → NeuralNetworkOptions → IARTOptions

For Beginners

IART walks through a video forward and backward in time. To line up the previous frame with the current one, each pixel follows the motion and then learns how to blend the four pixels around where it lands.

How It Works

Defaults follow Xu et al., CVPR 2024, and the authors’ release (kai422/IART, archs/iart_arch.py, archs/implicit_alignment.py and options/IART_REDS_N16_600K.yml): 64-channel reconstruction, 120-channel embeddings, three RSTBs of six window-attention layers with 6 heads over 3x8x8 windows, MLP ratio 2, stochastic depth up to 0.1, an implicit warp with 6 heads and sine codes at temperature 0.01, 16-frame training sequences, Charbonnier loss (eps 1e-12 under the square root), Adam (0.9, 0.99) at 2e-4 with SPyNet at 0.125 times the rate and frozen for the first 5,000 iterations, cosine-annealed to 1e-7 over 600,000 iterations, and an exponential moving average of the weights (decay 0.999) for inference.

Constructors

Constructor Summary
IARTOptions() Initializes a new instance with default values.
IARTOptions(IARTOptions other) Initializes a new instance by copying from another instance.

Properties

Property Summary
OnnxModelOptions OnnxOptions { get; set; } Gets or sets the ONNX runtime options.
double CharbonnierEpsilon { get; set; } Gets or sets the Charbonnier ε; BasicSR’s loss is sqrt(d² + 1e-12), so ε = 1e-6.
double DropPathRate { get; set; } Gets or sets the largest stochastic-depth rate, reached by the last layer of each transformer (0.1).
double LearningRate { get; set; } Gets or sets Adam’s starting learning rate (2e-4).
double MinLearningRate { get; set; } Gets or sets the floor of the cosine schedule (1e-7).
double MlpRatio { get; set; } Gets or sets the MLP width as a multiple of the embedding (2).
double PositionTemperature { get; set; } Gets or sets the temperature of the implicit warp’s sine position codes (pe_temp = 0.01).
double SpyNetLearningRateMultiplier { get; set; } Gets or sets the multiplier of SPyNet’s learning rate (0.125, i.e.
double? EmaMomentum { get; set; } Gets or sets the momentum of the weights’ moving average used for inference (0.001, ema_decay 0.999); null disables it and inference uses the trained weights.
int CosineIterations { get; set; } Gets or sets the iterations of the cosine schedule (600,000).
int EmbedDim { get; set; } Gets or sets the embedding width of every alignment transformer (120).
int FlowFreezeIterations { get; set; } Gets or sets the iterations SPyNet stays frozen at the start of training (5,000).
int MidChannels { get; set; } Gets or sets the width of the reconstruction convolutions (64).
int NumFrames { get; set; } Gets or sets the frames of a training sequence (16, the REDS recipe).
int NumHeads { get; set; } Gets or sets the window-attention heads (6).
int WindowSize { get; set; } Gets or sets the spatial window size (8; windows are 3x8x8, the three aligned frames).
int[] Depths { get; set; } Gets or sets the window-attention layers of each RSTB in an alignment transformer ([6, 6, 6]).
string ModelPath { get; set; } Gets or sets the path to the ONNX model file.