RVM<T>

Models & Types · AiDotNet.Video.Matting

RVM: Robust Video Matting for real-time human segmentation.

Inheritance: NeuralNetworkBase<T> → RVM<T>

For Beginners

RVM extracts people from video backgrounds in real-time. Unlike simple background removal that creates hard edges, matting produces a soft alpha matte that preserves hair details and semi-transparent regions.

Key capabilities:

  • Real-time video matting without green screen
  • High-quality alpha matte output
  • Temporal consistency across frames
  • Works with any background

Outputs:

  • Alpha matte: Transparency at each pixel (0=background, 1=foreground)
  • Foreground: RGB colors of the person with pre-multiplied alpha

Example usage:

How It Works

Technical Details:

  • A four-stage stride-2 encoder (a ReLU6 conv stack standing in for MobileNetV3) with an LR-ASPP head
  • The paper’s recurrent decoder: a ConvGRU at the bottleneck and in each of the three upsampling blocks, on half

of the channels, carrying temporal state from frame to frame (reset with ResetState)

  • Output: alpha and a foreground residual added to the source frame, both clamped to [0, 1]
  • The deep guided filter for high-resolution refinement is not included; frames are matted at input resolution

Reference: “Robust High-Resolution Video Matting with Temporal Guidance” https://arxiv.org/abs/2108.11515

Constructors

Constructor Summary
RVM() Initializes a new instance with default architecture settings.
RVM(NeuralNetworkArchitecture<T> architecture, RVMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction)
RVM(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, RVMOptions options)

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
List<ValueTuple<Tensor<T>, Tensor<T>>> MatteVideo(List<Tensor<T>> frames) Processes a video to extract alpha mattes and foregrounds.
ModelMetadata<T> GetModelMetadata()
Tensor<T> CompositeWithBackground(Tensor<T> foreground, Tensor<T> alpha, Tensor<T> background) Composites the foreground onto a new background.
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> GetAlpha(Tensor<T> frame) Extracts just the alpha matte.
Tensor<T> GreenScreenExtract(Tensor<T> frame) Creates a green screen effect (extracts foreground).
ValueTuple<Tensor<T>, Tensor<T>> MatteSingleFrame(Tensor<T> frame) Mattes a single frame, maintaining temporal consistency with previous frames.
void ResetState() Resets the recurrent state for a new video.