XMem<T>

Models & Types · AiDotNet.Video.Segmentation

XMem: Long-Term Video Object Segmentation with Atkinson-Shiffrin memory model.

Inheritance: NeuralNetworkBase<T> → XMem<T>

For Beginners

XMem can track objects in hour-long videos without running out of memory. It uses three types of memory:

  • Sensory memory: Very recent frames (high detail, fast to forget)
  • Working memory: Important recent frames (moderate detail)
  • Long-term memory: Key historical frames (compressed, permanent)

Example usage (native mode for training):

Example usage (ONNX mode for inference):

How It Works

XMem is designed for tracking objects in very long videos using a three-tier memory system inspired by human memory.

Reference: “XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model” https://arxiv.org/abs/2207.07115

Constructors

Constructor Summary
XMem() Creates an XMem model using native layers for training and inference.
XMem(NeuralNetworkArchitecture<T> architecture, XMemOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction)
XMem(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, XMemOptions options) Creates an XMem model using a pretrained ONNX model for inference.

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
List<Tensor<T>> TrackObjectLongTerm(List<Tensor<T>> frames, Tensor<T> initialMask) Tracks an object through a long video sequence.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input) Routes training through XMem’s grouped encoder, memory-read, and decoder graph.
Tensor<T> SegmentFrame(Tensor<T> frame) Segments a single frame using the memory hierarchy.
ValueTuple<int, int, int> GetMemoryStats() Gets memory statistics.
void ClearAllMemory() Clears all memory banks.