XMem<T>
Models & Types · AiDotNet.Video.Segmentation
XMem: Long-Term Video Object Segmentation with Atkinson-Shiffrin memory model.
Inheritance: NeuralNetworkBase<T> → XMem<T>
For Beginners
XMem can track objects in hour-long videos without running out of memory. It uses three types of memory:
- Sensory memory: Very recent frames (high detail, fast to forget)
- Working memory: Important recent frames (moderate detail)
- Long-term memory: Key historical frames (compressed, permanent)
Example usage (native mode for training):
Example usage (ONNX mode for inference):
How It Works
XMem is designed for tracking objects in very long videos using a three-tier memory system inspired by human memory.
Reference: “XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model”
https://arxiv.org/abs/2207.07115
Constructors
| Constructor | Summary |
|---|---|
XMem() |
Creates an XMem model using native layers for training and inference. |
XMem(NeuralNetworkArchitecture<T> architecture, XMemOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
|
XMem(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, XMemOptions options) |
Creates an XMem model using a pretrained ONNX model for inference. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
List<Tensor<T>> TrackObjectLongTerm(List<Tensor<T>> frames, Tensor<T> initialMask) |
Tracks an object through a long video sequence. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
Routes training through XMem’s grouped encoder, memory-read, and decoder graph. |
Tensor<T> SegmentFrame(Tensor<T> frame) |
Segments a single frame using the memory hierarchy. |
ValueTuple<int, int, int> GetMemoryStats() |
Gets memory statistics. |
void ClearAllMemory() |
Clears all memory banks. |