VisionMamba<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Mamba

Vision Mamba (Vim): Bidirectional State Space Model for vision.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → SemanticSegmentationBase<T> → VisionMamba<T>

For Beginners

Efficient image segmentation with linear complexity. Dense prediction replacing ViT backbone.

Common use cases:

  • Efficient image segmentation with linear complexity
  • Dense prediction replacing ViT backbone
  • Large-scale visual understanding
  • Memory-efficient visual feature extraction

How It Works

Technical Details: the official segmenter (hustvl/Vim seg/): the Vim backbone without a class token (bidirectional Mamba layers on the patch sequence), the mixer outputs of layers 6, 12, 18 and 24 turned into stride-4, 8, 16 and 32 maps by a BEiT-style neck, mmsegmentation’s UPerNet head, and an FCN auxiliary head that adds 0.4 of its cross-entropy to the training loss. Vim-Small adds a 2D rotary position embedding, as its official configuration does. Image sides must be multiples of the 16-pixel patch. The network is checked against the official code.

Reference: Zhu et al., “Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model”, ICML 2024.

Example

// Create a Vision Mamba model for efficient semantic segmentation
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.MultiClassClassification,
    inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 150);
var model = new VisionMamba<double>(architecture);
// Every tunable is now set through the options object; these are the defaults:
var custom = new VisionMamba<double>(architecture,
    options: new VisionMambaOptions { NumClasses = 150, DropRate = 0.1, ModelSize = VisionMambaModelSize.Tiny });
            
// Or load a pre-trained ONNX model for linear-complexity segmentation
var onnxModel = new VisionMamba<double>(architecture, "vim.onnx");

Constructors

Constructor Summary
VisionMamba(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, VisionMambaOptions options) Initializes VisionMamba in native (trainable) mode.
VisionMamba(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, VisionMambaOptions options) Initializes VisionMamba in ONNX (inference-only) mode.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)