VisionMamba<T>
Models & Types · AiDotNet.ComputerVision.Segmentation.Mamba
Vision Mamba (Vim): Bidirectional State Space Model for vision.
Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → SemanticSegmentationBase<T> → VisionMamba<T>
For Beginners
Efficient image segmentation with linear complexity. Dense prediction replacing ViT backbone.
Common use cases:
- Efficient image segmentation with linear complexity
- Dense prediction replacing ViT backbone
- Large-scale visual understanding
- Memory-efficient visual feature extraction
How It Works
Technical Details: the official segmenter (hustvl/Vim seg/): the Vim backbone without a class token
(bidirectional Mamba layers on the patch sequence), the mixer outputs of layers 6, 12, 18 and 24 turned into
stride-4, 8, 16 and 32 maps by a BEiT-style neck, mmsegmentation’s UPerNet head, and an FCN auxiliary head that adds
0.4 of its cross-entropy to the training loss. Vim-Small adds a 2D rotary position embedding, as its official
configuration does. Image sides must be multiples of the 16-pixel patch. The network is checked against the
official code.
Reference: Zhu et al., “Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model”, ICML 2024.
Example
// Create a Vision Mamba model for efficient semantic segmentation
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.MultiClassClassification,
inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 150);
var model = new VisionMamba<double>(architecture);
// Every tunable is now set through the options object; these are the defaults:
var custom = new VisionMamba<double>(architecture,
options: new VisionMambaOptions { NumClasses = 150, DropRate = 0.1, ModelSize = VisionMambaModelSize.Tiny });
// Or load a pre-trained ONNX model for linear-complexity segmentation
var onnxModel = new VisionMamba<double>(architecture, "vim.onnx");
Constructors
| Constructor | Summary |
|---|---|
VisionMamba(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, VisionMambaOptions options) |
Initializes VisionMamba in native (trainable) mode. |
VisionMamba(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, VisionMambaOptions options) |
Initializes VisionMamba in ONNX (inference-only) mode. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |