MaskAdapter<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.OpenVocabulary

Mask-Adapter: Adding SAM to open-vocabulary segmentation via mask prediction.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → OpenVocabSegmentationBase<T> → MaskAdapter<T>

For Beginners

Enhanced open-vocabulary segmentation. SAM-guided mask proposals for open-vocab models.

Common use cases:

  • Enhanced open-vocabulary segmentation
  • SAM-guided mask proposals for open-vocab models
  • Improved boundary quality in zero-shot segmentation
  • Plug-and-play mask refinement

How It Works

Technical Details:

  • SAM as mask proposal generator for open-vocabulary segmentation
  • Mask-level adaptation of CLIP features
  • Plug-and-play adapter compatible with multiple backbones
  • Improves both boundary quality and category accuracy

Reference: Xie et al., “Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation”, arXiv 2025.

Example

// Create a MaskAdapter model for SAM-enhanced open-vocabulary segmentation
// Any CLIP-style ITextEncoder supplies the text embeddings, e.g. a CLIP model from VisionLanguage.Encoders.
ITextEncoder<double> textEncoder = clipModel;
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.MultiClassClassification,
    inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 150);
var model = new MaskAdapter<double>(architecture, textEncoder);
// Every tunable is now set through the options object; these are the defaults:
var custom = new MaskAdapter<double>(architecture, textEncoder,
    options: new MaskAdapterOptions { NumClasses = 150, DropRate = 0.1 });
            
// Or load a pre-trained ONNX model for improved boundary quality
var onnxModel = new MaskAdapter<double>(architecture, "maskadapter.onnx", textEncoder, options: null);

Constructors

Constructor Summary
MaskAdapter(NeuralNetworkArchitecture<T> architecture, ITextEncoder<T> textEncoder, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, MaskAdapterOptions options) Initializes MaskAdapter in native (trainable) mode.
MaskAdapter(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, ITextEncoder<T> textEncoder, MaskAdapterOptions options) Initializes MaskAdapter in ONNX (inference-only) mode.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata() Collects metadata describing this model’s configuration.