Mask2Former<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Foundation

Mask2Former: Masked-attention Mask Transformer for Universal Image Segmentation.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → Mask2Former<T>

For Beginners

Mask2Former is a universal segmentation model that handles semantic, instance, and panoptic segmentation with a single unified architecture. Instead of designing separate models for each task, Mask2Former uses a transformer decoder with “masked cross-attention” that restricts each query to attend only to its predicted mask region. This makes it highly efficient and accurate across all segmentation tasks.

Common use cases:

  • Panoptic segmentation (stuff + things in one pass, e.g., road + cars + people)
  • Instance segmentation (individual object masks, e.g., “car 1”, “car 2”)
  • Semantic segmentation (per-pixel class labels, e.g., “road”, “sky”, “building”)
  • Multi-task deployment where one model serves all segmentation needs

How It Works

Technical Details:

  • Backbone: Swin Transformer or ResNet producing multi-scale features
  • Pixel Decoder: Multi-Scale Deformable Attention Transformer (MSDeformAttn)
  • Transformer Decoder: 9 layers with masked cross-attention (restricts attention to predicted masks)
  • 100 learnable object queries predict class labels and binary masks
  • Achieves 57.8 PQ on COCO panoptic, 83.3 AP on Cityscapes instance

Reference: Cheng et al., “Masked-attention Mask Transformer for Universal Image Segmentation”, CVPR 2022.

Example

// Create a Mask2Former model for universal image segmentation
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.MultiClassClassification,
    inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 150);
var model = new Mask2Former<double>(architecture, options: new Mask2FormerOptions { NumClasses = 150 });
            
// Or load a pre-trained ONNX model for panoptic segmentation
var onnxModel = new Mask2Former<double>(architecture, "mask2former.onnx", options: new Mask2FormerOptions { NumClasses = 150 });

Constructors

Constructor Summary
Mask2Former(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, Mask2FormerOptions options) Initializes Mask2Former in native (trainable) mode.
Mask2Former(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, Mask2FormerOptions options) Initializes Mask2Former in ONNX (inference-only) mode.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata() Collects model metadata.
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image)