Mask2Former<T>
Models & Types · AiDotNet.ComputerVision.Segmentation.Foundation
Mask2Former: Masked-attention Mask Transformer for Universal Image Segmentation.
Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → Mask2Former<T>
For Beginners
Mask2Former is a universal segmentation model that handles semantic, instance, and panoptic segmentation with a single unified architecture. Instead of designing separate models for each task, Mask2Former uses a transformer decoder with “masked cross-attention” that restricts each query to attend only to its predicted mask region. This makes it highly efficient and accurate across all segmentation tasks.
Common use cases:
- Panoptic segmentation (stuff + things in one pass, e.g., road + cars + people)
- Instance segmentation (individual object masks, e.g., “car 1”, “car 2”)
- Semantic segmentation (per-pixel class labels, e.g., “road”, “sky”, “building”)
- Multi-task deployment where one model serves all segmentation needs
How It Works
Technical Details:
- Backbone: Swin Transformer or ResNet producing multi-scale features
- Pixel Decoder: Multi-Scale Deformable Attention Transformer (MSDeformAttn)
- Transformer Decoder: 9 layers with masked cross-attention (restricts attention to predicted masks)
- 100 learnable object queries predict class labels and binary masks
- Achieves 57.8 PQ on COCO panoptic, 83.3 AP on Cityscapes instance
Reference: Cheng et al., “Masked-attention Mask Transformer for Universal Image Segmentation”, CVPR 2022.
Example
// Create a Mask2Former model for universal image segmentation
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.MultiClassClassification,
inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 150);
var model = new Mask2Former<double>(architecture, options: new Mask2FormerOptions { NumClasses = 150 });
// Or load a pre-trained ONNX model for panoptic segmentation
var onnxModel = new Mask2Former<double>(architecture, "mask2former.onnx", options: new Mask2FormerOptions { NumClasses = 150 });
Constructors
| Constructor | Summary |
|---|---|
Mask2Former(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, Mask2FormerOptions options) |
Initializes Mask2Former in native (trainable) mode. |
Mask2Former(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, Mask2FormerOptions options) |
Initializes Mask2Former in ONNX (inference-only) mode. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
Collects model metadata. |
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image) |