MaskDINO<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Foundation

Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → MaskDINO<T>

For Beginners

Mask DINO extends the powerful DINO object detector with a mask prediction branch, creating a unified architecture that handles object detection, instance segmentation, panoptic segmentation, and semantic segmentation all in one model. Instead of building separate models for each task, Mask DINO uses a shared backbone and query-based transformer to do everything.

Common use cases:

  • Joint object detection + instance segmentation
  • Panoptic segmentation (detecting all things and stuff)
  • Research requiring a unified detection-segmentation framework
  • Production systems needing both detection boxes and segmentation masks

How It Works

Technical Details:

  • Built on DINO detector with deformable attention transformer encoder-decoder
  • Adds a mask branch using dot product between query embeddings and pixel embeddings
  • Unified query matching for both box and mask predictions via Hungarian matching
  • Backbone: ResNet-50 or Swin-L
  • Achieves 54.5 AP on COCO instance, 59.4 PQ on COCO panoptic

Reference: Li et al., “Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation”, CVPR 2023.

Example

// Create a MaskDINO model for joint detection and segmentation
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.MultiClassClassification,
    inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 80);
var model = new MaskDINO<double>(architecture, options: new MaskDINOOptions { NumClasses = 80 });
            
// Or load a pre-trained ONNX model for unified detection-segmentation
var onnxModel = new MaskDINO<double>(architecture, "maskdino.onnx", options: new MaskDINOOptions { NumClasses = 80 });

Constructors

Constructor Summary
MaskDINO(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, MaskDINOOptions options) Initializes Mask DINO in native (trainable) mode.
MaskDINO(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, MaskDINOOptions options) Initializes Mask DINO in ONNX (inference-only) mode.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata() Collects metadata describing this Mask DINO model’s configuration.
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image)