EoMT<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Foundation

EoMT: Encoder-only Mask Transformer for universal image segmentation.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → EoMT<T>

For Beginners

EoMT dramatically simplifies segmentation by removing the complex pixel decoder and transformer decoder used by models like Mask2Former. Instead, mask queries are inserted directly into a plain Vision Transformer (DINOv2), making EoMT 4.4x faster while maintaining competitive accuracy. Think of it as the “minimalist” approach to segmentation.

Common use cases:

  • Real-time panoptic segmentation
  • Latency-sensitive deployment (4.4x faster than Mask2Former)
  • Research into simpler segmentation architectures
  • Any scenario where speed matters more than peak accuracy

How It Works

Technical Details:

  • Uses DINOv2 as frozen backbone (ViT-S/B/L)
  • Queries inserted at intermediate ViT layers, processed alongside image tokens
  • No separate pixel decoder or transformer decoder needed
  • Query-to-mask via dot product with intermediate ViT features
  • 4.4x faster than Mask2Former-Swin-L with competitive results

Reference: Saporta et al., “Encoder-only Mask Transformer”, CVPR 2025 Highlight.

Example

// Create an EoMT model for fast panoptic segmentation (4.4x faster than Mask2Former)
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.MultiClassClassification,
    inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 150);
var model = new EoMT<double>(architecture, options: new EoMTOptions { NumClasses = 150 });
            
// Or load a pre-trained ONNX model for encoder-only segmentation
var onnxModel = new EoMT<double>(architecture, "eomt.onnx", options: new EoMTOptions { NumClasses = 150 });

Constructors

Constructor Summary
EoMT(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, EoMTOptions options) Initializes EoMT from an already-resolved options instance.
EoMT(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, EoMTOptions options) Initializes ONNX-mode EoMT from an already-resolved options instance.

Methods

Method Summary
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
ModelMetadata<T> GetModelMetadata() Collects metadata describing this EoMT model’s configuration.
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image)