EoMT<T>
Models & Types · AiDotNet.ComputerVision.Segmentation.Foundation
EoMT: Encoder-only Mask Transformer for universal image segmentation.
Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → EoMT<T>
For Beginners
EoMT dramatically simplifies segmentation by removing the complex pixel decoder and transformer decoder used by models like Mask2Former. Instead, mask queries are inserted directly into a plain Vision Transformer (DINOv2), making EoMT 4.4x faster while maintaining competitive accuracy. Think of it as the “minimalist” approach to segmentation.
Common use cases:
- Real-time panoptic segmentation
- Latency-sensitive deployment (4.4x faster than Mask2Former)
- Research into simpler segmentation architectures
- Any scenario where speed matters more than peak accuracy
How It Works
Technical Details:
- Uses DINOv2 as frozen backbone (ViT-S/B/L)
- Queries inserted at intermediate ViT layers, processed alongside image tokens
- No separate pixel decoder or transformer decoder needed
- Query-to-mask via dot product with intermediate ViT features
- 4.4x faster than Mask2Former-Swin-L with competitive results
Reference: Saporta et al., “Encoder-only Mask Transformer”, CVPR 2025 Highlight.
Example
// Create an EoMT model for fast panoptic segmentation (4.4x faster than Mask2Former)
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.MultiClassClassification,
inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 150);
var model = new EoMT<double>(architecture, options: new EoMTOptions { NumClasses = 150 });
// Or load a pre-trained ONNX model for encoder-only segmentation
var onnxModel = new EoMT<double>(architecture, "eomt.onnx", options: new EoMTOptions { NumClasses = 150 });
Constructors
| Constructor | Summary |
|---|---|
EoMT(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, EoMTOptions options) |
Initializes EoMT from an already-resolved options instance. |
EoMT(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, EoMTOptions options) |
Initializes ONNX-mode EoMT from an already-resolved options instance. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
ModelMetadata<T> GetModelMetadata() |
Collects metadata describing this EoMT model’s configuration. |
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image) |