OMGSeg<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Foundation

OMG-Seg: Is One Model Good Enough For All Segmentation?

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → OMGSeg<T>

For Beginners

OMG-Seg answers “yes” — one model handles over 10 segmentation tasks with only 70M trainable parameters. Instead of training separate models for semantic, instance, panoptic, video, interactive, and open-vocabulary segmentation, OMG-Seg uses task-specific queries to switch between tasks at inference time.

Common use cases:

  • Multi-task segmentation systems needing many task types
  • Resource-constrained environments where one model must serve all needs
  • Research comparing different segmentation paradigms
  • Production systems with diverse segmentation requirements

How It Works

Technical Details:

  • Shared transformer backbone with task-specific query embeddings
  • Each task type has its own set of queries that specialize during training
  • Supports image (semantic, instance, panoptic) and video segmentation
  • Open-vocabulary capability through text-conditioned queries
  • Only 70M trainable parameters with frozen backbone

Reference: Li et al., “OMG-Seg: Is One Model Good Enough For All Segmentation?”, CVPR 2024.

Example

// Create an OMG-Seg model for multi-task segmentation
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.MultiClassClassification,
    inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 150);
var model = new OMGSeg<double>(architecture, options: new OMGSegOptions { NumClasses = 150 });
            
// Or load a pre-trained ONNX model for unified segmentation
var onnxModel = new OMGSeg<double>(architecture, "omgseg.onnx", options: new OMGSegOptions { NumClasses = 150 });

Constructors

Constructor Summary
OMGSeg(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, OMGSegOptions options) Initializes OMG-Seg in native (trainable) mode.
OMGSeg(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, OMGSegOptions options) Initializes OMG-Seg in ONNX (inference-only) mode.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata() Collects metadata describing this OMG-Seg model’s configuration.
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image)