OneFormer<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Foundation

OneFormer: One Transformer to Rule Universal Image Segmentation.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → OneFormer<T>

For Beginners

OneFormer is trained once on panoptic data and can then perform any segmentation task — semantic, instance, or panoptic — by simply providing a text prompt that describes which task to perform. This “one model for all tasks” approach dramatically simplifies deployment compared to maintaining separate models for each task.

Example usage:

  • Pass “the task is semantic” to get per-pixel class labels
  • Pass “the task is instance” to get individual object masks
  • Pass “the task is panoptic” to get both stuff and thing segments

Common use cases:

  • Multi-task segmentation systems needing all three task types
  • Research comparing segmentation approaches
  • Production systems where maintaining one model is simpler than three

How It Works

Technical Details:

  • Builds on Mask2Former with a text encoder (CLIP-based) for task conditioning
  • Task-conditioned joint training on panoptic, semantic, and instance data simultaneously
  • Uses a task-guided query initialization that focuses queries on the specified task
  • Backbone: Swin-L or DiNAT-L (Dilated Neighborhood Attention Transformer)
  • SOTA on ADE20K, Cityscapes, and COCO across all three tasks with a single model

Reference: Jain et al., “OneFormer: One Transformer to Rule Universal Image Segmentation”, CVPR 2023.

Example

// Create a OneFormer model for task-conditioned universal segmentation
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.MultiClassClassification,
    inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 150);
var model = new OneFormer<double>(architecture, options: new OneFormerOptions { NumClasses = 150 });
            
// Or load a pre-trained ONNX model for text-prompted segmentation
var onnxModel = new OneFormer<double>(architecture, "oneformer.onnx", options: new OneFormerOptions { NumClasses = 150 });

Constructors

Constructor Summary
OneFormer(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, OneFormerOptions options) Initializes OneFormer in native (trainable) mode.
OneFormer(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, OneFormerOptions options) Initializes OneFormer in ONNX (inference-only) mode.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) Captures activations after applying OneFormer’s required leading batch reshape.
ModelMetadata<T> GetModelMetadata() Collects model metadata.
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image)
Tensor<T> ForwardForTraining(Tensor<T> input) Routes training through OneFormer’s native NCHW path without removing the batch dimension.