OneFormer<T>
Models & Types · AiDotNet.ComputerVision.Segmentation.Foundation
OneFormer: One Transformer to Rule Universal Image Segmentation.
Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → OneFormer<T>
For Beginners
OneFormer is trained once on panoptic data and can then perform any segmentation task — semantic, instance, or panoptic — by simply providing a text prompt that describes which task to perform. This “one model for all tasks” approach dramatically simplifies deployment compared to maintaining separate models for each task.
Example usage:
- Pass “the task is semantic” to get per-pixel class labels
- Pass “the task is instance” to get individual object masks
- Pass “the task is panoptic” to get both stuff and thing segments
Common use cases:
- Multi-task segmentation systems needing all three task types
- Research comparing segmentation approaches
- Production systems where maintaining one model is simpler than three
How It Works
Technical Details:
- Builds on Mask2Former with a text encoder (CLIP-based) for task conditioning
- Task-conditioned joint training on panoptic, semantic, and instance data simultaneously
- Uses a task-guided query initialization that focuses queries on the specified task
- Backbone: Swin-L or DiNAT-L (Dilated Neighborhood Attention Transformer)
- SOTA on ADE20K, Cityscapes, and COCO across all three tasks with a single model
Reference: Jain et al., “OneFormer: One Transformer to Rule Universal Image Segmentation”, CVPR 2023.
Example
// Create a OneFormer model for task-conditioned universal segmentation
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.MultiClassClassification,
inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 150);
var model = new OneFormer<double>(architecture, options: new OneFormerOptions { NumClasses = 150 });
// Or load a pre-trained ONNX model for text-prompted segmentation
var onnxModel = new OneFormer<double>(architecture, "oneformer.onnx", options: new OneFormerOptions { NumClasses = 150 });
Constructors
| Constructor | Summary |
|---|---|
OneFormer(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, OneFormerOptions options) |
Initializes OneFormer in native (trainable) mode. |
OneFormer(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, OneFormerOptions options) |
Initializes OneFormer in ONNX (inference-only) mode. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
Captures activations after applying OneFormer’s required leading batch reshape. |
ModelMetadata<T> GetModelMetadata() |
Collects model metadata. |
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
Routes training through OneFormer’s native NCHW path without removing the batch dimension. |