CUPS<T>
Models & Types · AiDotNet.ComputerVision.Segmentation.Panoptic
CUPS: Scene-Centric Unsupervised Panoptic Segmentation.
Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → CUPS<T>
For Beginners
Unified panoptic segmentation. Semantic-aware scene understanding.
Common use cases:
- Unified panoptic segmentation
- Semantic-aware scene understanding
- Comprehensive pixel-level labeling
- Autonomous driving perception
How It Works
Technical Details: CUPS trains a panoptic network without labels. Semantic pseudo labels come from
clustering self-supervised features into 27 pseudo classes and instance pseudo labels from motion in stereo
video; GeneratePanopticPseudoLabels fuses the two, and FitThingStuffSplit decides
which pseudo classes are things (more than psi_ts = 0.08 of their pixels inside instance masks).
The panoptic network is the paper’s Panoptic Cascade Mask R-CNN: a semantic head plus a Cascade Mask R-CNN
(RPN over an FPN on the backbone’s four stages, three cascade box stages at IoU 0.5/0.6/0.7 and a mask head),
trained on the pseudo labels with TrainPanoptic; SegmentPanoptic combines the
detected thing instances with the stuff regions the way Panoptic FPN does.
Reference: Hahn et al., “Scene-Centric Unsupervised Panoptic Segmentation”, CVPR 2025.
Example
// Create a CUPS model for unified panoptic segmentation
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.MultiClassClassification,
inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 133);
var model = new CUPS<double>(architecture);
// Every tunable is now set through the options object; these are the defaults:
var custom = new CUPS<double>(architecture,
options: new CUPSOptions { NumClasses = 27, DropRate = 0.1 });
// Or load a pre-trained ONNX model for scene understanding
var onnxModel = new CUPS<double>(architecture, "cups.onnx");
Constructors
| Constructor | Summary |
|---|---|
CUPS(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, CUPSOptions options) |
Initializes CUPS in native (trainable) mode. |
CUPS(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, CUPSOptions options) |
Initializes CUPS in ONNX (inference-only) mode. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<bool> FitThingStuffSplit(IReadOnlyList<Tensor<T>> semanticPseudoLabels, IReadOnlyList<Tensor<T>> instanceMasks) |
Decides which pseudo classes are things from a set of pseudo-labelled images (Hahn et al. |
ModelMetadata<T> GetModelMetadata() |
Collects metadata describing this model’s configuration. |
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image) |
|
T TrainPanoptic(Tensor<T> image, Tensor<T> semanticLabels, Tensor<T> instanceIds) |
One training step of the Panoptic Cascade Mask R-CNN on a panoptic pseudo label (Hahn et al. |
Tensor<T> ForwardForTraining(Tensor<T> input) |
The training forward is the semantic forward. |
ValueTuple<Tensor<T>, Tensor<T>> GeneratePanopticPseudoLabels(Tensor<T> semanticPseudoLabels, Tensor<T> instanceMasks) |
Fuses semantic and instance pseudo labels into a panoptic pseudo label (Hahn et al. |
bool IsThingClass(int classId) |
Whether pseudo class classId is currently treated as a thing. |