CUPS<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Panoptic

CUPS: Scene-Centric Unsupervised Panoptic Segmentation.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → CUPS<T>

For Beginners

Unified panoptic segmentation. Semantic-aware scene understanding.

Common use cases:

  • Unified panoptic segmentation
  • Semantic-aware scene understanding
  • Comprehensive pixel-level labeling
  • Autonomous driving perception

How It Works

Technical Details: CUPS trains a panoptic network without labels. Semantic pseudo labels come from clustering self-supervised features into 27 pseudo classes and instance pseudo labels from motion in stereo video; GeneratePanopticPseudoLabels fuses the two, and FitThingStuffSplit decides which pseudo classes are things (more than psi_ts = 0.08 of their pixels inside instance masks). The panoptic network is the paper’s Panoptic Cascade Mask R-CNN: a semantic head plus a Cascade Mask R-CNN (RPN over an FPN on the backbone’s four stages, three cascade box stages at IoU 0.5/0.6/0.7 and a mask head), trained on the pseudo labels with TrainPanoptic; SegmentPanoptic combines the detected thing instances with the stuff regions the way Panoptic FPN does.

Reference: Hahn et al., “Scene-Centric Unsupervised Panoptic Segmentation”, CVPR 2025.

Example

// Create a CUPS model for unified panoptic segmentation
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.MultiClassClassification,
    inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 133);
var model = new CUPS<double>(architecture);
// Every tunable is now set through the options object; these are the defaults:
var custom = new CUPS<double>(architecture,
    options: new CUPSOptions { NumClasses = 27, DropRate = 0.1 });
            
// Or load a pre-trained ONNX model for scene understanding
var onnxModel = new CUPS<double>(architecture, "cups.onnx");

Constructors

Constructor Summary
CUPS(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, CUPSOptions options) Initializes CUPS in native (trainable) mode.
CUPS(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, CUPSOptions options) Initializes CUPS in ONNX (inference-only) mode.

Methods

Method Summary
IReadOnlyList<bool> FitThingStuffSplit(IReadOnlyList<Tensor<T>> semanticPseudoLabels, IReadOnlyList<Tensor<T>> instanceMasks) Decides which pseudo classes are things from a set of pseudo-labelled images (Hahn et al.
ModelMetadata<T> GetModelMetadata() Collects metadata describing this model’s configuration.
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image)
T TrainPanoptic(Tensor<T> image, Tensor<T> semanticLabels, Tensor<T> instanceIds) One training step of the Panoptic Cascade Mask R-CNN on a panoptic pseudo label (Hahn et al.
Tensor<T> ForwardForTraining(Tensor<T> input) The training forward is the semantic forward.
ValueTuple<Tensor<T>, Tensor<T>> GeneratePanopticPseudoLabels(Tensor<T> semanticPseudoLabels, Tensor<T> instanceMasks) Fuses semantic and instance pseudo labels into a panoptic pseudo label (Hahn et al.
bool IsThingClass(int classId) Whether pseudo class classId is currently treated as a thing.