ODISE<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Panoptic

ODISE: Open-vocabulary DIffusion-based panoptic SEgmentation.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → ODISE<T>

For Beginners

Open-vocabulary panoptic segmentation. Segmenting novel categories not seen during training.

Common use cases:

  • Open-vocabulary panoptic segmentation
  • Segmenting novel categories not seen during training
  • Creative content understanding
  • Cross-domain scene parsing

How It Works

Technical Details:

  • Leverages Stable Diffusion internal representations for segmentation
  • Text-image discriminative model (CLIP) for category assignment
  • Mask generator trained on diffusion features
  • Zero-shot panoptic segmentation capability

Reference: Xu et al., “Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models”, CVPR 2023.

Example

// Create an ODISE model for open-vocabulary panoptic segmentation
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.MultiClassClassification,
    inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 133);
// textEncoder and clipImageEncoder are a frozen CLIP's two towers (for example an OpenCLIP model from
// AiDotNet.VisionLanguage.Encoders, which implements both ITextEncoder and IVisualEncoder).
var model = new ODISE<double>(architecture, clip, clip);
// Every tunable is now set through the options object; these are the defaults:
var custom = new ODISE<double>(architecture, clip, clip,
    options: new ODISEOptions { NumClasses = 133, DropRate = 0.1, ModelSize = ODISEModelSize.Base });
            
// Or load a pre-trained ONNX model for diffusion-based segmentation
var onnxModel = new ODISE<double>(architecture, "odise.onnx", clip, clip, options: null);

Constructors

Constructor Summary
ODISE(NeuralNetworkArchitecture<T> architecture, ITextEncoder<T> textEncoder, IVisualEncoder<T> clipImageEncoder, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, ODISEOptions options) Initializes ODISE in native (trainable) mode.
ODISE(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, ITextEncoder<T> textEncoder, IVisualEncoder<T> clipImageEncoder, ODISEOptions options) Initializes ODISE in ONNX (inference-only) mode.

Methods

Method Summary
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
ModelMetadata<T> GetModelMetadata() Collects metadata describing this model’s configuration.
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image)
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image, IReadOnlyList<string> stuffClassNames, IReadOnlyList<string> thingClassNames) Open-vocabulary panoptic segmentation: segments image into the given stuff and thing categories, which need not be the training classes.
Tensor<T> ForwardForTraining(Tensor<T> input) Training forward MUST use the same skip-connection path as inference so the lazy decoder convolutions resolve their input channel counts against the CONCATENATED feature maps.