ODISESegmentation<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Diffusion

ODISE Segmentation: Panoptic segmentation via diffusion model features.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → ODISESegmentation<T>

For Beginners

Diffusion-based panoptic segmentation. Open-vocabulary scene understanding.

Common use cases:

  • Diffusion-based panoptic segmentation
  • Open-vocabulary scene understanding
  • Text-guided segmentation via diffusion
  • Novel category segmentation

How It Works

Technical Details:

  • Stable Diffusion UNet features for dense prediction
  • CLIP text encoder for open-vocabulary category matching
  • Mask generator trained on frozen diffusion features
  • Joint text-image representation for panoptic labels

Reference: Xu et al., “Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models”, CVPR 2023.

Example

// Create an ODISE model for open-vocabulary panoptic segmentation
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.MultiClassClassification,
    inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 150);
var model = new ODISESegmentation<double>(architecture);
// Every tunable is now set through the options object; these are the defaults:
var custom = new ODISESegmentation<double>(architecture,
    options: new ODISESegmentationOptions { NumClasses = 133, DropRate = 0.1 });
            
// Or load a pre-trained ONNX model
var onnxModel = new ODISESegmentation<double>(architecture, "odise_model.onnx");

Constructors

Constructor Summary
ODISESegmentation(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, ODISESegmentationOptions options) Initializes ODISESegmentation in native (trainable) mode.
ODISESegmentation(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, ODISESegmentationOptions options) Initializes ODISESegmentation in ONNX (inference-only) mode.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata() Collects metadata describing this model’s configuration.
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image)