ODISE<T>
Models & Types · AiDotNet.ComputerVision.Segmentation.Panoptic
ODISE: Open-vocabulary DIffusion-based panoptic SEgmentation.
Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → ODISE<T>
For Beginners
Open-vocabulary panoptic segmentation. Segmenting novel categories not seen during training.
Common use cases:
- Open-vocabulary panoptic segmentation
- Segmenting novel categories not seen during training
- Creative content understanding
- Cross-domain scene parsing
How It Works
Technical Details:
- Leverages Stable Diffusion internal representations for segmentation
- Text-image discriminative model (CLIP) for category assignment
- Mask generator trained on diffusion features
- Zero-shot panoptic segmentation capability
Reference: Xu et al., “Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models”, CVPR 2023.
Example
// Create an ODISE model for open-vocabulary panoptic segmentation
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.MultiClassClassification,
inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 133);
// textEncoder and clipImageEncoder are a frozen CLIP's two towers (for example an OpenCLIP model from
// AiDotNet.VisionLanguage.Encoders, which implements both ITextEncoder and IVisualEncoder).
var model = new ODISE<double>(architecture, clip, clip);
// Every tunable is now set through the options object; these are the defaults:
var custom = new ODISE<double>(architecture, clip, clip,
options: new ODISEOptions { NumClasses = 133, DropRate = 0.1, ModelSize = ODISEModelSize.Base });
// Or load a pre-trained ONNX model for diffusion-based segmentation
var onnxModel = new ODISE<double>(architecture, "odise.onnx", clip, clip, options: null);
Constructors
| Constructor | Summary |
|---|---|
ODISE(NeuralNetworkArchitecture<T> architecture, ITextEncoder<T> textEncoder, IVisualEncoder<T> clipImageEncoder, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, ODISEOptions options) |
Initializes ODISE in native (trainable) mode. |
ODISE(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, ITextEncoder<T> textEncoder, IVisualEncoder<T> clipImageEncoder, ODISEOptions options) |
Initializes ODISE in ONNX (inference-only) mode. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
ModelMetadata<T> GetModelMetadata() |
Collects metadata describing this model’s configuration. |
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image) |
|
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image, IReadOnlyList<string> stuffClassNames, IReadOnlyList<string> thingClassNames) |
Open-vocabulary panoptic segmentation: segments image into the given stuff and thing categories, which need not be the training classes. |
Tensor<T> ForwardForTraining(Tensor<T> input) |
Training forward MUST use the same skip-connection path as inference so the lazy decoder convolutions resolve their input channel counts against the CONCATENATED feature maps. |