XDecoder<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Foundation

X-Decoder: Generalized Decoding for Pixel, Image, and Language.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → XDecoder<T>

For Beginners

X-Decoder is a generalist model that simultaneously handles referring segmentation (find and segment an object from a text description), open-vocabulary segmentation (segment objects from any text class list), and image captioning — all with one shared decoder. It bridges the gap between pixel-level understanding and language understanding.

Common use cases:

  • Referring segmentation (“segment the red car on the left”)
  • Open-vocabulary semantic segmentation with arbitrary class names
  • Image captioning and visual question answering
  • Multi-modal vision-language systems

How It Works

Technical Details:

  • Two-path decoder: pixel path (mask predictions) and token path (text predictions)
  • Both paths share the same cross-attention mechanism
  • Supports any combination of text, pixel, and image inputs/outputs
  • Backbone: Focal-T/B/L transformer
  • Single model handles 7+ vision-language tasks

Reference: Zou et al., “Generalized Decoding for Pixel, Image, and Language”, CVPR 2023.

Example

// Create an X-Decoder model for vision-language segmentation and captioning
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.ImageSegmentation,
    inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 150);
var model = new XDecoder<double>(architecture, options: new XDecoderOptions { NumClasses = 150 });
            
// Or load a pre-trained ONNX model for referring and open-vocabulary segmentation
var onnxModel = new XDecoder<double>(architecture, "xdecoder.onnx", options: new XDecoderOptions { NumClasses = 150 });

Constructors

Constructor Summary
XDecoder(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, XDecoderOptions options) Initializes X-Decoder from an already-resolved options instance.
XDecoder(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, XDecoderOptions options) Initializes ONNX-mode X-Decoder from an already-resolved options instance.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata() Collects metadata describing this X-Decoder model’s configuration.
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image)