XDecoder<T>
Models & Types · AiDotNet.ComputerVision.Segmentation.Foundation
X-Decoder: Generalized Decoding for Pixel, Image, and Language.
Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → XDecoder<T>
For Beginners
X-Decoder is a generalist model that simultaneously handles referring segmentation (find and segment an object from a text description), open-vocabulary segmentation (segment objects from any text class list), and image captioning — all with one shared decoder. It bridges the gap between pixel-level understanding and language understanding.
Common use cases:
- Referring segmentation (“segment the red car on the left”)
- Open-vocabulary semantic segmentation with arbitrary class names
- Image captioning and visual question answering
- Multi-modal vision-language systems
How It Works
Technical Details:
- Two-path decoder: pixel path (mask predictions) and token path (text predictions)
- Both paths share the same cross-attention mechanism
- Supports any combination of text, pixel, and image inputs/outputs
- Backbone: Focal-T/B/L transformer
- Single model handles 7+ vision-language tasks
Reference: Zou et al., “Generalized Decoding for Pixel, Image, and Language”, CVPR 2023.
Example
// Create an X-Decoder model for vision-language segmentation and captioning
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.ImageSegmentation,
inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 150);
var model = new XDecoder<double>(architecture, options: new XDecoderOptions { NumClasses = 150 });
// Or load a pre-trained ONNX model for referring and open-vocabulary segmentation
var onnxModel = new XDecoder<double>(architecture, "xdecoder.onnx", options: new XDecoderOptions { NumClasses = 150 });
Constructors
| Constructor | Summary |
|---|---|
XDecoder(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, XDecoderOptions options) |
Initializes X-Decoder from an already-resolved options instance. |
XDecoder(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, XDecoderOptions options) |
Initializes ONNX-mode X-Decoder from an already-resolved options instance. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
Collects metadata describing this X-Decoder model’s configuration. |
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image) |