MixedQueryTransformer<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Foundation

MixedQueryTransformer (MQ-Former): Dynamic Query Melding for Multi-Dataset Segmentation.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → MixedQueryTransformer<T>

For Beginners

MixedQueryTransformer scales mask-based segmentation across multiple diverse datasets by dynamically melding (fusing) instance queries and stuff queries through cross-attention. This allows the model to generalize well across different segmentation benchmarks without dataset-specific fine-tuning.

Common use cases:

  • Multi-dataset panoptic segmentation
  • Cross-domain segmentation transfer
  • Production systems trained on diverse data sources
  • Research in universal segmentation scaling

How It Works

Technical Details:

  • Dynamic query melding: instance and stuff queries interact via cross-attention layers
  • Multi-dataset training with unified query representations
  • Backbone: ResNet-50 or Swin-L transformer
  • Built on Mask2Former architecture with query interaction extensions

Reference: “MixedQueryTransformer: Dynamic Query Melding for Multi-Dataset Segmentation”, CVPR 2025.

Example

// Create a MixedQueryTransformer model for multi-dataset panoptic segmentation
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.MultiClassClassification,
    inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 133);
var model = new MixedQueryTransformer<double>(architecture, options: new MixedQueryTransformerOptions { NumClasses = 133 });
            
// Or load a pre-trained ONNX model for cross-dataset segmentation
var onnxModel = new MixedQueryTransformer<double>(architecture, "querymeldnet.onnx", options: new MixedQueryTransformerOptions { NumClasses = 133 });

Constructors

Constructor Summary
MixedQueryTransformer(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, MixedQueryTransformerOptions options) Initializes MixedQueryTransformer in native (trainable) mode.
MixedQueryTransformer(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, MixedQueryTransformerOptions options) Initializes MixedQueryTransformer in ONNX (inference-only) mode.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata() Collects metadata describing this MixedQueryTransformer model’s configuration.
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image)