MixedQueryTransformer<T>
Models & Types · AiDotNet.ComputerVision.Segmentation.Foundation
MixedQueryTransformer (MQ-Former): Dynamic Query Melding for Multi-Dataset Segmentation.
Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PanopticSegmentationBase<T> → MixedQueryTransformer<T>
For Beginners
MixedQueryTransformer scales mask-based segmentation across multiple diverse datasets by dynamically melding (fusing) instance queries and stuff queries through cross-attention. This allows the model to generalize well across different segmentation benchmarks without dataset-specific fine-tuning.
Common use cases:
- Multi-dataset panoptic segmentation
- Cross-domain segmentation transfer
- Production systems trained on diverse data sources
- Research in universal segmentation scaling
How It Works
Technical Details:
- Dynamic query melding: instance and stuff queries interact via cross-attention layers
- Multi-dataset training with unified query representations
- Backbone: ResNet-50 or Swin-L transformer
- Built on Mask2Former architecture with query interaction extensions
Reference: “MixedQueryTransformer: Dynamic Query Melding for Multi-Dataset Segmentation”, CVPR 2025.
Example
// Create a MixedQueryTransformer model for multi-dataset panoptic segmentation
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.MultiClassClassification,
inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 133);
var model = new MixedQueryTransformer<double>(architecture, options: new MixedQueryTransformerOptions { NumClasses = 133 });
// Or load a pre-trained ONNX model for cross-dataset segmentation
var onnxModel = new MixedQueryTransformer<double>(architecture, "querymeldnet.onnx", options: new MixedQueryTransformerOptions { NumClasses = 133 });
Constructors
| Constructor | Summary |
|---|---|
MixedQueryTransformer(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, MixedQueryTransformerOptions options) |
Initializes MixedQueryTransformer in native (trainable) mode. |
MixedQueryTransformer(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, MixedQueryTransformerOptions options) |
Initializes MixedQueryTransformer in ONNX (inference-only) mode. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
Collects metadata describing this MixedQueryTransformer model’s configuration. |
PanopticSegmentationResult<T> SegmentPanoptic(Tensor<T> image) |