ViTCoMer<T>
Models & Types · AiDotNet.ComputerVision.Segmentation.Semantic
ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction.
Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → SemanticSegmentationBase<T> → ViTCoMer<T>
For Beginners
ViT-CoMer is a hybrid model that combines a CNN branch with a Vision Transformer branch, getting the best of both worlds. CNNs excel at capturing fine local details (edges, textures), while transformers capture global context (relationships between distant objects). By fusing them, ViT-CoMer produces segmentation maps with excellent boundary quality.
Common use cases:
- High-precision boundary segmentation (medical imaging, industrial inspection)
- Scene understanding where both local and global context matter
- Applications where ViTs alone miss fine details at object boundaries
How It Works
Technical Details:
- Parallel CNN and transformer branches with cross-branch feature interaction
- CNN branch provides multi-scale local features at each ViT stage
- Bidirectional feature interaction module fuses CNN and transformer features
- Improved boundary quality over pure ViT or pure CNN approaches
Reference: Xia et al., “ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions”, CVPR 2024.
Example
// Create a ViT-CoMer hybrid CNN-Transformer model for high-quality segmentation
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 150);
var model = new ViTCoMer<double>(architecture);
// Every tunable is now set through the options object; these are the defaults:
var custom = new ViTCoMer<double>(architecture,
options: new ViTCoMerOptions { NumClasses = 150, DropRate = 0.1, ModelSize = ViTCoMerModelSize.Small });
// Or load a pre-trained ONNX model for inference
var onnxModel = new ViTCoMer<double>(architecture, "vitcomer.onnx");
Constructors
| Constructor | Summary |
|---|---|
ViTCoMer(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, ViTCoMerOptions options) |
Initializes a new instance of ViT-CoMer in native (trainable) mode. |
ViTCoMer(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, ViTCoMerOptions options) |
Initializes a new instance of ViT-CoMer in ONNX (inference-only) mode. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
ModelMetadata<T> GetModelMetadata() |
Collects model metadata. |