SegFormer<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Semantic

SegFormer: Simple and Efficient Semantic Segmentation with Transformers.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → SemanticSegmentationBase<T> → SegFormer<T>

For Beginners

SegFormer is a semantic segmentation model that classifies every pixel in an image into a category (e.g., road, sky, person, building). It uses a hierarchical transformer encoder (Mix Transformer / MiT) and a lightweight MLP decoder — no complex decoders or positional encodings needed.

Common use cases:

  • Autonomous driving (road, lane, obstacle detection)
  • Medical imaging (organ/lesion segmentation)
  • Scene understanding (indoor/outdoor parsing)
  • Agriculture (crop/weed detection from drone images)

How It Works

Technical Details:

  • 4-stage hierarchical encoder producing multi-scale features (1/4, 1/8, 1/16, 1/32)
  • Overlapping patch embedding replaces non-overlapping ViT patches
  • Efficient Self-Attention with spatial reduction for computational savings
  • Mix-FFN uses 3x3 depthwise convolutions instead of positional encodings
  • Lightweight All-MLP decode head for fast inference
  • Six model sizes (B0–B5) from 3.8M to 82.0M parameters

Reference: Xie et al., “SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers”, NeurIPS 2021.

Example

// Create a SegFormer model for efficient semantic segmentation
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.MultiClassClassification,
    inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 150);
var model = new SegFormer<double>(architecture);
// Every tunable is now set through the options object; these are the defaults:
var custom = new SegFormer<double>(architecture,
    options: new SegFormerOptions { NumClasses = 150, DropRate = 0.1, ModelSize = SegFormerModelSize.B0 });
            
// Or load a pre-trained ONNX model for autonomous driving
var onnxModel = new SegFormer<double>(architecture, "segformer_b0.onnx");

Constructors

Constructor Summary
SegFormer(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, SegFormerOptions options) Initializes a new instance of SegFormer in native (trainable) mode.
SegFormer(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, SegFormerOptions options) Initializes a new instance of SegFormer in ONNX (inference-only) mode.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata() Collects metadata describing this SegFormer model’s configuration and current state.