DiffSeg<T>
Models & Types · AiDotNet.ComputerVision.Segmentation.Semantic
DiffSeg: Unsupervised Semantic Segmentation from Diffusion Model Self-Attention Maps.
Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → SemanticSegmentationBase<T> → DiffSeg<T>
For Beginners
DiffSeg produces segmentation maps without any training labels by leveraging the self-attention maps inside a diffusion model. The idea is that diffusion models learn to “attend” to semantically similar regions when generating images, and DiffSeg repurposes those attention patterns to group pixels into coherent segments.
Common use cases:
- Unsupervised image segmentation (no labels needed at all)
- Automatic annotation/pre-labeling for new datasets
- Understanding what a diffusion model has learned about image structure
- Research into emergent visual representations in generative models
How It Works
Technical Details:
- Extracts self-attention maps from a pre-trained Stable Diffusion UNet
- Merges attention heads and layers into a single affinity matrix
- Applies iterative attention map merging to produce coherent segments
- Completely training-free: uses frozen diffusion model weights only
Reference: Tian et al., “Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion”, arXiv 2023.
Example
// Create a DiffSeg model for unsupervised segmentation from diffusion attention
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.MultiClassClassification,
inputHeight: 512, inputWidth: 512, inputDepth: 3, outputSize: 150);
var model = new DiffSeg<double>(architecture);
// Every tunable is now set through the options object; these are the defaults:
var custom = new DiffSeg<double>(architecture,
options: new DiffSegOptions { NumClasses = 150, DropRate = 0.1 });
// Or load a pre-trained ONNX model for label-free segmentation
var onnxModel = new DiffSeg<double>(architecture, "diffseg.onnx");
Constructors
| Constructor | Summary |
|---|---|
DiffSeg(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, DiffSegOptions options) |
Initializes DiffSeg in native (trainable) mode. |
DiffSeg(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, DiffSegOptions options) |
Initializes DiffSeg in ONNX (inference-only) mode. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
Collects model metadata. |