SegGPT<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Interactive

SegGPT: Segmenting Everything In Context via in-context learning.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PromptableSegmentationBase<T> → SegGPT<T>

For Beginners

In-context segmentation from examples. Few-shot segmentation without fine-tuning.

Common use cases:

  • In-context segmentation from examples
  • Few-shot segmentation without fine-tuning
  • Interactive image editing
  • Versatile segmentation across domains

How It Works

Technical Details:

  • In-context learning for segmentation (no fine-tuning needed)
  • ViT-Large backbone with random color mapping training
  • Feature ensemble from multiple in-context examples
  • Unified framework for semantic, instance, and part segmentation

Reference: Wang et al., “SegGPT: Segmenting Everything In Context”, ICCV 2023.

Example

// Create a SegGPT model for in-context few-shot segmentation
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.BinaryClassification,
    inputHeight: 448, inputWidth: 448, inputDepth: 3, outputSize: 1);
var model = new SegGPT<double>(architecture);
// Every tunable is now set through the options object; these are the defaults:
var custom = new SegGPT<double>(architecture,
    options: new SegGPTOptions { NumClasses = 1, DropRate = 0.1, ModelSize = SegGPTModelSize.ViTLarge });
            
// Or load a pre-trained ONNX model for example-guided segmentation
var onnxModel = new SegGPT<double>(architecture, "seggpt.onnx");

Constructors

Constructor Summary
SegGPT(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, SegGPTOptions options) Initializes SegGPT in native (trainable) mode.
SegGPT(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, SegGPTOptions options) Initializes SegGPT in ONNX (inference-only) mode.

Methods

Method Summary
List<PromptedSegmentationResult<T>> SegmentEverything()
ModelMetadata<T> GetModelMetadata() Collects metadata describing this model’s configuration.
PromptedSegmentationResult<T> SegmentFromBox(Tensor<T> box)
PromptedSegmentationResult<T> SegmentFromMask(Tensor<T> mask)
PromptedSegmentationResult<T> SegmentFromPoints(Tensor<T> points, Tensor<T> labels)
void SetImage(Tensor<T> image)