ReferringSegmentationBase<T>

Base Classes · AiDotNet.ComputerVision.Segmentation.Common

Abstract base class for referring segmentation models that segment objects from natural language descriptions with complex reasoning about spatial relationships and attributes.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → ReferringSegmentationBase<T>

Implements: IReferringSegmentation<T>

Known derived types: LISA<T>, OMGLLaVA<T>, PixelLM<T>, VideoLISA<T>

For Beginners

Referring segmentation goes beyond open-vocabulary by understanding complex descriptions like “the person standing behind the counter” or “the animal that could be dangerous”. These models typically combine a large language model (LLM) with a segmentation backbone.

Models extending this base class: LISA, VideoLISA, OMG-LLaVA, PixelLM.

Properties

Property Summary
ITextEncoder<T> TextEncoder { get; } Gets the text encoder whose embedding of the expression the pixel embeddings are compared with.
bool SupportsConversation { get; }
bool SupportsVideoInput { get; }
int MaxTextLength { get; }

Methods

Method Summary
List<ReferringSegmentationResult<T>> SegmentVideoFromExpression(Tensor<T> frames, string expression)
ReferringSegmentationResult<T> SegmentFromConversation(Tensor<T> image, IReadOnlyList<ValueTuple<string, string>> conversationHistory, string currentQuery)
ReferringSegmentationResult<T> SegmentFromExpression(Tensor<T> image, string expression)
Tensor<T> ForwardForTraining(Tensor<T> input) The training forward: the base layer walk (seed wiring, checkpointing and weight streaming included) followed by ClassifyPixels, so training fits the same per-name logits that Predict returns rather than the pixel embeddings the layers en…
Tensor<T> ForwardGpu(Tensor<T> input) The GPU-resident forward: the base layer walk followed by ClassifyPixels, the same output as Forward and Predict instead of the decoder’s pixel embeddings.