ReferringSegmentationBase<T>
Base Classes · AiDotNet.ComputerVision.Segmentation.Common
Abstract base class for referring segmentation models that segment objects from natural language descriptions with complex reasoning about spatial relationships and attributes.
Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → ReferringSegmentationBase<T>
Implements: IReferringSegmentation<T>
Known derived types: LISA<T>, OMGLLaVA<T>, PixelLM<T>, VideoLISA<T>
For Beginners
Referring segmentation goes beyond open-vocabulary by understanding complex descriptions like “the person standing behind the counter” or “the animal that could be dangerous”. These models typically combine a large language model (LLM) with a segmentation backbone.
Models extending this base class: LISA, VideoLISA, OMG-LLaVA, PixelLM.
Properties
| Property | Summary |
|---|---|
ITextEncoder<T> TextEncoder { get; } |
Gets the text encoder whose embedding of the expression the pixel embeddings are compared with. |
bool SupportsConversation { get; } |
|
bool SupportsVideoInput { get; } |
|
int MaxTextLength { get; } |
Methods
| Method | Summary |
|---|---|
List<ReferringSegmentationResult<T>> SegmentVideoFromExpression(Tensor<T> frames, string expression) |
|
ReferringSegmentationResult<T> SegmentFromConversation(Tensor<T> image, IReadOnlyList<ValueTuple<string, string>> conversationHistory, string currentQuery) |
|
ReferringSegmentationResult<T> SegmentFromExpression(Tensor<T> image, string expression) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
The training forward: the base layer walk (seed wiring, checkpointing and weight streaming included) followed by ClassifyPixels, so training fits the same per-name logits that Predict returns rather than the pixel embeddings the layers en… |
Tensor<T> ForwardGpu(Tensor<T> input) |
The GPU-resident forward: the base layer walk followed by ClassifyPixels, the same output as Forward and Predict instead of the decoder’s pixel embeddings. |