IReferringSegmentation<T>
Interfaces · AiDotNet.Interfaces
Interface for referring segmentation models that segment objects based on natural language descriptions, including complex reasoning about spatial relationships and attributes.
Implements: ISegmentationModel<T>
Known implementations: ReferringSegmentationBase<T>
For Beginners
Referring segmentation lets you describe exactly what you want using natural language — even complex descriptions.
Examples of what you can ask:
- “The person standing behind the counter” (spatial reasoning)
- “The animal that could be dangerous” (world knowledge / reasoning)
- “The object that doesn’t belong in this kitchen” (contextual reasoning)
- “Track the person in the red shirt throughout the video” (video + language)
This is more powerful than open-vocabulary segmentation because it understands context and relationships, not just category names.
Models implementing this interface:
- LISA (CVPR 2024 Oral, LLaVA + SAM reasoning segmentation)
- VideoLISA (NeurIPS 2024, video reasoning segmentation)
- OMG-LLaVA (NeurIPS 2024, pixel-level reasoning)
- PixelLM (CVPR 2024, segmentation codebook)
How It Works
Referring segmentation goes beyond open-vocabulary segmentation by supporting complex natural language queries that require reasoning about object attributes, spatial relationships, and even world knowledge. These models typically use large language models (LLMs) to understand complex queries.
Properties
| Property | Summary |
|---|---|
bool SupportsConversation { get; } |
Gets whether this model supports multi-turn conversation. |
bool SupportsVideoInput { get; } |
Gets whether this model supports video input. |
int MaxTextLength { get; } |
Gets the maximum input text length in tokens. |
Methods
| Method | Summary |
|---|---|
List<ReferringSegmentationResult<T>> SegmentVideoFromExpression(Tensor<T> frames, string expression) |
Segments objects in a video based on a natural language expression. |
ReferringSegmentationResult<T> SegmentFromConversation(Tensor<T> image, IReadOnlyList<ValueTuple<string, string>> conversationHistory, string currentQuery) |
Segments objects from a multi-turn conversation context. |
ReferringSegmentationResult<T> SegmentFromExpression(Tensor<T> image, string expression) |
Segments objects described by a natural language expression. |