IReferringSegmentation<T>

Interfaces · AiDotNet.Interfaces

Interface for referring segmentation models that segment objects based on natural language descriptions, including complex reasoning about spatial relationships and attributes.

Implements: ISegmentationModel<T>

Known implementations: ReferringSegmentationBase<T>

For Beginners

Referring segmentation lets you describe exactly what you want using natural language — even complex descriptions.

Examples of what you can ask:

  • “The person standing behind the counter” (spatial reasoning)
  • “The animal that could be dangerous” (world knowledge / reasoning)
  • “The object that doesn’t belong in this kitchen” (contextual reasoning)
  • “Track the person in the red shirt throughout the video” (video + language)

This is more powerful than open-vocabulary segmentation because it understands context and relationships, not just category names.

Models implementing this interface:

  • LISA (CVPR 2024 Oral, LLaVA + SAM reasoning segmentation)
  • VideoLISA (NeurIPS 2024, video reasoning segmentation)
  • OMG-LLaVA (NeurIPS 2024, pixel-level reasoning)
  • PixelLM (CVPR 2024, segmentation codebook)

How It Works

Referring segmentation goes beyond open-vocabulary segmentation by supporting complex natural language queries that require reasoning about object attributes, spatial relationships, and even world knowledge. These models typically use large language models (LLMs) to understand complex queries.

Properties

Property Summary
bool SupportsConversation { get; } Gets whether this model supports multi-turn conversation.
bool SupportsVideoInput { get; } Gets whether this model supports video input.
int MaxTextLength { get; } Gets the maximum input text length in tokens.

Methods

Method Summary
List<ReferringSegmentationResult<T>> SegmentVideoFromExpression(Tensor<T> frames, string expression) Segments objects in a video based on a natural language expression.
ReferringSegmentationResult<T> SegmentFromConversation(Tensor<T> image, IReadOnlyList<ValueTuple<string, string>> conversationHistory, string currentQuery) Segments objects from a multi-turn conversation context.
ReferringSegmentationResult<T> SegmentFromExpression(Tensor<T> image, string expression) Segments objects described by a natural language expression.