ILLaVAModel<T>

Interfaces · AiDotNet.Interfaces

Defines the contract for LLaVA (Large Language and Vision Assistant) models.

Implements: IMultimodalEmbedding<T>

Known implementations: LLaVANeuralNetwork<T>

For Beginners

LLaVA is like giving eyes to ChatGPT!

Architecture:

  1. Vision Encoder (CLIP ViT): Converts images to feature vectors
  2. Projection Layer: Maps visual features to LLM’s text embedding space
  3. Large Language Model (LLaMA/Vicuna): Generates responses

Key capabilities:

  • Visual conversations: “What’s in this image?” followed by “What color is the car?”
  • Visual reasoning: Understanding relationships, counting, spatial awareness
  • Instruction following: “Describe this image as if you were a poet”
  • Multi-turn dialogue: Context-aware conversations about images

Why LLaVA is popular:

  • Simple but effective architecture
  • Open-source and reproducible
  • Strong performance on visual understanding benchmarks
  • Efficient training with visual instruction tuning

How It Works

LLaVA connects a vision encoder (like CLIP ViT) with a large language model (like LLaMA/Vicuna) through a simple projection layer, enabling visual instruction-following and conversational AI about images.

Properties

Property Summary
LanguageModelBackbone LanguageModelBackbone { get; } Gets the language model backbone used for generation.
int NumVisualTokens { get; } Gets the maximum number of visual tokens used per image.
string VisionEncoderType { get; } Gets the vision encoder type.

Methods

Method Summary
IEnumerable<ValueTuple<string, T>> GenerateMultiple(Tensor<T> image, string prompt, int numResponses, double temperature) Generates multiple diverse responses for the same prompt.
IEnumerable<string> DescribeRegions(Tensor<T> image, IEnumerable<Vector<T>> regions) Generates a detailed description of specific regions in an image.
Tensor<T> ExtractVisualFeatures(Tensor<T> image) Extracts visual features before projection to LLM space.
Tensor<T> ProjectToLanguageSpace(Tensor<T> visualFeatures) Projects visual features to the LLM’s embedding space.
Vector<T> GroundObject(Tensor<T> image, string description) Performs visual grounding to locate objects described by text.
string Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage, int maxLength, double temperature) Continues a multi-turn conversation about an image.
string CompareImages(Tensor<T> image1, Tensor<T> image2, IEnumerable<string> aspectsToCompare) Compares two images and describes their differences.
string Generate(Tensor<T> image, string prompt, int maxLength, double temperature, double topP) Generates a response to a text prompt about an image.