ILLaVAModel<T>
Interfaces · AiDotNet.Interfaces
Defines the contract for LLaVA (Large Language and Vision Assistant) models.
Implements: IMultimodalEmbedding<T>
Known implementations: LLaVANeuralNetwork<T>
For Beginners
LLaVA is like giving eyes to ChatGPT!
Architecture:
- Vision Encoder (CLIP ViT): Converts images to feature vectors
- Projection Layer: Maps visual features to LLM’s text embedding space
- Large Language Model (LLaMA/Vicuna): Generates responses
Key capabilities:
- Visual conversations: “What’s in this image?” followed by “What color is the car?”
- Visual reasoning: Understanding relationships, counting, spatial awareness
- Instruction following: “Describe this image as if you were a poet”
- Multi-turn dialogue: Context-aware conversations about images
Why LLaVA is popular:
- Simple but effective architecture
- Open-source and reproducible
- Strong performance on visual understanding benchmarks
- Efficient training with visual instruction tuning
How It Works
LLaVA connects a vision encoder (like CLIP ViT) with a large language model (like LLaMA/Vicuna) through a simple projection layer, enabling visual instruction-following and conversational AI about images.
Properties
| Property | Summary |
|---|---|
LanguageModelBackbone LanguageModelBackbone { get; } |
Gets the language model backbone used for generation. |
int NumVisualTokens { get; } |
Gets the maximum number of visual tokens used per image. |
string VisionEncoderType { get; } |
Gets the vision encoder type. |
Methods
| Method | Summary |
|---|---|
IEnumerable<ValueTuple<string, T>> GenerateMultiple(Tensor<T> image, string prompt, int numResponses, double temperature) |
Generates multiple diverse responses for the same prompt. |
IEnumerable<string> DescribeRegions(Tensor<T> image, IEnumerable<Vector<T>> regions) |
Generates a detailed description of specific regions in an image. |
Tensor<T> ExtractVisualFeatures(Tensor<T> image) |
Extracts visual features before projection to LLM space. |
Tensor<T> ProjectToLanguageSpace(Tensor<T> visualFeatures) |
Projects visual features to the LLM’s embedding space. |
Vector<T> GroundObject(Tensor<T> image, string description) |
Performs visual grounding to locate objects described by text. |
string Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage, int maxLength, double temperature) |
Continues a multi-turn conversation about an image. |
string CompareImages(Tensor<T> image1, Tensor<T> image2, IEnumerable<string> aspectsToCompare) |
Compares two images and describes their differences. |
string Generate(Tensor<T> image, string prompt, int maxLength, double temperature, double topP) |
Generates a response to a text prompt about an image. |