IBlip2Model<T>
Interfaces · AiDotNet.Interfaces
Defines the contract for BLIP-2 (Bootstrapped Language-Image Pre-training 2) models.
Implements: IMultimodalEmbedding<T>
Known implementations: Blip2NeuralNetwork<T>
For Beginners
BLIP-2 is like having a smart translator between images and language!
Key innovation - the Q-Former:
- Uses special “query tokens” to ask questions about the image
- These queries learn to extract the most useful visual information
- The extracted features then connect to powerful language models (LLMs)
Why BLIP-2 is special:
- Uses frozen (pre-trained) image encoders like ViT-G
- Uses frozen LLMs like OPT or Flan-T5
- Only trains the small Q-Former bridge (much cheaper!)
- Gets state-of-the-art results with less compute
Use cases (same as BLIP but better):
- More accurate image captioning
- Better visual question answering
- More nuanced image-text understanding
- Can leverage larger LLMs for better generation
How It Works
BLIP-2 is a more efficient and powerful successor to BLIP that uses a Q-Former (Querying Transformer) to bridge frozen image encoders with frozen large language models. This architecture enables better vision-language understanding with significantly less training compute.
Properties
| Property | Summary |
|---|---|
LanguageModelBackbone LanguageModelBackbone { get; } |
Gets the type of language model backbone used for generation. |
int NumQueryTokens { get; } |
Gets the number of learnable query tokens used by the Q-Former. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, IEnumerable<string> classLabels, bool useItm) |
Performs zero-shot image classification using text prompts. |
IEnumerable<ValueTuple<int, T>> RetrieveImages(string query, IEnumerable<Tensor<T>> imageFeatures, int topK, bool useItmReranking, int rerankTopN) |
Retrieves the most relevant images for a text query. |
IEnumerable<ValueTuple<string, T>> GenerateCaptions(Tensor<T> image, int numCaptions, string prompt, int maxLength, double temperature, double topP) |
Generates multiple diverse captions for an image. |
T ComputeContrastiveSimilarity(Tensor<T> image, string text) |
Computes image-text contrastive similarity using Q-Former features. |
T ComputeImageTextMatch(Tensor<T> image, string text) |
Computes image-text matching score using the Q-Former’s ITM head. |
Tensor<T> ExtractQFormerFeatures(Tensor<T> image) |
Extracts visual features using the Q-Former’s learnable queries. |
Vector<T> GroundText(Tensor<T> image, string description) |
Performs visual grounding to locate objects described in text. |
string AnswerQuestion(Tensor<T> image, string question, int maxLength) |
Answers a question about an image using the LLM backend. |
string GenerateCaption(Tensor<T> image, string prompt, int maxLength, int numBeams, double temperature) |
Generates a caption for an image using the LLM backend. |
string GenerateWithInstruction(Tensor<T> image, string instruction, int maxLength) |
Generates text conditioned on both image and text context (instructed generation). |