IBlip2Model<T>

Interfaces · AiDotNet.Interfaces

Defines the contract for BLIP-2 (Bootstrapped Language-Image Pre-training 2) models.

Implements: IMultimodalEmbedding<T>

Known implementations: Blip2NeuralNetwork<T>

For Beginners

BLIP-2 is like having a smart translator between images and language!

Key innovation - the Q-Former:

  • Uses special “query tokens” to ask questions about the image
  • These queries learn to extract the most useful visual information
  • The extracted features then connect to powerful language models (LLMs)

Why BLIP-2 is special:

  • Uses frozen (pre-trained) image encoders like ViT-G
  • Uses frozen LLMs like OPT or Flan-T5
  • Only trains the small Q-Former bridge (much cheaper!)
  • Gets state-of-the-art results with less compute

Use cases (same as BLIP but better):

  • More accurate image captioning
  • Better visual question answering
  • More nuanced image-text understanding
  • Can leverage larger LLMs for better generation

How It Works

BLIP-2 is a more efficient and powerful successor to BLIP that uses a Q-Former (Querying Transformer) to bridge frozen image encoders with frozen large language models. This architecture enables better vision-language understanding with significantly less training compute.

Properties

Property Summary
LanguageModelBackbone LanguageModelBackbone { get; } Gets the type of language model backbone used for generation.
int NumQueryTokens { get; } Gets the number of learnable query tokens used by the Q-Former.

Methods

Method Summary
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, IEnumerable<string> classLabels, bool useItm) Performs zero-shot image classification using text prompts.
IEnumerable<ValueTuple<int, T>> RetrieveImages(string query, IEnumerable<Tensor<T>> imageFeatures, int topK, bool useItmReranking, int rerankTopN) Retrieves the most relevant images for a text query.
IEnumerable<ValueTuple<string, T>> GenerateCaptions(Tensor<T> image, int numCaptions, string prompt, int maxLength, double temperature, double topP) Generates multiple diverse captions for an image.
T ComputeContrastiveSimilarity(Tensor<T> image, string text) Computes image-text contrastive similarity using Q-Former features.
T ComputeImageTextMatch(Tensor<T> image, string text) Computes image-text matching score using the Q-Former’s ITM head.
Tensor<T> ExtractQFormerFeatures(Tensor<T> image) Extracts visual features using the Q-Former’s learnable queries.
Vector<T> GroundText(Tensor<T> image, string description) Performs visual grounding to locate objects described in text.
string AnswerQuestion(Tensor<T> image, string question, int maxLength) Answers a question about an image using the LLM backend.
string GenerateCaption(Tensor<T> image, string prompt, int maxLength, int numBeams, double temperature) Generates a caption for an image using the LLM backend.
string GenerateWithInstruction(Tensor<T> image, string instruction, int maxLength) Generates text conditioned on both image and text context (instructed generation).