IBlipModel<T>
Interfaces · AiDotNet.Interfaces
Defines the contract for BLIP (Bootstrapped Language-Image Pre-training) models.
Implements: IMultimodalEmbedding<T>
Known implementations: BlipNeuralNetwork<T>
For Beginners
BLIP is like CLIP but with extra superpowers!
What CLIP can do:
- Compare images and text (are they related?)
- Zero-shot classification (classify without training)
What BLIP adds:
- Generate captions for images (describe what you see)
- Answer questions about images (VQA)
- Better image-text matching with cross-attention
BLIP was trained on a larger, cleaner dataset using a special “bootstrapping” technique that improves the quality of training data automatically.
How It Works
BLIP extends CLIP’s capabilities with additional vision-language tasks:
image captioning, image-text matching, and visual question answering.
This interface extends IMultimodalEmbedding with these features.
Methods
| Method | Summary |
|---|---|
IEnumerable<ValueTuple<int, T>> RetrieveImages(string query, IEnumerable<Vector<T>> imageEmbeddings, int topK) |
Retrieves the most relevant images for a text query from a collection. |
IEnumerable<ValueTuple<int, T>> RetrieveTexts(Tensor<T> image, IEnumerable<Vector<T>> textEmbeddings, int topK) |
Retrieves the most relevant texts for an image from a collection. |
IEnumerable<ValueTuple<string, T>> RankCaptions(Tensor<T> image, IEnumerable<string> candidates) |
Ranks a set of candidate captions by how well they match an image. |
IEnumerable<string> GenerateCaptions(Tensor<T> image, int numCaptions, int maxLength) |
Generates multiple candidate captions for an image. |
T ComputeImageTextMatch(Tensor<T> image, string text) |
Determines whether a given text accurately describes an image. |
string AnswerQuestion(Tensor<T> image, string question, int maxLength) |
Answers a question about an image’s content. |
string GenerateCaption(Tensor<T> image, int maxLength, int numBeams) |
Generates a caption describing the content of an image. |