IFlamingoModel<T>
Interfaces · AiDotNet.Interfaces
Defines the contract for Flamingo-style models with in-context visual learning capabilities.
Implements: IMultimodalEmbedding<T>
Known implementations: FlamingoNeuralNetwork<T>
For Beginners
Flamingo learns new visual tasks from examples you show it!
Key innovation - In-context learning:
- Show Flamingo a few example image-text pairs
- It learns the pattern from these examples
- Apply the pattern to new images WITHOUT any training
Architecture:
- Vision Encoder: Extracts image features (Perceiver Resampler)
- Gated Cross-Attention: Injects visual info into language model
- Frozen LLM: Chinchilla-based language model
Example use case:
- Show 3 examples: [image1] “A red apple” [image2] “A blue car” [image3] “A green tree”
- Ask about new image: [image4] “What color?”
- Flamingo learns from examples that you want the color, answers correctly!
Why Flamingo is revolutionary:
- No fine-tuning needed for new tasks
- Adapts to new visual concepts on-the-fly
- Strong performance with minimal examples
How It Works
Flamingo is a visual language model that excels at few-shot learning - it can learn new tasks from just a few examples provided in the context. It uses gated cross-attention layers interleaved with frozen LLM layers to integrate visual information.
Properties
| Property | Summary |
|---|---|
LanguageModelBackbone LanguageModelBackbone { get; } |
Gets the language model backbone used for generation. |
int MaxImagesInContext { get; } |
Gets the maximum number of images that can be processed in a single context. |
int NumPerceiverTokens { get; } |
Gets the number of visual tokens per image after the Perceiver Resampler. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> InContextClassify(IEnumerable<ValueTuple<Tensor<T>, string>> labeledExamples, Tensor<T> queryImage) |
Performs in-context visual classification without explicit labels. |
IEnumerable<ValueTuple<int, T>> FewShotImageRetrieval(IEnumerable<Tensor<T>> queryExamples, string queryDescription, IEnumerable<Tensor<T>> candidateImages, int topK) |
Retrieves the most similar images from a database using few-shot context. |
T ScoreImageText(Tensor<T> image, string text) |
Computes the log probability of a given text completion for an image. |
Tensor<T> ExtractPerceiverFeatures(Tensor<T> image) |
Extracts visual features using the Perceiver Resampler. |
string DescribeVideo(IEnumerable<Tensor<T>> frames, string prompt, int maxLength) |
Generates captions for a video represented as a sequence of frames. |
string FewShotGenerate(IEnumerable<ValueTuple<Tensor<T>, string>> examples, Tensor<T> queryImage, string queryPrompt, int maxLength) |
Performs few-shot visual learning with interleaved image-text examples. |
string FewShotVQA(IEnumerable<ValueTuple<Tensor<T>, string, string>> examples, Tensor<T> queryImage, string question) |
Performs visual question answering with few-shot examples. |
string GenerateWithMultipleImages(IEnumerable<Tensor<T>> images, string prompt, int maxLength) |
Generates text for multiple images interleaved in a single context. |