IFlamingoModel<T>

Interfaces · AiDotNet.Interfaces

Defines the contract for Flamingo-style models with in-context visual learning capabilities.

Implements: IMultimodalEmbedding<T>

Known implementations: FlamingoNeuralNetwork<T>

For Beginners

Flamingo learns new visual tasks from examples you show it!

Key innovation - In-context learning:

  • Show Flamingo a few example image-text pairs
  • It learns the pattern from these examples
  • Apply the pattern to new images WITHOUT any training

Architecture:

  1. Vision Encoder: Extracts image features (Perceiver Resampler)
  2. Gated Cross-Attention: Injects visual info into language model
  3. Frozen LLM: Chinchilla-based language model

Example use case:

  • Show 3 examples: [image1] “A red apple” [image2] “A blue car” [image3] “A green tree”
  • Ask about new image: [image4] “What color?”
  • Flamingo learns from examples that you want the color, answers correctly!

Why Flamingo is revolutionary:

  • No fine-tuning needed for new tasks
  • Adapts to new visual concepts on-the-fly
  • Strong performance with minimal examples

How It Works

Flamingo is a visual language model that excels at few-shot learning - it can learn new tasks from just a few examples provided in the context. It uses gated cross-attention layers interleaved with frozen LLM layers to integrate visual information.

Properties

Property Summary
LanguageModelBackbone LanguageModelBackbone { get; } Gets the language model backbone used for generation.
int MaxImagesInContext { get; } Gets the maximum number of images that can be processed in a single context.
int NumPerceiverTokens { get; } Gets the number of visual tokens per image after the Perceiver Resampler.

Methods

Method Summary
Dictionary<string, T> InContextClassify(IEnumerable<ValueTuple<Tensor<T>, string>> labeledExamples, Tensor<T> queryImage) Performs in-context visual classification without explicit labels.
IEnumerable<ValueTuple<int, T>> FewShotImageRetrieval(IEnumerable<Tensor<T>> queryExamples, string queryDescription, IEnumerable<Tensor<T>> candidateImages, int topK) Retrieves the most similar images from a database using few-shot context.
T ScoreImageText(Tensor<T> image, string text) Computes the log probability of a given text completion for an image.
Tensor<T> ExtractPerceiverFeatures(Tensor<T> image) Extracts visual features using the Perceiver Resampler.
string DescribeVideo(IEnumerable<Tensor<T>> frames, string prompt, int maxLength) Generates captions for a video represented as a sequence of frames.
string FewShotGenerate(IEnumerable<ValueTuple<Tensor<T>, string>> examples, Tensor<T> queryImage, string queryPrompt, int maxLength) Performs few-shot visual learning with interleaved image-text examples.
string FewShotVQA(IEnumerable<ValueTuple<Tensor<T>, string, string>> examples, Tensor<T> queryImage, string question) Performs visual question answering with few-shot examples.
string GenerateWithMultipleImages(IEnumerable<Tensor<T>> images, string prompt, int maxLength) Generates text for multiple images interleaved in a single context.