FlamingoNeuralNetwork<T>

Models & Types · AiDotNet.NeuralNetworks

Flamingo neural network for in-context visual learning and few-shot tasks.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → FlamingoNeuralNetwork<T>

Implements: IFlamingoModel<T>

For Beginners

Flamingo is a visual AI that can answer questions about images with just a few examples. Show it 2-3 examples of image-answer pairs, and it can handle similar questions about new images. It works by feeding image features into a language model through special cross-attention layers, letting the language model “see” the image while generating text responses.

How It Works

Flamingo is a visual language model that excels at few-shot learning. It uses a Perceiver Resampler to compress visual features and gated cross-attention layers to integrate visual information into a frozen language model.

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 224, 224 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new FlamingoNeuralNetwork<float>(architecture))
    .Build(trainX, trainY);
var output = result.Predict(image);

Constructors

Constructor Summary
FlamingoNeuralNetwork(NeuralNetworkArchitecture<T> architecture, FlamingoOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Initializes a new instance using native layers.
FlamingoNeuralNetwork(NeuralNetworkArchitecture<T> architecture, string visionEncoderPath, string languageModelPath, ITokenizer tokenizer, FlamingoOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) ONNX construction is not supported: a vision encoder and a language model alone cannot run Flamingo.

Properties

Property Summary
LanguageModelBackbone LanguageModelBackbone { get; }
int EmbeddingDimension { get; }
int ImageSize { get; }
int MaxImagesInContext { get; }
int MaxSequenceLength { get; }
int NumPerceiverTokens { get; }

Methods

Method Summary
Dictionary<string, T> InContextClassify(IEnumerable<ValueTuple<Tensor<T>, string>> labeledExamples, Tensor<T> queryImage)
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, IEnumerable<string> classLabels)
Dictionary<string, T> ZeroShotClassify(double[] imageData, IEnumerable<string> labels)
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
IEnumerable<ValueTuple<int, T>> FewShotImageRetrieval(IEnumerable<Tensor<T>> queryExamples, string queryDescription, IEnumerable<Tensor<T>> candidateImages, int topK)
IEnumerable<ValueTuple<int, T>> RetrieveImages(string query, IEnumerable<Vector<T>> imageEmbeddings, int topK)
IEnumerable<ValueTuple<int, T>> RetrieveTexts(Tensor<T> image, IEnumerable<string> texts, int topK)
IEnumerable<Vector<T>> GetImageEmbeddings(IEnumerable<Tensor<T>> images)
IEnumerable<Vector<T>> GetTextEmbeddings(IEnumerable<string> texts)
Matrix<T> EncodeImageBatch(IEnumerable<double[]> imageDataBatch)
Matrix<T> EncodeTextBatch(IEnumerable<string> texts)
ModelMetadata<T> GetModelMetadata()
T ComputeImageTextSimilarity(Tensor<T> image, string text)
T ComputeSimilarity(Vector<T> embedding1, Vector<T> embedding2)
T ScoreImageText(Tensor<T> image, string text)
Tensor<T> ExtractPerceiverFeatures(Tensor<T> image)
Tensor<T> ForwardForTraining(Tensor<T> input)
Vector<T> EncodeImage(double[] imageData)
Vector<T> EncodeText(string text)
Vector<T> GetImageEmbedding(Tensor<T> image)
Vector<T> GetTextEmbedding(string text)
string AnswerQuestion(Tensor<T> image, string question, int maxLength)
string DescribeVideo(IEnumerable<Tensor<T>> frames, string prompt, int maxLength)
string FewShotGenerate(IEnumerable<ValueTuple<Tensor<T>, string>> examples, Tensor<T> queryImage, string queryPrompt, int maxLength)
string FewShotVQA(IEnumerable<ValueTuple<Tensor<T>, string, string>> examples, Tensor<T> queryImage, string question)
string GenerateCaption(Tensor<T> image, int maxLength)
string GenerateWithMultipleImages(IEnumerable<Tensor<T>> images, string prompt, int maxLength)