FlamingoNeuralNetwork<T>
Models & Types · AiDotNet.NeuralNetworks
Flamingo neural network for in-context visual learning and few-shot tasks.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → FlamingoNeuralNetwork<T>
Implements: IFlamingoModel<T>
For Beginners
Flamingo is a visual AI that can answer questions about images with just a few examples. Show it 2-3 examples of image-answer pairs, and it can handle similar questions about new images. It works by feeding image features into a language model through special cross-attention layers, letting the language model “see” the image while generating text responses.
How It Works
Flamingo is a visual language model that excels at few-shot learning. It uses a Perceiver Resampler to compress visual features and gated cross-attention layers to integrate visual information into a frozen language model.
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 224, 224 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new FlamingoNeuralNetwork<float>(architecture))
.Build(trainX, trainY);
var output = result.Predict(image);
Constructors
| Constructor | Summary |
|---|---|
FlamingoNeuralNetwork(NeuralNetworkArchitecture<T> architecture, FlamingoOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Initializes a new instance using native layers. |
FlamingoNeuralNetwork(NeuralNetworkArchitecture<T> architecture, string visionEncoderPath, string languageModelPath, ITokenizer tokenizer, FlamingoOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
ONNX construction is not supported: a vision encoder and a language model alone cannot run Flamingo. |
Properties
| Property | Summary |
|---|---|
LanguageModelBackbone LanguageModelBackbone { get; } |
|
int EmbeddingDimension { get; } |
|
int ImageSize { get; } |
|
int MaxImagesInContext { get; } |
|
int MaxSequenceLength { get; } |
|
int NumPerceiverTokens { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> InContextClassify(IEnumerable<ValueTuple<Tensor<T>, string>> labeledExamples, Tensor<T> queryImage) |
|
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, IEnumerable<string> classLabels) |
|
Dictionary<string, T> ZeroShotClassify(double[] imageData, IEnumerable<string> labels) |
|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
IEnumerable<ValueTuple<int, T>> FewShotImageRetrieval(IEnumerable<Tensor<T>> queryExamples, string queryDescription, IEnumerable<Tensor<T>> candidateImages, int topK) |
|
IEnumerable<ValueTuple<int, T>> RetrieveImages(string query, IEnumerable<Vector<T>> imageEmbeddings, int topK) |
|
IEnumerable<ValueTuple<int, T>> RetrieveTexts(Tensor<T> image, IEnumerable<string> texts, int topK) |
|
IEnumerable<Vector<T>> GetImageEmbeddings(IEnumerable<Tensor<T>> images) |
|
IEnumerable<Vector<T>> GetTextEmbeddings(IEnumerable<string> texts) |
|
Matrix<T> EncodeImageBatch(IEnumerable<double[]> imageDataBatch) |
|
Matrix<T> EncodeTextBatch(IEnumerable<string> texts) |
|
ModelMetadata<T> GetModelMetadata() |
|
T ComputeImageTextSimilarity(Tensor<T> image, string text) |
|
T ComputeSimilarity(Vector<T> embedding1, Vector<T> embedding2) |
|
T ScoreImageText(Tensor<T> image, string text) |
|
Tensor<T> ExtractPerceiverFeatures(Tensor<T> image) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Vector<T> EncodeImage(double[] imageData) |
|
Vector<T> EncodeText(string text) |
|
Vector<T> GetImageEmbedding(Tensor<T> image) |
|
Vector<T> GetTextEmbedding(string text) |
|
string AnswerQuestion(Tensor<T> image, string question, int maxLength) |
|
string DescribeVideo(IEnumerable<Tensor<T>> frames, string prompt, int maxLength) |
|
string FewShotGenerate(IEnumerable<ValueTuple<Tensor<T>, string>> examples, Tensor<T> queryImage, string queryPrompt, int maxLength) |
|
string FewShotVQA(IEnumerable<ValueTuple<Tensor<T>, string, string>> examples, Tensor<T> queryImage, string question) |
|
string GenerateCaption(Tensor<T> image, int maxLength) |
|
string GenerateWithMultipleImages(IEnumerable<Tensor<T>> images, string prompt, int maxLength) |