BlipNeuralNetwork<T>

Models & Types · AiDotNet.NeuralNetworks

BLIP (Bootstrapped Language-Image Pre-training) neural network for vision-language tasks.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → BlipNeuralNetwork<T>

Implements: IBlipModel<T>

For Beginners

BLIP is a more powerful version of CLIP!

CLIP can:

  • Match images with text descriptions
  • Zero-shot classification

BLIP adds:

  • Generate captions (“a dog playing in the park”)
  • Answer questions (“What color is the car?” -> “Red”)
  • More accurate image-text matching

Training innovation:

  • BLIP was trained on noisy web data
  • It learned to filter out bad captions automatically
  • Then it generated better captions to train on!
  • This “bootstrapping” creates a cleaner dataset

Use cases:

  • Accessibility (auto-generate alt-text for images)
  • Content moderation (answer “is there violence in this image?”)
  • Visual search (find images matching a description)
  • Image organization (auto-tag photos)

How It Works

BLIP extends CLIP’s capabilities with image captioning, image-text matching, and visual question answering. It uses a unified framework with both understanding and generation tasks. This implementation supports both ONNX pretrained models and native library layers.

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 384, 384 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new BlipNeuralNetwork<float>(architecture))
    .Build(trainX, trainY);
var output = result.Predict(image);

Constructors

Constructor Summary
BlipNeuralNetwork(NeuralNetworkArchitecture<T> architecture, BlipOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a BLIP network using native library layers.
BlipNeuralNetwork(NeuralNetworkArchitecture<T> architecture, string visionEncoderPath, string textEncoderPath, string textDecoderPath, ITokenizer tokenizer, BlipOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a BLIP network using pretrained ONNX models.

Properties

Property Summary
bool SupportsTraining { get; }
int EmbeddingDimension { get; }
int ImageSize { get; }
int MaxSequenceLength { get; }

Methods

Method Summary
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, IEnumerable<string> classLabels)
Dictionary<string, T> ZeroShotClassify(double[] imageData, IEnumerable<string> labels)
IEnumerable<ValueTuple<int, T>> RetrieveImages(string query, IEnumerable<Vector<T>> imageEmbeddings, int topK)
IEnumerable<ValueTuple<int, T>> RetrieveTexts(Tensor<T> image, IEnumerable<Vector<T>> textEmbeddings, int topK)
IEnumerable<ValueTuple<string, T>> RankCaptions(Tensor<T> image, IEnumerable<string> candidates)
IEnumerable<Vector<T>> GetImageEmbeddings(IEnumerable<Tensor<T>> images)
IEnumerable<Vector<T>> GetTextEmbeddings(IEnumerable<string> texts)
IEnumerable<string> GenerateCaptions(Tensor<T> image, int numCaptions, int maxLength)
Matrix<T> EncodeImageBatch(IEnumerable<double[]> imageDataBatch)
Matrix<T> EncodeTextBatch(IEnumerable<string> texts)
ModelMetadata<T> GetModelMetadata() Retrieves metadata about the BLIP neural network model.
T ComputeImageTextMatch(Tensor<T> image, string text)
T ComputeSimilarity(Vector<T> textEmbedding, Vector<T> imageEmbedding)
Task<Matrix<T>> EmbedBatchAsync(IEnumerable<string> texts)
Task<Vector<T>> EmbedAsync(string text)
Vector<T> EncodeImage(double[] imageData)
Vector<T> EncodeText(string text)
Vector<T> GetImageEmbedding(Tensor<T> image)
Vector<T> GetTextEmbedding(string text)
string AnswerQuestion(Tensor<T> image, string question, int maxLength)
string GenerateCaption(Tensor<T> image, int maxLength, int numBeams)