BlipNeuralNetwork<T>
Models & Types · AiDotNet.NeuralNetworks
BLIP (Bootstrapped Language-Image Pre-training) neural network for vision-language tasks.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → BlipNeuralNetwork<T>
Implements: IBlipModel<T>
For Beginners
BLIP is a more powerful version of CLIP!
CLIP can:
- Match images with text descriptions
- Zero-shot classification
BLIP adds:
- Generate captions (“a dog playing in the park”)
- Answer questions (“What color is the car?” -> “Red”)
- More accurate image-text matching
Training innovation:
- BLIP was trained on noisy web data
- It learned to filter out bad captions automatically
- Then it generated better captions to train on!
- This “bootstrapping” creates a cleaner dataset
Use cases:
- Accessibility (auto-generate alt-text for images)
- Content moderation (answer “is there violence in this image?”)
- Visual search (find images matching a description)
- Image organization (auto-tag photos)
How It Works
BLIP extends CLIP’s capabilities with image captioning, image-text matching, and visual question answering. It uses a unified framework with both understanding and generation tasks. This implementation supports both ONNX pretrained models and native library layers.
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 384, 384 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new BlipNeuralNetwork<float>(architecture))
.Build(trainX, trainY);
var output = result.Predict(image);
Constructors
| Constructor | Summary |
|---|---|
BlipNeuralNetwork(NeuralNetworkArchitecture<T> architecture, BlipOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a BLIP network using native library layers. |
BlipNeuralNetwork(NeuralNetworkArchitecture<T> architecture, string visionEncoderPath, string textEncoderPath, string textDecoderPath, ITokenizer tokenizer, BlipOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a BLIP network using pretrained ONNX models. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int EmbeddingDimension { get; } |
|
int ImageSize { get; } |
|
int MaxSequenceLength { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, IEnumerable<string> classLabels) |
|
Dictionary<string, T> ZeroShotClassify(double[] imageData, IEnumerable<string> labels) |
|
IEnumerable<ValueTuple<int, T>> RetrieveImages(string query, IEnumerable<Vector<T>> imageEmbeddings, int topK) |
|
IEnumerable<ValueTuple<int, T>> RetrieveTexts(Tensor<T> image, IEnumerable<Vector<T>> textEmbeddings, int topK) |
|
IEnumerable<ValueTuple<string, T>> RankCaptions(Tensor<T> image, IEnumerable<string> candidates) |
|
IEnumerable<Vector<T>> GetImageEmbeddings(IEnumerable<Tensor<T>> images) |
|
IEnumerable<Vector<T>> GetTextEmbeddings(IEnumerable<string> texts) |
|
IEnumerable<string> GenerateCaptions(Tensor<T> image, int numCaptions, int maxLength) |
|
Matrix<T> EncodeImageBatch(IEnumerable<double[]> imageDataBatch) |
|
Matrix<T> EncodeTextBatch(IEnumerable<string> texts) |
|
ModelMetadata<T> GetModelMetadata() |
Retrieves metadata about the BLIP neural network model. |
T ComputeImageTextMatch(Tensor<T> image, string text) |
|
T ComputeSimilarity(Vector<T> textEmbedding, Vector<T> imageEmbedding) |
|
Task<Matrix<T>> EmbedBatchAsync(IEnumerable<string> texts) |
|
Task<Vector<T>> EmbedAsync(string text) |
|
Vector<T> EncodeImage(double[] imageData) |
|
Vector<T> EncodeText(string text) |
|
Vector<T> GetImageEmbedding(Tensor<T> image) |
|
Vector<T> GetTextEmbedding(string text) |
|
string AnswerQuestion(Tensor<T> image, string question, int maxLength) |
|
string GenerateCaption(Tensor<T> image, int maxLength, int numBeams) |