ClipNeuralNetwork<T>
Models & Types · AiDotNet.NeuralNetworks
CLIP (Contrastive Language-Image Pre-training) neural network that encodes both text and images into a shared embedding space, enabling cross-modal similarity and zero-shot classification.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → ClipNeuralNetwork<T>
Implements: IMultimodalEmbedding<T>
For Beginners
CLIP learns to connect images and text by training on millions of image-caption pairs from the internet. It creates a shared space where similar images and text descriptions are close together. This enables powerful capabilities like zero-shot image classification (categorizing images without specific training), image search using text queries, and measuring how well an image matches a description.
Example
var options = new ClipOptions { };
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 224, 224 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new ClipNeuralNetwork<float>(
architecture,
imageEncoderPath: "image_encoder.onnx", textEncoderPath: "text_encoder.onnx",
tokenizer: CharacterTokenizer.CreateAscii()))
.Build(trainX, trainY);
var embedding = result.Predict(image);
Constructors
| Constructor | Summary |
|---|---|
ClipNeuralNetwork(NeuralNetworkArchitecture<T> architecture, string imageEncoderPath, string textEncoderPath, ITokenizer tokenizer, ClipOptions options, ILossFunction<T> lossFunction) |
Initializes a new instance of the CLIP neural network. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Gets whether this network supports training. |
int EmbeddingDimension { get; } |
Gets the embedding dimension of the CLIP model. |
int ImageSize { get; } |
Gets the expected image size (square images: ImageSize x ImageSize pixels). |
int MaxSequenceLength { get; } |
Gets the maximum sequence length for text input. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ZeroShotClassify(double[] imageData, IEnumerable<string> labels) |
|
Matrix<T> EncodeImageBatch(IEnumerable<double[]> imageDataBatch) |
|
Matrix<T> EncodeTextBatch(IEnumerable<string> texts) |
|
ModelMetadata<T> GetModelMetadata() |
|
T ComputeSimilarity(Vector<T> embedding1, Vector<T> embedding2) |
|
Task<Matrix<T>> EmbedBatchAsync(IEnumerable<string> texts) |
|
Task<Vector<T>> EmbedAsync(string text) |
|
Tensor<T> Predict(Tensor<T> input) |
|
Vector<T> EncodeImage(double[] imageData) |
|
Vector<T> EncodeText(string text) |