ClipNeuralNetwork<T>

Models & Types · AiDotNet.NeuralNetworks

CLIP (Contrastive Language-Image Pre-training) neural network that encodes both text and images into a shared embedding space, enabling cross-modal similarity and zero-shot classification.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → ClipNeuralNetwork<T>

Implements: IMultimodalEmbedding<T>

For Beginners

CLIP learns to connect images and text by training on millions of image-caption pairs from the internet. It creates a shared space where similar images and text descriptions are close together. This enables powerful capabilities like zero-shot image classification (categorizing images without specific training), image search using text queries, and measuring how well an image matches a description.

Example

var options = new ClipOptions { };
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 224, 224 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new ClipNeuralNetwork<float>(
        architecture,
        imageEncoderPath: "image_encoder.onnx", textEncoderPath: "text_encoder.onnx",
        tokenizer: CharacterTokenizer.CreateAscii()))
    .Build(trainX, trainY);
var embedding = result.Predict(image);

Constructors

Constructor Summary
ClipNeuralNetwork(NeuralNetworkArchitecture<T> architecture, string imageEncoderPath, string textEncoderPath, ITokenizer tokenizer, ClipOptions options, ILossFunction<T> lossFunction) Initializes a new instance of the CLIP neural network.

Properties

Property Summary
bool SupportsTraining { get; } Gets whether this network supports training.
int EmbeddingDimension { get; } Gets the embedding dimension of the CLIP model.
int ImageSize { get; } Gets the expected image size (square images: ImageSize x ImageSize pixels).
int MaxSequenceLength { get; } Gets the maximum sequence length for text input.

Methods

Method Summary
Dictionary<string, T> ZeroShotClassify(double[] imageData, IEnumerable<string> labels)
Matrix<T> EncodeImageBatch(IEnumerable<double[]> imageDataBatch)
Matrix<T> EncodeTextBatch(IEnumerable<string> texts)
ModelMetadata<T> GetModelMetadata()
T ComputeSimilarity(Vector<T> embedding1, Vector<T> embedding2)
Task<Matrix<T>> EmbedBatchAsync(IEnumerable<string> texts)
Task<Vector<T>> EmbedAsync(string text)
Tensor<T> Predict(Tensor<T> input)
Vector<T> EncodeImage(double[] imageData)
Vector<T> EncodeText(string text)