OpenCLIP<T>

Models & Types · AiDotNet.VisionLanguage.Encoders

OpenCLIP (Open Contrastive Language-Image Pre-training) model for zero-shot classification and cross-modal retrieval using open-source training data.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → OpenCLIP<T>

Implements: IContrastiveVisionLanguageModel<T>

For Beginners

OpenCLIP works exactly like CLIP - it understands both images and text. You give it an image and some text labels, and it tells you which label best matches the image, without any additional training. The “Open” in OpenCLIP means it was trained on publicly available data, so anyone can reproduce and verify the results.

Usage:

How It Works

OpenCLIP (Ilharco et al., 2021) is an open-source reproduction and extension of OpenAI’s CLIP, trained on publicly available datasets like LAION-2B (2 billion image-text pairs) and LAION-5B (5 billion pairs). It reproduces CLIP’s dual-encoder architecture with a Vision Transformer (ViT) for images and a text transformer for text, both projecting into a shared embedding space.

Architecture:

  • Vision encoder: ViT-B/32 through ViT-bigG/14, processing images as patch sequences
  • Text encoder: GPT-2-style transformer with causal attention mask
  • Projection heads: Linear projections mapping both modalities to shared 512/768/1024-dim space
  • Contrastive loss: InfoNCE with learnable temperature (symmetric cross-entropy)

Key Innovation: OpenCLIP demonstrates that CLIP’s training recipe works effectively with public data. Models trained on LAION-2B match or exceed OpenAI CLIP performance, with the largest ViT-bigG/14 variant achieving 80.1% zero-shot accuracy on ImageNet.

References:

  • Paper: “Reproducible Scaling Laws for Contrastive Language-Image Learning” (Cherti et al., CVPR 2023)
  • Repository: https://github.com/mlfoundations/open_clip

Example

// Create an OpenCLIP model for open-source contrastive image-text alignment
// trained on LAION-2B/5B public datasets matching CLIP performance
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new OpenCLIP<double>(architecture, "openclip.onnx");
            
// Training mode with native layers
var trainModel = new OpenCLIP<double>(architecture, new OpenCLIPOptions());

Constructors

Constructor Summary
OpenCLIP(NeuralNetworkArchitecture<T> architecture, OpenCLIPOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an OpenCLIP model in native training mode.
OpenCLIP(NeuralNetworkArchitecture<T> architecture, string imageEncoderModelPath, OpenCLIPOptions options) Creates an OpenCLIP model in native training mode.

Properties

Property Summary
T Temperature { get; }
int EmbeddingDimension { get; }
int MaxSequenceLength { get; }
int ProjectionDimension { get; }
int TextEmbeddingDimension { get; }

Methods

Method Summary
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, string[] labels)
ModelMetadata<T> GetModelMetadata()
T ComputeSimilarity(Tensor<T> image, string text)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> EncodeText(string text)
Tensor<T>[] EncodeTexts(string[] texts)