BiomedCLIP<T>

Models & Types · AiDotNet.VisionLanguage.Encoders

BiomedCLIP model fine-tuned on 15M biomedical image-text pairs from PubMed Central.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → BiomedCLIP<T>

Implements: IContrastiveVisionLanguageModel<T>

For Beginners

BiomedCLIP adapts CLIP for the biomedical domain by fine-tuning on 15 million image-text pairs from PubMed Central articles. It uses ViT-B/16 for images and PubMedBERT for text, achieving state-of-the-art zero-shot biomedical image classification for tasks like pathology slide classification and radiology report matching. Default values follow the original paper settings.

How It Works

BiomedCLIP (Zhang et al., 2023) adapts CLIP for the biomedical domain using PMC-15M, achieving state-of-the-art zero-shot biomedical image classification with ViT-B/16 + PubMedBERT.

References:

  • Paper: “BiomedCLIP: A Multimodal Biomedical Foundation Model” (Zhang et al., 2023)

Architecture layout: Mirrors PyTorch / HuggingFace CLIP — vision encoder (patch embedding + transformer + projection) lives in Layers; text encoder lives in a separate field surfaced through GetExtraTrainableLayers. The default Predict + Train path is vision-only — it patch-embeds the input and walks Layers. Text-encoder access goes through EncodeText; contrastive image-text similarity through ComputeSimilarity / ZeroShotClassify.

Example

// Create a BiomedCLIP model for biomedical image-text alignment
// fine-tuned on 15M PubMed Central image-text pairs
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new BiomedCLIP<double>(architecture, "biomedclip.onnx");
            
// Training mode with native layers
var trainModel = new BiomedCLIP<double>(architecture, new BiomedCLIPOptions());

Constructors

Constructor Summary
BiomedCLIP(NeuralNetworkArchitecture<T> architecture, BiomedCLIPOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
BiomedCLIP(NeuralNetworkArchitecture<T> architecture, string imageEncoderModelPath, BiomedCLIPOptions options)

Properties

Property Summary
T Temperature { get; }
int EmbeddingDimension { get; }
int MaxSequenceLength { get; }
int ProjectionDimension { get; }
int TextEmbeddingDimension { get; }

Methods

Method Summary
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, string[] labels)
ModelMetadata<T> GetModelMetadata()
T ComputeSimilarity(Tensor<T> image, string text)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> EncodeText(string text)
Tensor<T>[] EncodeTexts(string[] texts)