BiomedCLIP<T>
Models & Types · AiDotNet.VisionLanguage.Encoders
BiomedCLIP model fine-tuned on 15M biomedical image-text pairs from PubMed Central.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → BiomedCLIP<T>
Implements: IContrastiveVisionLanguageModel<T>
For Beginners
BiomedCLIP adapts CLIP for the biomedical domain by fine-tuning on 15 million image-text pairs from PubMed Central articles. It uses ViT-B/16 for images and PubMedBERT for text, achieving state-of-the-art zero-shot biomedical image classification for tasks like pathology slide classification and radiology report matching. Default values follow the original paper settings.
How It Works
BiomedCLIP (Zhang et al., 2023) adapts CLIP for the biomedical domain using PMC-15M, achieving state-of-the-art zero-shot biomedical image classification with ViT-B/16 + PubMedBERT.
References:
- Paper: “BiomedCLIP: A Multimodal Biomedical Foundation Model” (Zhang et al., 2023)
Architecture layout: Mirrors PyTorch / HuggingFace CLIP — vision encoder (patch
embedding + transformer + projection) lives in Layers;
text encoder lives in a separate field surfaced through GetExtraTrainableLayers.
The default Predict + Train path is
vision-only — it patch-embeds the input and walks Layers. Text-encoder access goes
through EncodeText; contrastive image-text similarity through
ComputeSimilarity / ZeroShotClassify.
Example
// Create a BiomedCLIP model for biomedical image-text alignment
// fine-tuned on 15M PubMed Central image-text pairs
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new BiomedCLIP<double>(architecture, "biomedclip.onnx");
// Training mode with native layers
var trainModel = new BiomedCLIP<double>(architecture, new BiomedCLIPOptions());
Constructors
| Constructor | Summary |
|---|---|
BiomedCLIP(NeuralNetworkArchitecture<T> architecture, BiomedCLIPOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
BiomedCLIP(NeuralNetworkArchitecture<T> architecture, string imageEncoderModelPath, BiomedCLIPOptions options) |
Properties
| Property | Summary |
|---|---|
T Temperature { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxSequenceLength { get; } |
|
int ProjectionDimension { get; } |
|
int TextEmbeddingDimension { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, string[] labels) |
|
ModelMetadata<T> GetModelMetadata() |
|
T ComputeSimilarity(Tensor<T> image, string text) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> EncodeText(string text) |
|
Tensor<T>[] EncodeTexts(string[] texts) |