MedCLIP<T>
Models & Types · AiDotNet.VisionLanguage.Encoders
MedCLIP model using decoupled semantic matching for medical image-text alignment.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → MedCLIP<T>
Implements: IContrastiveVisionLanguageModel<T>
For Beginners
MedCLIP adapts CLIP for medical imaging by solving a key problem: medical datasets are small and images are not always paired with matching text. It uses decoupled semantic matching — any medical image can be paired with any text that shares the same medical concepts (like “pneumonia” or “chest X-ray”), allowing it to learn from unpaired data. Default values follow the original paper settings.
How It Works
MedCLIP (Wang et al., 2022) addresses limited medical data by decoupling image-text inputs: any image can be paired with any text sharing the same medical concepts (diagnosis, anatomy), using a semantic matching loss alongside contrastive learning.
References:
- Paper: “MedCLIP: Contrastive Learning from Unpaired Medical Images and Text” (Wang et al., EMNLP 2022)
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new MedCLIP<double>(architecture, new MedCLIPOptions());
Constructors
| Constructor | Summary |
|---|---|
MedCLIP(NeuralNetworkArchitecture<T> architecture, MedCLIPOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
MedCLIP(NeuralNetworkArchitecture<T> architecture, string imageEncoderModelPath, MedCLIPOptions options) |
Properties
| Property | Summary |
|---|---|
T Temperature { get; } |
Gets the current learned temperature (the reciprocal of the clamped logit scale). |
bool UsesBiasFreeReferenceProjections { get; } |
Gets whether both reference 512-dimensional projections are bias-free. |
bool UsesTokenTypeEmbeddings { get; } |
Gets whether the ClinicalBERT embedding stack retains token-type embeddings. |
int EmbeddingDimension { get; } |
|
int MaxSequenceLength { get; } |
|
int ProjectionDimension { get; } |
|
int TextEmbeddingDimension { get; } |
|
int TextTransformerBlockCount { get; } |
Gets the number of ClinicalBERT transformer blocks. |
int VisionBottleneckBlockCount { get; } |
Gets the number of ResNet-50 bottleneck blocks. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, string[] labels) |
|
ModelMetadata<T> GetModelMetadata() |
|
T ComputeSemanticMatchingLoss(Tensor<T> imageEmbeddings, Tensor<T> textEmbeddings, Tensor<T> semanticScores) |
Computes MedCLIP’s bidirectional semantic matching objective for decoupled image/report batches. |
T ComputeSimilarity(Tensor<T> image, string text) |
|
Tensor<T> ComputeSemanticMatchingLossTensor(Tensor<T> imageEmbeddings, Tensor<T> textEmbeddings, Tensor<T> semanticScores) |
Computes the tape-connected MedCLIP semantic objective. |
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> EncodeText(string text) |
|
Tensor<T>[] EncodeTexts(string[] texts) |