DFNCLIP<T>
Models & Types · AiDotNet.VisionLanguage.Encoders
DFN-CLIP (Data Filtering Networks for CLIP) model using filtered high-quality training data.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → DFNCLIP<T>
Implements: IContrastiveVisionLanguageModel<T>
For Beginners
DFN-CLIP improves CLIP by using a small trained CLIP model to score and filter image-text pairs from a massive noisy dataset, keeping only high-quality pairs. The larger model trained on this filtered data achieves 83.0% zero-shot ImageNet accuracy with ViT-H/14, demonstrating that data quality matters more than quantity. Default values follow the original paper settings.
How It Works
DFN-CLIP (Fang et al., 2023) uses a small CLIP model to score and filter image-text pairs from a large noisy pool, then trains a larger model on only high-quality data. Achieves 83.0% zero-shot on ImageNet with ViT-H/14.
References:
- Paper: “Data Filtering Networks” (Fang et al., 2023)
Architecture layout: Mirrors the PyTorch / HuggingFace CLIP layout —
vision encoder (patch embedding + transformer + projection) lives in
Layers so the default forward and tape-based
training paths walk it correctly. The text encoder lives in a separate field
surfaced through GetExtraTrainableLayers so it participates in
streaming offload / weight-registry hooks but isn’t on the vision-only forward
path. Real contrastive training is done through paired-data APIs
(ComputeSimilarity / ZeroShotClassify); the
inherited Predict + Train
surface is the vision-encoder-only path that lets CLIP plug into
generic NN consumers without crashing on cross-stack layer walks.
Example
// Create a DFN-CLIP model trained on filtered high-quality data
// using data filtering networks for improved CLIP training
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new DFNCLIP<double>(architecture, "dfnclip.onnx");
// Training mode with native layers
var trainModel = new DFNCLIP<double>(architecture, new DFNCLIPOptions());
Constructors
| Constructor | Summary |
|---|---|
DFNCLIP(NeuralNetworkArchitecture<T> architecture, DFNCLIPOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
DFNCLIP(NeuralNetworkArchitecture<T> architecture, string imageEncoderModelPath, DFNCLIPOptions options) |
Properties
| Property | Summary |
|---|---|
T Temperature { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxSequenceLength { get; } |
|
int ProjectionDimension { get; } |
|
int TextEmbeddingDimension { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, string[] labels) |
|
ModelMetadata<T> GetModelMetadata() |
|
T ComputeSimilarity(Tensor<T> image, string text) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> EncodeText(string text) |
|
Tensor<T>[] EncodeTexts(string[] texts) |