FLIP<T>
Models & Types · AiDotNet.VisionLanguage.Encoders
FLIP (Fast Language-Image Pre-training) model that accelerates CLIP training via random masking of image patches.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → FLIP<T>
Implements: IContrastiveVisionLanguageModel<T>
For Beginners
FLIP speeds up CLIP training by randomly hiding 50-75% of image patches during training, dramatically reducing computation. At inference time, all patches are used for full accuracy. This simple trick allows training larger models or using more data within the same compute budget. Default values follow the original paper settings.
How It Works
FLIP (Li et al., 2022) randomly masks 50-75% of image patches during training, reducing computation while maintaining strong zero-shot performance. At inference, all patches are used.
References:
- Paper: “Scaling Language-Image Pre-training via Masking” (Li et al., 2022)
Example
// Create a FLIP model for accelerated contrastive training
// using random image patch masking for 2-4x training speedup
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new FLIP<double>(architecture, "flip.onnx");
// Training mode with native layers
var trainModel = new FLIP<double>(architecture, new FLIPOptions());
Constructors
| Constructor | Summary |
|---|---|
FLIP(NeuralNetworkArchitecture<T> architecture, FLIPOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
FLIP(NeuralNetworkArchitecture<T> architecture, string imageEncoderModelPath, FLIPOptions options) |
Properties
| Property | Summary |
|---|---|
T Temperature { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxSequenceLength { get; } |
|
int ProjectionDimension { get; } |
|
int TextEmbeddingDimension { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, string[] labels) |
|
ModelMetadata<T> GetModelMetadata() |
|
T ComputeSimilarity(Tensor<T> image, string text) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> EncodeText(string text) |
|
Tensor<T>[] EncodeTexts(string[] texts) |