SigLIP2<T>
Models & Types · AiDotNet.VisionLanguage.Encoders
SigLIP 2 (Multilingual Vision-Language Encoders with Improved Semantic Understanding) for contrastive encoding, zero-shot classification, and captioning.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → SigLIP2<T>
Implements: IContrastiveVisionLanguageModel<T>
For Beginners
SigLIP 2 improves on SigLIP by learning from three tasks at once: matching images with text, describing what’s in images, and filling in missing parts of images. This multi-task approach produces better visual representations. The model also supports many languages, making it useful for global applications.
Usage:
How It Works
SigLIP 2 (Tschannen et al., 2025) extends SigLIP with a multi-objective training framework that combines three losses: (1) sigmoid contrastive loss for image-text alignment, (2) an autoregressive captioning loss via a lightweight decoder for semantic understanding, and (3) a self-supervised masked image modeling (MIM) loss for spatial understanding. The model supports 32+ languages through an extended multilingual vocabulary.
Architecture:
- Vision encoder: ViT with patch sizes 14 or 16, supporting 256-512px images
- Text encoder: Transformer with multilingual SentencePiece tokenization (250K vocab)
- Captioning decoder: Lightweight 4-layer autoregressive transformer that attends
to vision encoder features via cross-attention, trained with next-token prediction
- MIM decoder: 2-layer MLP decoder that predicts masked patch features from
unmasked patch embeddings
- Sigmoid contrastive loss: Per-pair binary classification as in SigLIP
Key Innovation: By training with multiple objectives simultaneously, SigLIP 2 produces vision encoders with richer semantic understanding than contrastive-only training. The captioning objective forces the encoder to capture fine-grained details, while MIM improves spatial awareness. Online data curation dynamically adjusts the training data mixture for better representation quality.
References:
- Paper: “SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding”
(Tschannen et al., 2025)
- Original SigLIP: “Sigmoid Loss for Language Image Pre-Training” (Zhai et al., ICCV 2023)
- Repository:
https://github.com/google-research/big_vision
Example
// Create a SigLIP 2 model for multi-objective contrastive training
// with captioning, MIM, and multilingual support for 32+ languages
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new SigLIP2<double>(architecture, "siglip2.onnx");
// Training mode with native layers
var trainModel = new SigLIP2<double>(architecture, new SigLIP2Options());
Constructors
| Constructor | Summary |
|---|---|
SigLIP2(NeuralNetworkArchitecture<T> architecture, SigLIP2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a SigLIP 2 model in native training mode. |
SigLIP2(NeuralNetworkArchitecture<T> architecture, string imageEncoderModelPath, SigLIP2Options options) |
Creates a SigLIP 2 model in native training mode. |
Properties
| Property | Summary |
|---|---|
T Temperature { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxSequenceLength { get; } |
|
int ProjectionDimension { get; } |
|
int TextEmbeddingDimension { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, string[] labels) |
|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
ModelMetadata<T> GetModelMetadata() |
|
T ComputeSimilarity(Tensor<T> image, string text) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> EncodeText(string text) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> GenerateCaption(Tensor<T> image) |
Generates a caption for the given image with SigLIP 2’s LocCa decoder. |
Tensor<T> PredictMaskedPatches(Tensor<T> image, int[] maskIndices) |
Predicts the features of masked patches with SigLIP 2’s masked-prediction head. |
Tensor<T>[] EncodeTexts(string[] texts) |
|
void TrainCaption(Tensor<T> image, IReadOnlyList<int> captionIds) |
One training step of SigLIP 2’s decoder objectives on an image and its caption (section 2.2-2.3): LocCa’s captioning loss weighted by CaptioningLossWeight, teacher-forced or, for half of the steps as LocCa does, with parallel prediction;… |