SigLIP2<T>

Models & Types · AiDotNet.VisionLanguage.Encoders

SigLIP 2 (Multilingual Vision-Language Encoders with Improved Semantic Understanding) for contrastive encoding, zero-shot classification, and captioning.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → SigLIP2<T>

Implements: IContrastiveVisionLanguageModel<T>

For Beginners

SigLIP 2 improves on SigLIP by learning from three tasks at once: matching images with text, describing what’s in images, and filling in missing parts of images. This multi-task approach produces better visual representations. The model also supports many languages, making it useful for global applications.

Usage:

How It Works

SigLIP 2 (Tschannen et al., 2025) extends SigLIP with a multi-objective training framework that combines three losses: (1) sigmoid contrastive loss for image-text alignment, (2) an autoregressive captioning loss via a lightweight decoder for semantic understanding, and (3) a self-supervised masked image modeling (MIM) loss for spatial understanding. The model supports 32+ languages through an extended multilingual vocabulary.

Architecture:

  • Vision encoder: ViT with patch sizes 14 or 16, supporting 256-512px images
  • Text encoder: Transformer with multilingual SentencePiece tokenization (250K vocab)
  • Captioning decoder: Lightweight 4-layer autoregressive transformer that attends

to vision encoder features via cross-attention, trained with next-token prediction

  • MIM decoder: 2-layer MLP decoder that predicts masked patch features from

unmasked patch embeddings

  • Sigmoid contrastive loss: Per-pair binary classification as in SigLIP

Key Innovation: By training with multiple objectives simultaneously, SigLIP 2 produces vision encoders with richer semantic understanding than contrastive-only training. The captioning objective forces the encoder to capture fine-grained details, while MIM improves spatial awareness. Online data curation dynamically adjusts the training data mixture for better representation quality.

References:

  • Paper: “SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding”

(Tschannen et al., 2025)

  • Original SigLIP: “Sigmoid Loss for Language Image Pre-Training” (Zhai et al., ICCV 2023)
  • Repository: https://github.com/google-research/big_vision

Example

// Create a SigLIP 2 model for multi-objective contrastive training
// with captioning, MIM, and multilingual support for 32+ languages
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new SigLIP2<double>(architecture, "siglip2.onnx");
            
// Training mode with native layers
var trainModel = new SigLIP2<double>(architecture, new SigLIP2Options());

Constructors

Constructor Summary
SigLIP2(NeuralNetworkArchitecture<T> architecture, SigLIP2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a SigLIP 2 model in native training mode.
SigLIP2(NeuralNetworkArchitecture<T> architecture, string imageEncoderModelPath, SigLIP2Options options) Creates a SigLIP 2 model in native training mode.

Properties

Property Summary
T Temperature { get; }
int EmbeddingDimension { get; }
int MaxSequenceLength { get; }
int ProjectionDimension { get; }
int TextEmbeddingDimension { get; }

Methods

Method Summary
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, string[] labels)
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
ModelMetadata<T> GetModelMetadata()
T ComputeSimilarity(Tensor<T> image, string text)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> EncodeText(string text)
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> GenerateCaption(Tensor<T> image) Generates a caption for the given image with SigLIP 2’s LocCa decoder.
Tensor<T> PredictMaskedPatches(Tensor<T> image, int[] maskIndices) Predicts the features of masked patches with SigLIP 2’s masked-prediction head.
Tensor<T>[] EncodeTexts(string[] texts)
void TrainCaption(Tensor<T> image, IReadOnlyList<int> captionIds) One training step of SigLIP 2’s decoder objectives on an image and its caption (section 2.2-2.3): LocCa’s captioning loss weighted by CaptioningLossWeight, teacher-forced or, for half of the steps as LocCa does, with parallel prediction;…