PaLI3<T>

Models & Types · AiDotNet.VisionLanguage.Generative

PaLI-3: efficient PaLI with SigLIP ViT encoder, smaller and better.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → PaLI3<T>

Implements: IGenerativeVisionLanguageModel<T>

For Beginners

PaLI-3 achieves strong vision-language performance with a much smaller model than PaLI-X by replacing the contrastive ViT encoder with a SigLIP ViT that uses sigmoid-based contrastive loss instead of softmax. This produces better vision representations while being significantly more efficient, demonstrating that smarter pretraining can compensate for smaller model size. Default values follow the original paper settings.

How It Works

PaLI-3 (Chen et al., 2023) achieves strong performance with a smaller model by replacing the contrastive ViT with a SigLIP ViT encoder. The architecture retains the encoder-decoder design but benefits from the improved vision representations of SigLIP pretraining.

References:

  • Paper: “PaLI-3 Vision Language Models: Smaller, Faster, Stronger” (Chen et al., 2023)

Example

// Create a PaLI-3 model for efficient vision-language understanding
// with SigLIP ViT encoder for smaller, faster, stronger performance
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new PaLI3<double>(architecture, "pali3.onnx");
            
// Training mode with native layers
var trainModel = new PaLI3<double>(architecture, new PaLI3Options());

Constructors

Constructor Summary
PaLI3(NeuralNetworkArchitecture<T> architecture, PaLI3Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
PaLI3(NeuralNetworkArchitecture<T> architecture, string modelPath, PaLI3Options options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using PaLI-3’s efficient SigLIP-based architecture.