PaLI3<T>
Models & Types · AiDotNet.VisionLanguage.Generative
PaLI-3: efficient PaLI with SigLIP ViT encoder, smaller and better.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → PaLI3<T>
Implements: IGenerativeVisionLanguageModel<T>
For Beginners
PaLI-3 achieves strong vision-language performance with a much smaller model than PaLI-X by replacing the contrastive ViT encoder with a SigLIP ViT that uses sigmoid-based contrastive loss instead of softmax. This produces better vision representations while being significantly more efficient, demonstrating that smarter pretraining can compensate for smaller model size. Default values follow the original paper settings.
How It Works
PaLI-3 (Chen et al., 2023) achieves strong performance with a smaller model by replacing the contrastive ViT with a SigLIP ViT encoder. The architecture retains the encoder-decoder design but benefits from the improved vision representations of SigLIP pretraining.
References:
- Paper: “PaLI-3 Vision Language Models: Smaller, Faster, Stronger” (Chen et al., 2023)
Example
// Create a PaLI-3 model for efficient vision-language understanding
// with SigLIP ViT encoder for smaller, faster, stronger performance
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new PaLI3<double>(architecture, "pali3.onnx");
// Training mode with native layers
var trainModel = new PaLI3<double>(architecture, new PaLI3Options());
Constructors
| Constructor | Summary |
|---|---|
PaLI3(NeuralNetworkArchitecture<T> architecture, PaLI3Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
PaLI3(NeuralNetworkArchitecture<T> architecture, string modelPath, PaLI3Options options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using PaLI-3’s efficient SigLIP-based architecture. |