PaLIX<T>
Models & Types · AiDotNet.VisionLanguage.Generative
PaLI-X: scaled PaLI to 55B with improved component fine-tuning.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → PaLIX<T>
Implements: IGenerativeVisionLanguageModel<T>
For Beginners
PaLI-X scales the PaLI architecture to 55 billion parameters by pairing the ViT-22B (the largest dense Vision Transformer) with a 32B UL2 language model. It introduces component-wise fine-tuning that allows separately optimizing the vision and language components for improved benchmark performance across captioning, VQA, and document understanding tasks. Default values follow the original paper settings.
How It Works
PaLI-X (Chen et al., 2023) scales PaLI to 55B parameters by using a ViT-22B vision encoder and a 32B UL2 language model. It introduces component-wise fine-tuning for improved performance on vision-language benchmarks.
References:
- Paper: “PaLI-X: On Scaling up a Multilingual Vision and Language Model” (Chen et al., 2023)
Example
// Create a PaLI-X model for scaled 55B vision-language understanding
// with ViT-22B encoder + 32B UL2 decoder and component-wise fine-tuning
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new PaLIX<double>(architecture, "palix.onnx");
// Training mode with native layers
var trainModel = new PaLIX<double>(architecture, new PaLIXOptions());
Constructors
| Constructor | Summary |
|---|---|
PaLIX(NeuralNetworkArchitecture<T> architecture, PaLIXOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
PaLIX(NeuralNetworkArchitecture<T> architecture, string modelPath, PaLIXOptions options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using PaLI-X’s scaled ViT-22B + UL2 architecture. |