PaLIX<T>

Models & Types · AiDotNet.VisionLanguage.Generative

PaLI-X: scaled PaLI to 55B with improved component fine-tuning.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → PaLIX<T>

Implements: IGenerativeVisionLanguageModel<T>

For Beginners

PaLI-X scales the PaLI architecture to 55 billion parameters by pairing the ViT-22B (the largest dense Vision Transformer) with a 32B UL2 language model. It introduces component-wise fine-tuning that allows separately optimizing the vision and language components for improved benchmark performance across captioning, VQA, and document understanding tasks. Default values follow the original paper settings.

How It Works

PaLI-X (Chen et al., 2023) scales PaLI to 55B parameters by using a ViT-22B vision encoder and a 32B UL2 language model. It introduces component-wise fine-tuning for improved performance on vision-language benchmarks.

References:

  • Paper: “PaLI-X: On Scaling up a Multilingual Vision and Language Model” (Chen et al., 2023)

Example

// Create a PaLI-X model for scaled 55B vision-language understanding
// with ViT-22B encoder + 32B UL2 decoder and component-wise fine-tuning
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new PaLIX<double>(architecture, "palix.onnx");
            
// Training mode with native layers
var trainModel = new PaLIX<double>(architecture, new PaLIXOptions());

Constructors

Constructor Summary
PaLIX(NeuralNetworkArchitecture<T> architecture, PaLIXOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
PaLIX(NeuralNetworkArchitecture<T> architecture, string modelPath, PaLIXOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using PaLI-X’s scaled ViT-22B + UL2 architecture.