PaLME<T>
Models & Types · AiDotNet.VisionLanguage.Robotics
PaLM-E: 562B embodied multimodal language model for robotic planning.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → PaLME<T>
Implements: IVisionLanguageAction<T>
For Beginners
PaLM-E is a massive embodied vision-language model from Google for robotic planning and multimodal reasoning. Default values follow the original paper settings.
How It Works
PaLM-E (Google, 2023) is a 562 billion parameter embodied multimodal language model that integrates vision, language, and robot control. It injects continuous sensor observations (images, point clouds, robot state) as tokens into the PaLM language model, enabling embodied reasoning, task planning, and real-world robotic manipulation from natural language.
References:
- Paper: “PaLM-E: An Embodied Multimodal Language Model (Google, 2023)”
Example
// Create a PaLM-E model for embodied multimodal robotic planning
// 562B parameter model integrating vision, language, and robot control
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new PaLME<double>(architecture, "palme.onnx");
// Training mode with native layers
var trainModel = new PaLME<double>(architecture, new PaLMEOptions());
Constructors
| Constructor | Summary |
|---|---|
PaLME(NeuralNetworkArchitecture<T> architecture, PaLMEOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
PaLME(NeuralNetworkArchitecture<T> architecture, string modelPath, PaLMEOptions options) |
Properties
| Property | Summary |
|---|---|
int ActionDimension { get; } |
|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
|
string LanguageModelName { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
Override the tape-driven training-mode forward to inject the same patch embedding + NCHW→BSC reshape Predict applies. |
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates from image using PaLM-E’s embodied multimodal approach. |
Tensor<T> PredictAction(Tensor<T> observation, string instruction) |
Predicts action using PaLM-E’s embodied multimodal approach. |