RT2<T>
Models & Types · AiDotNet.VisionLanguage.Robotics
RT-2: vision-language-action model that transfers web knowledge to robotic control (Brohan et al., Google DeepMind, 2023, arXiv:2307.15818).
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → RT2<T>
Implements: IVisionLanguageAction<T>
For Beginners
RT-2 is the model that proved a generic Internet-trained vision-language
model can drive a real robot just by treating “move arm forward 3cm” as a sentence to generate,
rather than by training a separate policy network. Default values follow the published 55B-parameter
PaLI-X configuration but can be scaled down via RT2Options.
How It Works
RT-2 fine-tunes a large vision-language model (PaLI-X or PaLM-E) on robot demonstration data by representing continuous robot actions as text tokens. Each action dimension is uniformly discretized into 256 bins (paper §3.2) and each bin is mapped to one of the 256 least frequently used vocabulary tokens. The VLM then emits these tokens autoregressively just as it would emit ordinary text, so all of the model’s web-scale knowledge transfers directly to robot control. This class composes a generic ViT-style vision encoder, MLP projection, LLM decoder, action-bin head and full-vocabulary projection so the same forward path can serve both vision-language reasoning and robotic action generation, mirroring the encoder-decoder PaLI/PaLM-E backbone used in the paper.
Paper-faithful pieces implemented here:
What is NOT verified in-session:
Example
var image = Tensor<double>.CreateRandom(1, 3, 32, 32);
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new RT2<double>(architecture, "rt2.onnx");
// Training mode with native layers
var trainModel = new RT2<double>(architecture, new RT2Options());
var action = trainModel.PredictAction(image, "pick up the red cup");
Constructors
| Constructor | Summary |
|---|---|
RT2(NeuralNetworkArchitecture<T> architecture, RT2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
RT2(NeuralNetworkArchitecture<T> architecture, string modelPath, RT2Options options) |
Properties
| Property | Summary |
|---|---|
int ActionDimension { get; } |
|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
|
string LanguageModelName { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Runs the full encoder + decoder pipeline once and returns vocabulary-sized logits at the final position. |
Tensor<T> PredictAction(Tensor<T> observation, string instruction) |
Predicts a continuous robot action using RT-2’s action-as-text formulation (Brohan et al. |