PointLLM<T>

Models & Types · AiDotNet.VisionLanguage.ThreeD

PointLLM: LLM understanding of colored 3D point clouds.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → PointLLM<T>

Implements: IThreeDVisionLanguageModel<T>

For Beginners

PointLLM is a vision-language model that enables LLMs to understand and reason about 3D point cloud data. Default values follow the original paper settings.

How It Works

PointLLM (OpenRobot Lab, 2024) empowers large language models to understand colored 3D point clouds. It encodes point cloud spatial coordinates and color features through a point cloud backbone, projects them into the LLM’s token space, and enables the model to reason about 3D object shapes, colors, and spatial relationships through natural language.

References:

  • Paper: “PointLLM: Empowering Large Language Models to Understand Point Clouds (OpenRobot Lab, 2024)”

Example

// Create a PointLLM model for understanding colored 3D point clouds
// with LLM-based reasoning over point cloud spatial features
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new PointLLM<double>(architecture, "pointllm.onnx");
            
// Training mode with native layers
var trainModel = new PointLLM<double>(architecture, new PointLLMOptions());

Constructors

Constructor Summary
PointLLM(NeuralNetworkArchitecture<T> architecture, PointLLMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
PointLLM(NeuralNetworkArchitecture<T> architecture, string modelPath, PointLLMOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
int MaxPoints { get; }
int PointChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFrom3D(Tensor<T> pointCloud, string prompt) Processes 3D point cloud using PointLLM’s point cloud tokenization approach.
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates from 2D image using PointLLM’s point cloud tokenization approach.