NVLM<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

NVLM 1.0: NVIDIA’s cross-attention + decoder-only hybrid VLM that retains text performance.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → NVLM<T>

Implements: IInstructionTunedVLM<T>

For Beginners

NVLM from NVIDIA takes a hybrid approach that combines two ways of processing visual information: cross-attention (where the language model attends to visual features through dedicated cross-attention layers) and decoder-only (where image tokens are simply concatenated with text tokens). Most models use one or the other, but NVLM uses both, which lets it retain strong text-only performance while adding powerful vision capabilities. This is important because many multimodal models lose some text ability when adding vision — NVLM avoids this trade-off. Default values follow the original paper settings.

How It Works

NVLM 1.0 (NVIDIA, 2024) introduces a hybrid architecture combining cross-attention and decoder-only approaches. This design allows the model to process visual features via both cross-attention layers and direct token concatenation, retaining strong text-only performance.

References:

  • Paper: “NVLM: Open Frontier-Class Multimodal LLMs” (NVIDIA, 2024)

Example

// Create an NVLM model for hybrid cross-attention + decoder-only VLM
// retaining strong text performance alongside vision capabilities
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new NVLM<double>(architecture, "nvlm.onnx");
            
// Training mode with native layers
var trainModel = new NVLM<double>(architecture, new NVLMOptions());

Constructors

Constructor Summary
NVLM(NeuralNetworkArchitecture<T> architecture, NVLMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
NVLM(NeuralNetworkArchitecture<T> architecture, string modelPath, NVLMOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using NVLM’s cross-attention + decoder-only hybrid architecture.