CogVLM2<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

CogVLM2: improved visual expert architecture with GLM-4 backbone and video understanding.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → CogVLM2<T>

Implements: IInstructionTunedVLM<T>

For Beginners

CogVLM2 is the successor to CogVLM that upgrades the language backbone from Vicuna to GLM-4 (or LLaMA-3) and adds video understanding. It keeps the same “visual expert” approach — dedicated visual expert modules in every decoder layer with separate attention weights for image tokens versus text tokens. The key additions are temporal attention for processing video frame sequences and improved training that gives it stronger performance on both image and video tasks. Default values follow the original paper settings.

How It Works

CogVLM2 (Hong et al., 2024) improves upon CogVLM with GLM-4 (or LLaMA-3) as the language backbone, enhanced visual expert modules, and video understanding capabilities. It maintains the deep fusion approach of visual experts in every decoder layer while adding temporal attention for video frame sequences.

References:

  • Paper: “CogVLM2: Visual Language Models for Image and Video Understanding” (Hong et al., 2024)

Example

// Create a CogVLM2 model for image and video understanding
// using visual expert modules with GLM-4 backbone
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new CogVLM2<double>(architecture, "cogvlm2.onnx");
            
// Training mode with native layers
var trainModel = new CogVLM2<double>(architecture, new CogVLM2Options());

Constructors

Constructor Summary
CogVLM2(NeuralNetworkArchitecture<T> architecture, CogVLM2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
CogVLM2(NeuralNetworkArchitecture<T> architecture, string modelPath, CogVLM2Options options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using CogVLM2’s improved visual expert + video architecture.