CogVLM2<T>
Models & Types · AiDotNet.VisionLanguage.InstructionTuned
CogVLM2: improved visual expert architecture with GLM-4 backbone and video understanding.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → CogVLM2<T>
Implements: IInstructionTunedVLM<T>
For Beginners
CogVLM2 is the successor to CogVLM that upgrades the language backbone from Vicuna to GLM-4 (or LLaMA-3) and adds video understanding. It keeps the same “visual expert” approach — dedicated visual expert modules in every decoder layer with separate attention weights for image tokens versus text tokens. The key additions are temporal attention for processing video frame sequences and improved training that gives it stronger performance on both image and video tasks. Default values follow the original paper settings.
How It Works
CogVLM2 (Hong et al., 2024) improves upon CogVLM with GLM-4 (or LLaMA-3) as the language backbone, enhanced visual expert modules, and video understanding capabilities. It maintains the deep fusion approach of visual experts in every decoder layer while adding temporal attention for video frame sequences.
References:
- Paper: “CogVLM2: Visual Language Models for Image and Video Understanding” (Hong et al., 2024)
Example
// Create a CogVLM2 model for image and video understanding
// using visual expert modules with GLM-4 backbone
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new CogVLM2<double>(architecture, "cogvlm2.onnx");
// Training mode with native layers
var trainModel = new CogVLM2<double>(architecture, new CogVLM2Options());
Constructors
| Constructor | Summary |
|---|---|
CogVLM2(NeuralNetworkArchitecture<T> architecture, CogVLM2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
CogVLM2(NeuralNetworkArchitecture<T> architecture, string modelPath, CogVLM2Options options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
|
string LanguageModelName { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using CogVLM2’s improved visual expert + video architecture. |