CogVLM<T>
Models & Types · AiDotNet.VisionLanguage.InstructionTuned
CogVLM: deep fusion via visual expert module in every LLM layer.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → CogVLM<T>
Implements: IInstructionTunedVLM<T>
For Beginners
CogVLM introduces a clever “visual expert” approach — instead of just prepending image tokens to the text, it adds a dedicated visual expert module inside every layer of the language model. Each decoder layer has separate attention weights and feed-forward weights for visual tokens versus text tokens. This deep fusion means the model can align visual and language features much more precisely without degrading the original language model’s text capabilities. It uses a massive EVA2-CLIP-E vision encoder (4.4B parameters, 63 layers) and Vicuna as the decoder. Default values follow the original paper settings.
How It Works
CogVLM (Wang et al., 2023) introduces a visual expert module in every layer of the language model for deep fusion of visual and language features. Each decoder layer has separate QKV matrices and FFN weights for visual tokens (the visual expert), enabling fine-grained visual-linguistic alignment without degrading language model capability.
References:
- Paper: “CogVLM: Visual Expert for Pretrained Language Models” (Wang et al., 2023)
Example
// Create a CogVLM model for deep visual-language fusion
// using visual expert modules in every LLM decoder layer
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new CogVLM<double>(architecture, "cogvlm.onnx");
// Training mode with native layers
var trainModel = new CogVLM<double>(architecture, new CogVLMOptions());
Constructors
| Constructor | Summary |
|---|---|
CogVLM(NeuralNetworkArchitecture<T> architecture, CogVLMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
CogVLM(NeuralNetworkArchitecture<T> architecture, string modelPath, CogVLMOptions options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
|
string LanguageModelName { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using CogVLM’s visual expert module architecture. |