Cambrian1<T>
Models & Types · AiDotNet.VisionLanguage.InstructionTuned
Cambrian-1: Spatial Vision Aggregator with 35+ vision encoder combinations.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Cambrian1<T>
Implements: IInstructionTunedVLM<T>
For Beginners
Cambrian-1 takes a unique approach to vision by combining multiple different vision encoders (CLIP, DINOv2, SigLIP, ConvNeXt — over 35 combinations tested) rather than using just one. Its Spatial Vision Aggregator (SVA) uses cross-attention to dynamically select and fuse the most useful features from each encoder based on what part of the image is relevant. This gives it stronger visual understanding than any single encoder alone, and it uses LLaMA-3 as its language backbone. Default values follow the original paper settings.
How It Works
Cambrian-1 (NYU, 2024) introduces the Spatial Vision Aggregator (SVA) that can combine features from multiple vision encoders (35+ combinations tested). It uses LLaMA-3 as the language backbone and demonstrates that diverse vision encoders improve multimodal understanding.
References:
- Paper: “Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs” (2024)
Example
// Create a Cambrian-1 model for multi-encoder visual understanding
// using Spatial Vision Aggregator with 35+ vision encoder combinations
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new Cambrian1<double>(architecture, "cambrian1.onnx");
// Training mode with native layers
var trainModel = new Cambrian1<double>(architecture, new Cambrian1Options());
Constructors
| Constructor | Summary |
|---|---|
Cambrian1(NeuralNetworkArchitecture<T> architecture, Cambrian1Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
Cambrian1(NeuralNetworkArchitecture<T> architecture, string modelPath, Cambrian1Options options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
|
string LanguageModelName { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text from an image using Cambrian-1’s Spatial Vision Aggregator (SVA). |