Cambrian1<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

Cambrian-1: Spatial Vision Aggregator with 35+ vision encoder combinations.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Cambrian1<T>

Implements: IInstructionTunedVLM<T>

For Beginners

Cambrian-1 takes a unique approach to vision by combining multiple different vision encoders (CLIP, DINOv2, SigLIP, ConvNeXt — over 35 combinations tested) rather than using just one. Its Spatial Vision Aggregator (SVA) uses cross-attention to dynamically select and fuse the most useful features from each encoder based on what part of the image is relevant. This gives it stronger visual understanding than any single encoder alone, and it uses LLaMA-3 as its language backbone. Default values follow the original paper settings.

How It Works

Cambrian-1 (NYU, 2024) introduces the Spatial Vision Aggregator (SVA) that can combine features from multiple vision encoders (35+ combinations tested). It uses LLaMA-3 as the language backbone and demonstrates that diverse vision encoders improve multimodal understanding.

References:

  • Paper: “Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs” (2024)

Example

// Create a Cambrian-1 model for multi-encoder visual understanding
// using Spatial Vision Aggregator with 35+ vision encoder combinations
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new Cambrian1<double>(architecture, "cambrian1.onnx");
            
// Training mode with native layers
var trainModel = new Cambrian1<double>(architecture, new Cambrian1Options());

Constructors

Constructor Summary
Cambrian1(NeuralNetworkArchitecture<T> architecture, Cambrian1Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Cambrian1(NeuralNetworkArchitecture<T> architecture, string modelPath, Cambrian1Options options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text from an image using Cambrian-1’s Spatial Vision Aggregator (SVA).