LLaVANeuralNetwork<T>

Models & Types · AiDotNet.NeuralNetworks

LLaVA (Large Language and Vision Assistant) neural network for visual instruction following.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → LLaVANeuralNetwork<T>

Implements: ILLaVAModel<T>

For Beginners

LLaVA is like giving eyes to ChatGPT!

Architecture overview:

  1. Vision Encoder (CLIP ViT-L/14): Extracts image patch features
  2. Projection Layer (MLP): Maps visual features to LLM’s embedding space
  3. Large Language Model (LLaMA/Vicuna): Generates text responses

Key capabilities:

  • Visual conversations: “What’s in this image?” followed by “What color is the car?”
  • Visual reasoning: Understanding relationships, counting, spatial awareness
  • Instruction following: “Describe this image as if you were a poet”
  • Multi-turn dialogue: Context-aware conversations about images

How It Works

LLaVA connects a vision encoder (CLIP ViT) with a large language model (LLaMA/Vicuna) through a simple projection layer, enabling visual conversations and instruction following.

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 336, 336 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new LLaVANeuralNetwork<float>(architecture))
    .Build(trainX, trainY);
var output = result.Predict(image);

Constructors

Constructor Summary
LLaVANeuralNetwork(NeuralNetworkArchitecture<T> architecture, LLaVAOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a LLaVA network using native library layers.
LLaVANeuralNetwork(NeuralNetworkArchitecture<T> architecture, string visionEncoderPath, string languageModelPath, ITokenizer tokenizer, LLaVAOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a LLaVA network using pretrained ONNX models.

Properties

Property Summary
LanguageModelBackbone LanguageModelBackbone { get; }
int EmbeddingDimension { get; }
int ImageSize { get; }
int MaxSequenceLength { get; }
int NumVisualTokens { get; }
string VisionEncoderType { get; }

Methods

Method Summary
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, IEnumerable<string> classLabels)
Dictionary<string, T> ZeroShotClassify(double[] imageData, IEnumerable<string> labels)
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) Exposes named intermediate activations along LLaVA’s ACTUAL inference forward path (CLIP-ViT vision encoder -> visual projection), mirroring PredictCore and ForwardForTraining.
IEnumerable<ValueTuple<string, T>> GenerateMultiple(Tensor<T> image, string prompt, int numResponses, double temperature)
IEnumerable<Vector<T>> GetImageEmbeddings(IEnumerable<Tensor<T>> images)
IEnumerable<Vector<T>> GetTextEmbeddings(IEnumerable<string> texts)
IEnumerable<string> DescribeRegions(Tensor<T> image, IEnumerable<Vector<T>> regions)
IReadOnlyList<int> EncodeInstructionTokens(string instruction) Tokenizes an instruction using this model’s configured tokenizer.
Matrix<T> EncodeImageBatch(IEnumerable<double[]> imageDataBatch)
Matrix<T> EncodeTextBatch(IEnumerable<string> texts)
ModelMetadata<T> GetModelMetadata()
T ComputeSimilarity(Vector<T> textEmbedding, Vector<T> imageEmbedding)
Tensor<T> EncodeJointHiddenStates(Tensor<T> image, IReadOnlyList<int> tokenIds, Tensor<T> trailingTokenEmbeddings) Encodes image patches, instruction tokens and optional trailing learned tokens through the native language stack, returning hidden states rather than vocabulary logits.
Tensor<T> ExtractVisualFeatures(Tensor<T> image)
Tensor<T> ForwardForTraining(Tensor<T> input) Training forward pass.
Tensor<T> ProjectToLanguageSpace(Tensor<T> visualFeatures)
Vector<T> EncodeImage(double[] imageData)
Vector<T> EncodeText(string text)
Vector<T> GetImageEmbedding(Tensor<T> image)
Vector<T> GetTextEmbedding(string text)
Vector<T> GroundObject(Tensor<T> image, string description)
string Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage, int maxLength, double temperature)
string CompareImages(Tensor<T> image1, Tensor<T> image2, IEnumerable<string> aspectsToCompare)
string Generate(Tensor<T> image, string prompt, int maxLength, double temperature, double topP)