LLaVANeuralNetwork<T>
Models & Types · AiDotNet.NeuralNetworks
LLaVA (Large Language and Vision Assistant) neural network for visual instruction following.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → LLaVANeuralNetwork<T>
Implements: ILLaVAModel<T>
For Beginners
LLaVA is like giving eyes to ChatGPT!
Architecture overview:
- Vision Encoder (CLIP ViT-L/14): Extracts image patch features
- Projection Layer (MLP): Maps visual features to LLM’s embedding space
- Large Language Model (LLaMA/Vicuna): Generates text responses
Key capabilities:
- Visual conversations: “What’s in this image?” followed by “What color is the car?”
- Visual reasoning: Understanding relationships, counting, spatial awareness
- Instruction following: “Describe this image as if you were a poet”
- Multi-turn dialogue: Context-aware conversations about images
How It Works
LLaVA connects a vision encoder (CLIP ViT) with a large language model (LLaMA/Vicuna) through a simple projection layer, enabling visual conversations and instruction following.
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 336, 336 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new LLaVANeuralNetwork<float>(architecture))
.Build(trainX, trainY);
var output = result.Predict(image);
Constructors
| Constructor | Summary |
|---|---|
LLaVANeuralNetwork(NeuralNetworkArchitecture<T> architecture, LLaVAOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a LLaVA network using native library layers. |
LLaVANeuralNetwork(NeuralNetworkArchitecture<T> architecture, string visionEncoderPath, string languageModelPath, ITokenizer tokenizer, LLaVAOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a LLaVA network using pretrained ONNX models. |
Properties
| Property | Summary |
|---|---|
LanguageModelBackbone LanguageModelBackbone { get; } |
|
int EmbeddingDimension { get; } |
|
int ImageSize { get; } |
|
int MaxSequenceLength { get; } |
|
int NumVisualTokens { get; } |
|
string VisionEncoderType { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, IEnumerable<string> classLabels) |
|
Dictionary<string, T> ZeroShotClassify(double[] imageData, IEnumerable<string> labels) |
|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
Exposes named intermediate activations along LLaVA’s ACTUAL inference forward path (CLIP-ViT vision encoder -> visual projection), mirroring PredictCore and ForwardForTraining. |
IEnumerable<ValueTuple<string, T>> GenerateMultiple(Tensor<T> image, string prompt, int numResponses, double temperature) |
|
IEnumerable<Vector<T>> GetImageEmbeddings(IEnumerable<Tensor<T>> images) |
|
IEnumerable<Vector<T>> GetTextEmbeddings(IEnumerable<string> texts) |
|
IEnumerable<string> DescribeRegions(Tensor<T> image, IEnumerable<Vector<T>> regions) |
|
IReadOnlyList<int> EncodeInstructionTokens(string instruction) |
Tokenizes an instruction using this model’s configured tokenizer. |
Matrix<T> EncodeImageBatch(IEnumerable<double[]> imageDataBatch) |
|
Matrix<T> EncodeTextBatch(IEnumerable<string> texts) |
|
ModelMetadata<T> GetModelMetadata() |
|
T ComputeSimilarity(Vector<T> textEmbedding, Vector<T> imageEmbedding) |
|
Tensor<T> EncodeJointHiddenStates(Tensor<T> image, IReadOnlyList<int> tokenIds, Tensor<T> trailingTokenEmbeddings) |
Encodes image patches, instruction tokens and optional trailing learned tokens through the native language stack, returning hidden states rather than vocabulary logits. |
Tensor<T> ExtractVisualFeatures(Tensor<T> image) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
Training forward pass. |
Tensor<T> ProjectToLanguageSpace(Tensor<T> visualFeatures) |
|
Vector<T> EncodeImage(double[] imageData) |
|
Vector<T> EncodeText(string text) |
|
Vector<T> GetImageEmbedding(Tensor<T> image) |
|
Vector<T> GetTextEmbedding(string text) |
|
Vector<T> GroundObject(Tensor<T> image, string description) |
|
string Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage, int maxLength, double temperature) |
|
string CompareImages(Tensor<T> image1, Tensor<T> image2, IEnumerable<string> aspectsToCompare) |
|
string Generate(Tensor<T> image, string prompt, int maxLength, double temperature, double topP) |