LLaVACoT<T>

Models & Types · AiDotNet.VisionLanguage.Reasoning

LLaVA-CoT: chain-of-thought visual reasoning with structured output.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → LLaVACoT<T>

Implements: IReasoningVLM<T>

For Beginners

LLaVA-CoT is a vision-language model that reasons step-by-step through visual problems using chain-of-thought. Default values follow the original paper settings.

How It Works

LLaVA-CoT (2024) extends the LLaVA architecture with chain-of-thought visual reasoning, generating structured step-by-step reasoning traces for complex visual tasks. The model produces intermediate reasoning steps including observation, hypothesis, and conclusion phases before arriving at the final answer, improving accuracy on visual reasoning benchmarks.

References:

  • Paper: “LLaVA-CoT: Let Vision Language Models Reason Step-by-Step” (2024)

Example

// Create a LLaVA-CoT model for step-by-step visual reasoning
// with chain-of-thought structured output for complex visual tasks
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new LLaVACoT<double>(architecture, "llavacot.onnx");
            
// Training mode with native layers
var trainModel = new LLaVACoT<double>(architecture, new LLaVACoTOptions());

Constructors

Constructor Summary
LLaVACoT(NeuralNetworkArchitecture<T> architecture, LLaVACoTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
LLaVACoT(NeuralNetworkArchitecture<T> architecture, string modelPath, LLaVACoTOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }
string ReasoningApproach { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text from image using LLaVA-CoT’s structured reasoning pipeline.
Tensor<T> ReasonWithChainOfThought(Tensor<T> image, string question) Generates structured chain-of-thought reasoning using LLaVA-CoT’s 4-stage pipeline.