InfographicVQA<T>

Models & Types · AiDotNet.Document.VisionLanguage

InfographicVQA for visual question answering on infographics.

Inheritance: NeuralNetworkBase<T> → DocumentNeuralNetworkBase<T> → InfographicVQA<T>

Implements: IDocumentQA<T>

For Beginners

InfographicVQA specializes in complex visual documents:

  1. Understands mixed content (text, charts, icons, diagrams)
  2. Handles complex multi-column layouts
  3. Performs visual reasoning across different element types
  4. Extracts information from visually rich documents

Key features:

  • Multi-scale visual processing
  • OCR integration for text extraction
  • Visual reasoning across elements
  • Trained on InfographicsVQA dataset

Example usage:

How It Works

InfographicVQA is designed to understand and answer questions about infographics, which combine text, icons, charts, diagrams, and other visual elements in complex layouts.

Reference: “InfographicVQA” (WACV 2022) https://arxiv.org/abs/2104.12756

Constructors

Constructor Summary
InfographicVQA(NeuralNetworkArchitecture<T> architecture, InfographicVQAOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates an InfographicVQA model using native layers for training and inference.
InfographicVQA(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, InfographicVQAOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates an InfographicVQA model using a pre-trained ONNX model for inference.

Properties

Property Summary
DocumentType SupportedDocumentTypes { get; }
IReadOnlyList<string> SupportedElementTypes { get; } Gets the supported infographic element types.
bool RequiresOCR { get; }
int ExpectedImageSize { get; }
int FusionDim { get; } Gets the fusion dimension.
int VisionDim { get; } Gets the vision encoder dimension.

Methods

Method Summary
Dictionary<string, DocumentQAResult<T>> ExtractFields(Tensor<T> documentImage, IEnumerable<string> fieldPrompts)
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question)
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question, int maxAnswerLength, double temperature)
IEnumerable<DocumentQAResult<T>> AnswerQuestions(Tensor<T> documentImage, IEnumerable<string> questions)
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeDocument(Tensor<T> documentImage)
Tensor<T> ForwardForTraining(Tensor<T> input)
string GetModelSummary()
void ValidateInputShape(Tensor<T> documentImage)