DocOwl<T>

Models & Types · AiDotNet.Document.VisionLanguage

DocOwl (mPLUG-DocOwl) for document understanding with multimodal large language model.

Inheritance: NeuralNetworkBase<T> → DocumentNeuralNetworkBase<T> → DocOwl<T>

Implements: IDocumentQA<T>, ILayoutDetector<T>

For Beginners

DocOwl brings LLM capabilities to documents:

  1. Understands complex document layouts
  2. Performs multi-page document understanding
  3. Handles diverse document types (forms, tables, charts)
  4. Generates detailed answers about document content

Key features:

  • Based on mPLUG-Owl multimodal architecture
  • Unified visual and text understanding
  • Fine-tuned on document-specific datasets
  • Strong generalization to unseen document types

Example usage:

How It Works

DocOwl is based on the mPLUG-Owl architecture, specifically fine-tuned for document understanding tasks. It combines a visual encoder with a large language model to understand and reason about document content.

Reference: “mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding” (arXiv 2023) https://arxiv.org/abs/2307.02499

Constructors

Constructor Summary
DocOwl(NeuralNetworkArchitecture<T> architecture, DocOwlOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a DocOwl model using native layers for training and inference.
DocOwl(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, DocOwlOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a DocOwl model using a pre-trained ONNX model for inference.

Properties

Property Summary
DocumentType SupportedDocumentTypes { get; }
IReadOnlyList<LayoutElementType> SupportedElementTypes { get; }
bool RequiresOCR { get; }
int ExpectedImageSize { get; }
int LanguageDim { get; } Gets the language model dimension.
int VisionDim { get; } Gets the vision encoder dimension.
int VisionNumHeads { get; } Gets the number of ViT attention heads.

Methods

Method Summary
Dictionary<string, DocumentQAResult<T>> ExtractFields(Tensor<T> documentImage, IEnumerable<string> fieldPrompts)
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) Reports activations through the multimodal walk rather than the linear chain.
DocumentLayoutResult<T> DetectLayout(Tensor<T> documentImage)
DocumentLayoutResult<T> DetectLayout(Tensor<T> documentImage, double confidenceThreshold)
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question)
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question, int maxAnswerLength, double temperature)
IEnumerable<DocumentQAResult<T>> AnswerQuestions(Tensor<T> documentImage, IEnumerable<string> questions)
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeDocument(Tensor<T> documentImage)
Tensor<T> ForwardForTraining(Tensor<T> input)
string GetModelSummary()
void ValidateInputShape(Tensor<T> documentImage)