DocOwl<T>
Models & Types · AiDotNet.Document.VisionLanguage
DocOwl (mPLUG-DocOwl) for document understanding with multimodal large language model.
Inheritance: NeuralNetworkBase<T> → DocumentNeuralNetworkBase<T> → DocOwl<T>
Implements: IDocumentQA<T>, ILayoutDetector<T>
For Beginners
DocOwl brings LLM capabilities to documents:
- Understands complex document layouts
- Performs multi-page document understanding
- Handles diverse document types (forms, tables, charts)
- Generates detailed answers about document content
Key features:
- Based on mPLUG-Owl multimodal architecture
- Unified visual and text understanding
- Fine-tuned on document-specific datasets
- Strong generalization to unseen document types
Example usage:
How It Works
DocOwl is based on the mPLUG-Owl architecture, specifically fine-tuned for document understanding tasks. It combines a visual encoder with a large language model to understand and reason about document content.
Reference: “mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding” (arXiv 2023)
https://arxiv.org/abs/2307.02499
Constructors
| Constructor | Summary |
|---|---|
DocOwl(NeuralNetworkArchitecture<T> architecture, DocOwlOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a DocOwl model using native layers for training and inference. |
DocOwl(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, DocOwlOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a DocOwl model using a pre-trained ONNX model for inference. |
Properties
| Property | Summary |
|---|---|
DocumentType SupportedDocumentTypes { get; } |
|
IReadOnlyList<LayoutElementType> SupportedElementTypes { get; } |
|
bool RequiresOCR { get; } |
|
int ExpectedImageSize { get; } |
|
int LanguageDim { get; } |
Gets the language model dimension. |
int VisionDim { get; } |
Gets the vision encoder dimension. |
int VisionNumHeads { get; } |
Gets the number of ViT attention heads. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, DocumentQAResult<T>> ExtractFields(Tensor<T> documentImage, IEnumerable<string> fieldPrompts) |
|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
Reports activations through the multimodal walk rather than the linear chain. |
DocumentLayoutResult<T> DetectLayout(Tensor<T> documentImage) |
|
DocumentLayoutResult<T> DetectLayout(Tensor<T> documentImage, double confidenceThreshold) |
|
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question) |
|
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question, int maxAnswerLength, double temperature) |
|
IEnumerable<DocumentQAResult<T>> AnswerQuestions(Tensor<T> documentImage, IEnumerable<string> questions) |
|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeDocument(Tensor<T> documentImage) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
string GetModelSummary() |
|
void ValidateInputShape(Tensor<T> documentImage) |