UDOP<T>

Models & Types · AiDotNet.Document.VisionLanguage

UDOP (Unifying Vision, Text, and Layout for Universal Document Processing) neural network.

Inheritance: NeuralNetworkBase<T> → DocumentNeuralNetworkBase<T> → UDOP<T>

Implements: IDocumentClassifier<T>, IDocumentQA<T>, ILayoutDetector<T>

For Beginners

UDOP can handle many document tasks with one model:

  1. Document classification
  2. Information extraction (NER, key-value pairs)
  3. Document question answering
  4. Document layout analysis
  5. Document generation

Example usage:

How It Works

UDOP is a foundation model for document AI that unifies text, image, and layout modalities within a single encoder-decoder framework. It can perform multiple document tasks through task-specific prompting.

Reference: “Unifying Vision, Text, and Layout for Universal Document Processing” (CVPR 2023) https://arxiv.org/abs/2212.02623

Constructors

Constructor Summary
UDOP(NeuralNetworkArchitecture<T> architecture, UDOPOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a trainable UDOP model built from options.
UDOP(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, ITokenizer tokenizer, UDOPOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a UDOP model that runs a pretrained ONNX export.

Properties

Property Summary
DocumentType SupportedDocumentTypes { get; }
IReadOnlyList<LayoutElementType> SupportedElementTypes { get; }
IReadOnlyList<string> AvailableCategories { get; }
bool RequiresOCR { get; }
int ExpectedImageSize { get; } Side of the square page image the patch embedding expects.

Methods

Method Summary
Dictionary<string, DocumentQAResult<T>> ExtractFields(Tensor<T> documentImage, IEnumerable<string> fieldPrompts)
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
DocumentClassificationResult<T> ClassifyDocument(Tensor<T> documentImage)
DocumentClassificationResult<T> ClassifyDocument(Tensor<T> documentImage, int topK) Classifies by prompting with UDOP’s RVL-CDIP task prompt and scoring every category name as a decoder target.
DocumentLayoutResult<T> DetectLayout(Tensor<T> documentImage)
DocumentLayoutResult<T> DetectLayout(Tensor<T> documentImage, double confidenceThreshold) Layout analysis by prompting: the decoder writes a category name followed by four location tokens per region, as UDOP’s layout task does.
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question)
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question, int maxAnswerLength, double temperature) Answers by prompting the encoder with `“Question answering.
IEnumerable<DocumentQAResult<T>> AnswerQuestions(Tensor<T> documentImage, IEnumerable<string> questions)
ModelMetadata<T> GetModelMetadata()
Tensor<T> DecoderLogits(Tensor<T> packedTokens, Tensor<T> pageImage, IReadOnlyList<int> decoderInput) Teacher-forced decoder logits [T, vocab]: row t predicts the token after decoderInput[t].
Tensor<T> EncodeDocument(Tensor<T> documentImage) Returns the encoder’s output states for a page: one row per unclaimed image patch.
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> GenerateTokens(Tensor<T> packedTokens, Tensor<T> pageImage, IReadOnlyList<int> promptTokens) Greedily decodes from the start token until EOS or MaxGenerationLength.
Tensor<T> PredictDocument(Tensor<T> packedTokens, Tensor<T> pageImage) Returns the decoder’s next-token logits [1, vocab] for the start token, given OCR tokens and a page.
string GetModelSummary()
void TrainDocument(Tensor<T> packedTokens, Tensor<T> pageImage, IReadOnlyList<int> targetTokens) One teacher-forced training step on target token ids, which should end with EOS (id 1).
void ValidateInputShape(Tensor<T> documentImage)