UDOP<T>
Models & Types · AiDotNet.Document.VisionLanguage
UDOP (Unifying Vision, Text, and Layout for Universal Document Processing) neural network.
Inheritance: NeuralNetworkBase<T> → DocumentNeuralNetworkBase<T> → UDOP<T>
Implements: IDocumentClassifier<T>, IDocumentQA<T>, ILayoutDetector<T>
For Beginners
UDOP can handle many document tasks with one model:
- Document classification
- Information extraction (NER, key-value pairs)
- Document question answering
- Document layout analysis
- Document generation
Example usage:
How It Works
UDOP is a foundation model for document AI that unifies text, image, and layout modalities within a single encoder-decoder framework. It can perform multiple document tasks through task-specific prompting.
Reference: “Unifying Vision, Text, and Layout for Universal Document Processing” (CVPR 2023)
https://arxiv.org/abs/2212.02623
Constructors
| Constructor | Summary |
|---|---|
UDOP(NeuralNetworkArchitecture<T> architecture, UDOPOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a trainable UDOP model built from options. |
UDOP(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, ITokenizer tokenizer, UDOPOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a UDOP model that runs a pretrained ONNX export. |
Properties
| Property | Summary |
|---|---|
DocumentType SupportedDocumentTypes { get; } |
|
IReadOnlyList<LayoutElementType> SupportedElementTypes { get; } |
|
IReadOnlyList<string> AvailableCategories { get; } |
|
bool RequiresOCR { get; } |
|
int ExpectedImageSize { get; } |
Side of the square page image the patch embedding expects. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, DocumentQAResult<T>> ExtractFields(Tensor<T> documentImage, IEnumerable<string> fieldPrompts) |
|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
DocumentClassificationResult<T> ClassifyDocument(Tensor<T> documentImage) |
|
DocumentClassificationResult<T> ClassifyDocument(Tensor<T> documentImage, int topK) |
Classifies by prompting with UDOP’s RVL-CDIP task prompt and scoring every category name as a decoder target. |
DocumentLayoutResult<T> DetectLayout(Tensor<T> documentImage) |
|
DocumentLayoutResult<T> DetectLayout(Tensor<T> documentImage, double confidenceThreshold) |
Layout analysis by prompting: the decoder writes a category name followed by four location tokens per region, as UDOP’s layout task does. |
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question) |
|
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question, int maxAnswerLength, double temperature) |
Answers by prompting the encoder with `“Question answering. |
IEnumerable<DocumentQAResult<T>> AnswerQuestions(Tensor<T> documentImage, IEnumerable<string> questions) |
|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> DecoderLogits(Tensor<T> packedTokens, Tensor<T> pageImage, IReadOnlyList<int> decoderInput) |
Teacher-forced decoder logits [T, vocab]: row t predicts the token after decoderInput[t]. |
Tensor<T> EncodeDocument(Tensor<T> documentImage) |
Returns the encoder’s output states for a page: one row per unclaimed image patch. |
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> GenerateTokens(Tensor<T> packedTokens, Tensor<T> pageImage, IReadOnlyList<int> promptTokens) |
Greedily decodes from the start token until EOS or MaxGenerationLength. |
Tensor<T> PredictDocument(Tensor<T> packedTokens, Tensor<T> pageImage) |
Returns the decoder’s next-token logits [1, vocab] for the start token, given OCR tokens and a page. |
string GetModelSummary() |
|
void TrainDocument(Tensor<T> packedTokens, Tensor<T> pageImage, IReadOnlyList<int> targetTokens) |
One teacher-forced training step on target token ids, which should end with EOS (id 1). |
void ValidateInputShape(Tensor<T> documentImage) |