LayoutXLM<T>

Models & Types · AiDotNet.Document.LayoutAware

LayoutXLM neural network for multilingual document understanding.

Inheritance: NeuralNetworkBase<T> → DocumentNeuralNetworkBase<T> → LayoutXLM<T>

Implements: IDocumentQA<T>, ILayoutDetector<T>

For Beginners

LayoutXLM understands documents in many languages:

  1. Supports 53 languages out-of-the-box
  2. Can handle mixed-language documents
  3. Zero-shot cross-lingual transfer (train on one language, test on another)

Key features:

  • XLM-RoBERTa multilingual text encoder
  • Visual backbone (ResNeXt-FPN) for image features
  • Language-agnostic layout understanding
  • Pre-trained on XFUND dataset (7 languages)

Example usage:

How It Works

LayoutXLM extends LayoutLMv2 to support multilingual documents by using XLM-RoBERTa as the text backbone and training on documents from multiple languages.

Reference: “LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding” (ACL 2022) https://arxiv.org/abs/2104.08836

Constructors

Constructor Summary
LayoutXLM(NeuralNetworkArchitecture<T> architecture, LayoutXLMOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a LayoutXLM model using native layers for training and inference.
LayoutXLM(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, ITokenizer tokenizer, LayoutXLMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a LayoutXLM model using a pre-trained ONNX model for inference.

Properties

Property Summary
DocumentType SupportedDocumentTypes { get; }
IReadOnlyList<LayoutElementType> SupportedElementTypes { get; }
bool RequiresOCR { get; }
int ExpectedImageSize { get; }
int NumLanguages { get; } Gets the number of languages supported.

Methods

Method Summary
Dictionary<string, DocumentQAResult<T>> ExtractFields(Tensor<T> documentImage, IEnumerable<string> fieldPrompts)
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
DocumentLayoutResult<T> DetectLayout(Tensor<T> documentImage)
DocumentLayoutResult<T> DetectLayout(Tensor<T> documentImage, double confidenceThreshold)
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question)
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question, int maxAnswerLength, double temperature)
IEnumerable<DocumentQAResult<T>> AnswerQuestions(Tensor<T> documentImage, IEnumerable<string> questions)
LayerInputDomain GetInputDomain(int[] inputShape) Generated rank-routed external input-domain contract.
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeDocument(Tensor<T> documentImage)
Tensor<T> EncodeMultimodal(Tensor<T> textTokens, Tensor<T> documentImage) Full text+image fusion entry (industry-standard LayoutXLM): encodes BOTH a token-ID sequence and a document image through the two visual/text streams and fuses them via the transformer.
Tensor<T> ForwardForTraining(Tensor<T> input)
string GetModelSummary()
void ValidateInputShape(Tensor<T> documentImage)