DocFormer<T>

Models & Types · AiDotNet.Document.LayoutAware

DocFormer neural network for end-to-end document understanding.

Inheritance: NeuralNetworkBase<T> → DocumentNeuralNetworkBase<T> → DocFormer<T>

Implements: IDocumentClassifier<T>, ILayoutDetector<T>

For Beginners

DocFormer combines three types of information:

  1. Text content (what the words say)
  2. Visual features (what the document looks like)
  3. Spatial layout (where elements are positioned)

Unlike LayoutLM which adds position embeddings to text, DocFormer uses shared spatial encodings that align all three modalities in the same coordinate space.

Example usage:

How It Works

DocFormer is a multi-modal transformer that jointly learns text, visual, and spatial features for document understanding tasks. It uses shared spatial encodings across all modalities.

Reference: “DocFormer: End-to-End Transformer for Document Understanding” (ICCV 2021) https://arxiv.org/abs/2106.11539

Constructors

Constructor Summary
DocFormer(NeuralNetworkArchitecture<T> architecture, DocFormerOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a DocFormer model using native layers for training and inference.
DocFormer(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, ITokenizer tokenizer, DocFormerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a DocFormer model using a pre-trained ONNX model for inference.

Properties

Property Summary
DocumentType SupportedDocumentTypes { get; }
IReadOnlyList<LayoutElementType> SupportedElementTypes { get; }
IReadOnlyList<string> AvailableCategories { get; } Gets the available document classification categories.
bool RequiresOCR { get; }
int ExpectedImageSize { get; }

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
DocumentClassificationResult<T> ClassifyDocument(Tensor<T> documentImage)
DocumentClassificationResult<T> ClassifyDocument(Tensor<T> documentImage, int topK)
DocumentLayoutResult<T> DetectLayout(Tensor<T> documentImage)
DocumentLayoutResult<T> DetectLayout(Tensor<T> documentImage, double confidenceThreshold)
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeDocument(Tensor<T> documentImage)
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> PredictDocument(Tensor<T> packedTokens, Tensor<T> pageImage) Token logits [S, numClasses] for packed rows [S, 5] (token id, x0, y0, x1, y1) and the page image [3, H, W], which drives the ResNet-50 visual stream.
string GetModelSummary()
void ValidateInputShape(Tensor<T> documentImage)