Donut<T>

Models & Types · AiDotNet.Document.PixelToSequence

Donut (Document Understanding Transformer) - OCR-free end-to-end document understanding model.

Inheritance: NeuralNetworkBase<T> → DocumentNeuralNetworkBase<T> → Donut<T>

Implements: IDocumentQA<T>, IOCRModel<T>

For Beginners

Unlike traditional document AI which first extracts text using OCR and then processes it, Donut looks directly at the document image pixels and generates text output. This makes it:

  • Simpler: No need for a separate OCR system
  • More robust: Less affected by OCR errors
  • End-to-end trainable: Can optimize for the final task directly

Donut is excellent for:

  • Document parsing (invoices, receipts, forms)
  • Information extraction
  • Document question answering
  • Document classification

Example usage:

How It Works

Donut is an OCR-free model that directly converts document images to structured text outputs without requiring a separate OCR stage. It uses a vision encoder (Swin Transformer) and text decoder (BART) architecture.

Reference: “OCR-free Document Understanding Transformer” (ECCV 2022) https://arxiv.org/abs/2111.15664

Constructors

Constructor Summary
Donut()
Donut(NeuralNetworkArchitecture<T> architecture, DonutOptions options, ITokenizer tokenizer, int[] depths, int[] numHeads, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a Donut model using native layers for training and inference.
Donut(NeuralNetworkArchitecture<T> architecture, string encoderPath, string decoderPath, ITokenizer tokenizer, DonutOptions options, int[] depths, int[] numHeads, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a Donut model using pre-trained ONNX models for inference.

Properties

Property Summary
DocumentType SupportedDocumentTypes { get; }
IReadOnlyList<string> SupportedLanguages { get; }
bool HasUninitializedParameters { get; } A native Donut builds its layers on first use (EnsureNativeInitialized), so until then its weights are deferred rather than absent; report them the way a lazily sized layer does.
bool IsOCRFree { get; }
bool RequiresOCR { get; }
int ExpectedImageSize { get; }
int MaxGenerationLength { get; } Gets the maximum generation length for output sequences.

Methods

Method Summary
Dictionary<string, DocumentQAResult<T>> ExtractFields(Tensor<T> documentImage, IEnumerable<string> fieldPrompts)
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question)
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question, int maxAnswerLength, double temperature)
IEnumerable<DocumentQAResult<T>> AnswerQuestions(Tensor<T> documentImage, IEnumerable<string> questions)
ModelMetadata<T> GetModelMetadata()
OCRResult<T> RecognizeText(Tensor<T> documentImage)
OCRResult<T> RecognizeTextInRegion(Tensor<T> documentImage, Vector<T> region)
Tensor<T> EncodeDocument(Tensor<T> documentImage)
string GetModelSummary()
string ParseDocument(Tensor<T> documentImage, string documentType) Parses a document and returns structured output based on the document type.
void ValidateInputShape(Tensor<T> documentImage)