Nougat<T>

Models & Types · AiDotNet.Document.PixelToSequence

Nougat neural network for academic document understanding.

Inheritance: NeuralNetworkBase<T> → DocumentNeuralNetworkBase<T> → Nougat<T>

Implements: IDocumentQA<T>

For Beginners

Nougat excels at:

  1. Converting PDF pages to Markdown
  2. Preserving mathematical equations (LaTeX)
  3. Understanding document structure (sections, tables, figures)
  4. Processing scientific papers without OCR

Example usage:

How It Works

Nougat (Neural Optical Understanding for Academic Documents) is an OCR-free transformer model specifically designed to parse academic papers from PDF images into structured Markdown format with mathematical notation support.

Reference: “Nougat: Neural Optical Understanding for Academic Documents” (arXiv 2023) https://arxiv.org/abs/2308.13418

Constructors

Constructor Summary
Nougat(NeuralNetworkArchitecture<T> architecture, NougatOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a Nougat model using native layers for training and inference.
Nougat(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, ITokenizer tokenizer, NougatOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a Nougat model using a pre-trained ONNX model for inference.

Properties

Property Summary
DocumentType SupportedDocumentTypes { get; }
bool RequiresOCR { get; }
bool SupportsLatex { get; } Gets whether this model supports LaTeX equation output.
int ExpectedImageSize { get; }

Methods

Method Summary
Dictionary<string, DocumentQAResult<T>> ExtractFields(Tensor<T> documentImage, IEnumerable<string> fieldPrompts)
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question)
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question, int maxAnswerLength, double temperature)
IEnumerable<DocumentQAResult<T>> AnswerQuestions(Tensor<T> documentImage, IEnumerable<string> questions)
IEnumerable<string> ExtractEquations(Tensor<T> pageImage) Parses a PDF page and extracts equations in LaTeX format.
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeDocument(Tensor<T> documentImage)
string GetModelSummary()
string ParseAcademicDocument(Tensor<T> pageImage) Parses an academic document page and generates Markdown output.
void ValidateInputShape(Tensor<T> documentImage)