Pix2Struct<T>

Models & Types · AiDotNet.Document.PixelToSequence

Pix2Struct neural network for screenshot to structured output conversion.

Inheritance: NeuralNetworkBase<T> → DocumentNeuralNetworkBase<T> → Pix2Struct<T>

Implements: IDocumentQA<T>

For Beginners

Pix2Struct can:

  1. Parse screenshots of web pages, charts, and documents
  2. Extract structured information (tables, code, data)
  3. Answer questions about visual content
  4. Handle images of different sizes without resizing

Example usage:

How It Works

Pix2Struct is a visually-situated language model that learns to parse screenshots into structured outputs. It uses a variable-resolution vision encoder with ViT and a text decoder to generate structured text from images.

Reference: “Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding” (ICML 2023) https://arxiv.org/abs/2210.03347

Constructors

Constructor Summary
Pix2Struct(NeuralNetworkArchitecture<T> architecture, Pix2StructOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a Pix2Struct model using native layers for training and inference.
Pix2Struct(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, ITokenizer tokenizer, Pix2StructOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a Pix2Struct model using a pre-trained ONNX model for inference.

Properties

Property Summary
DocumentType SupportedDocumentTypes { get; }
bool RequiresOCR { get; }
int ExpectedImageSize { get; }

Methods

Method Summary
Dictionary<string, DocumentQAResult<T>> ExtractFields(Tensor<T> documentImage, IEnumerable<string> fieldPrompts)
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question)
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question, int maxAnswerLength, double temperature)
IEnumerable<DocumentQAResult<T>> AnswerQuestions(Tensor<T> documentImage, IEnumerable<string> questions)
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeDocument(Tensor<T> documentImage)
Tensor<T> ForwardForTraining(Tensor<T> input)
string GetModelSummary()
string ParseScreenshot(Tensor<T> screenshotImage, string prompt) Parses a screenshot image and generates structured output.
void ValidateInputShape(Tensor<T> documentImage)