Pix2Struct<T>
Models & Types · AiDotNet.Document.PixelToSequence
Pix2Struct neural network for screenshot to structured output conversion.
Inheritance: NeuralNetworkBase<T> → DocumentNeuralNetworkBase<T> → Pix2Struct<T>
Implements: IDocumentQA<T>
For Beginners
Pix2Struct can:
- Parse screenshots of web pages, charts, and documents
- Extract structured information (tables, code, data)
- Answer questions about visual content
- Handle images of different sizes without resizing
Example usage:
How It Works
Pix2Struct is a visually-situated language model that learns to parse screenshots into structured outputs. It uses a variable-resolution vision encoder with ViT and a text decoder to generate structured text from images.
Reference: “Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding” (ICML 2023)
https://arxiv.org/abs/2210.03347
Constructors
| Constructor | Summary |
|---|---|
Pix2Struct(NeuralNetworkArchitecture<T> architecture, Pix2StructOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a Pix2Struct model using native layers for training and inference. |
Pix2Struct(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, ITokenizer tokenizer, Pix2StructOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a Pix2Struct model using a pre-trained ONNX model for inference. |
Properties
| Property | Summary |
|---|---|
DocumentType SupportedDocumentTypes { get; } |
|
bool RequiresOCR { get; } |
|
int ExpectedImageSize { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, DocumentQAResult<T>> ExtractFields(Tensor<T> documentImage, IEnumerable<string> fieldPrompts) |
|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question) |
|
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question, int maxAnswerLength, double temperature) |
|
IEnumerable<DocumentQAResult<T>> AnswerQuestions(Tensor<T> documentImage, IEnumerable<string> questions) |
|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeDocument(Tensor<T> documentImage) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
string GetModelSummary() |
|
string ParseScreenshot(Tensor<T> screenshotImage, string prompt) |
Parses a screenshot image and generates structured output. |
void ValidateInputShape(Tensor<T> documentImage) |