PICK<T>

Models & Types · AiDotNet.Document.GraphBased

PICK (Processing Key Information extraction) for document key information extraction.

Inheritance: NeuralNetworkBase<T> → DocumentNeuralNetworkBase<T> → PICK<T>

Implements: IFormUnderstanding<T>

For Beginners

Give PICK the text boxes an OCR engine found on an invoice or receipt, optionally with the page image. It labels every word (for example “TOTAL” or “DATE”) using both what the words say and where they sit relative to each other. Pass an IOCRModel to the constructor to run straight from page images.

How It Works

PICK treats each OCR text segment as a graph node. It encodes each segment and learns how the segments relate, and a CRF then tags every token (Yu et al., ICPR 2020; reference wenwenyu/PICK-pytorch):

  • Encoder: word embeddings plus a sinusoidal position embedding, plus the segment’s image embedding

(the ResNet page features ROI-aligned 7x7 inside the segment’s box, then a 7x7 conv, BatchNorm and ReLU). A transformer runs over each segment, and each segment is mean-pooled into its node.

  • Graph: PickGraphLayer learns a soft adjacency from the nodes (with its

graph-learning loss) and convolves over node-edge-node triplets seeded by six box-relation features.

  • Decoder: every token’s features plus its segment’s graph embedding, joined into one document

sequence, then a BiLSTM, a linear layer and a CRF.

  • Training minimizes the CRF negative log-likelihood plus 0.01 x the graph-learning loss.

Input.Predict takes a packed segment tensor [N, 4 + T]. Each row is one text segment: its box (x0, y0, x1, y1) in page pixels, then up to T token ids, padded with

  1. The output is the per-slot CRF emissions [N * T, NumEntityTypes] (zeros for padding), as the reference forward

returns its logits. PredictDocument also accepts the page image, which drives the image branch (without one, the image embedding is zero), and returns the CRF’s Viterbi tags one-hot per real token.

Reference: “PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networks” (ICPR 2020) https://arxiv.org/abs/2004.07464

Constructors

Constructor Summary
PICK(NeuralNetworkArchitecture<T> architecture, PICKOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, IOCRModel<T> ocr) Creates a trainable native PICK model.
PICK(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, ITokenizer tokenizer, PICKOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, IOCRModel<T> ocr) Creates a PICK model backed by an ONNX export.

Properties

Property Summary
DocumentType SupportedDocumentTypes { get; }
IReadOnlyList<string> SupportedEntityTypes { get; } Gets the supported entity types for extraction.
bool RequiresOCR { get; }
int ExpectedImageSize { get; }

Methods

Method Summary
Dictionary<string, string> ExtractKeyValuePairs(Tensor<T> documentImage)
FormFieldResult<T> ExtractFormFields(Tensor<T> documentImage)
FormFieldResult<T> ExtractFormFields(Tensor<T> documentImage, double confidenceThreshold)
IEnumerable<CheckboxResult<T>> DetectCheckboxes(Tensor<T> documentImage)
IEnumerable<SignatureResult<T>> DetectSignatures(Tensor<T> documentImage)
KeyInfoExtractionResult<T> ExtractKeyInfo(Tensor<T> documentImage) Extracts key information entities from a page image (requires an IOCRModel).
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeDocument(Tensor<T> documentImage)
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> PredictDocument(Tensor<T> segments, Tensor<T> pageImage) Tags a document: packed segments [N, 4 + T] and an optional page image [3, H, W].
string GetModelSummary()
void TrainDocument(Tensor<T> segments, Tensor<T> pageImage, Tensor<T> tags) Trains on packed segments with an optional page image and per-slot tags (see Train).
void ValidateInputShape(Tensor<T> documentImage)