PICK<T>
Models & Types · AiDotNet.Document.GraphBased
PICK (Processing Key Information extraction) for document key information extraction.
Inheritance: NeuralNetworkBase<T> → DocumentNeuralNetworkBase<T> → PICK<T>
Implements: IFormUnderstanding<T>
For Beginners
Give PICK the text boxes an OCR engine found on an invoice or receipt, optionally
with the page image. It labels every word (for example “TOTAL” or “DATE”) using both what the words say
and where they sit relative to each other. Pass an IOCRModel to the constructor to run
straight from page images.
How It Works
PICK treats each OCR text segment as a graph node. It encodes each segment and learns how the segments relate, and a CRF then tags every token (Yu et al., ICPR 2020; reference wenwenyu/PICK-pytorch):
- Encoder: word embeddings plus a sinusoidal position embedding, plus the segment’s image embedding
(the ResNet page features ROI-aligned 7x7 inside the segment’s box, then a 7x7 conv, BatchNorm and ReLU). A transformer runs over each segment, and each segment is mean-pooled into its node.
- Graph:
PickGraphLayerlearns a soft adjacency from the nodes (with its
graph-learning loss) and convolves over node-edge-node triplets seeded by six box-relation features.
- Decoder: every token’s features plus its segment’s graph embedding, joined into one document
sequence, then a BiLSTM, a linear layer and a CRF.
- Training minimizes the CRF negative log-likelihood plus 0.01 x the graph-learning loss.
Input.Predict takes a packed segment tensor [N, 4 + T]. Each row is
one text segment: its box (x0, y0, x1, y1) in page pixels, then up to T token ids, padded with
- The output is the per-slot CRF emissions
[N * T, NumEntityTypes](zeros for padding), as the reference forward
returns its logits. PredictDocument also accepts the page image, which drives the image branch (without
one, the image embedding is zero), and returns the CRF’s Viterbi tags one-hot per real token.
Reference: “PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networks” (ICPR 2020)
https://arxiv.org/abs/2004.07464
Constructors
| Constructor | Summary |
|---|---|
PICK(NeuralNetworkArchitecture<T> architecture, PICKOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, IOCRModel<T> ocr) |
Creates a trainable native PICK model. |
PICK(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, ITokenizer tokenizer, PICKOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, IOCRModel<T> ocr) |
Creates a PICK model backed by an ONNX export. |
Properties
| Property | Summary |
|---|---|
DocumentType SupportedDocumentTypes { get; } |
|
IReadOnlyList<string> SupportedEntityTypes { get; } |
Gets the supported entity types for extraction. |
bool RequiresOCR { get; } |
|
int ExpectedImageSize { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> ExtractKeyValuePairs(Tensor<T> documentImage) |
|
FormFieldResult<T> ExtractFormFields(Tensor<T> documentImage) |
|
FormFieldResult<T> ExtractFormFields(Tensor<T> documentImage, double confidenceThreshold) |
|
IEnumerable<CheckboxResult<T>> DetectCheckboxes(Tensor<T> documentImage) |
|
IEnumerable<SignatureResult<T>> DetectSignatures(Tensor<T> documentImage) |
|
KeyInfoExtractionResult<T> ExtractKeyInfo(Tensor<T> documentImage) |
Extracts key information entities from a page image (requires an IOCRModel). |
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeDocument(Tensor<T> documentImage) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> PredictDocument(Tensor<T> segments, Tensor<T> pageImage) |
Tags a document: packed segments [N, 4 + T] and an optional page image [3, H, W]. |
string GetModelSummary() |
|
void TrainDocument(Tensor<T> segments, Tensor<T> pageImage, Tensor<T> tags) |
Trains on packed segments with an optional page image and per-slot tags (see Train). |
void ValidateInputShape(Tensor<T> documentImage) |