MATCHA<T>

Models & Types · AiDotNet.Document.PixelToSequence

MATCHA (Math-Aware Transformer for Chart Harvesting and Analysis) for chart understanding.

Inheritance: NeuralNetworkBase<T> → DocumentNeuralNetworkBase<T> → MATCHA<T>

Implements: IDocumentQA<T>, ITableExtractor<T>

For Beginners

MATCHA specializes in understanding charts:

  1. Reads bar charts, line graphs, pie charts, scatter plots
  2. Extracts underlying numerical data
  3. Answers questions about chart content
  4. Generates summaries of chart insights

Key features:

  • Math-aware pre-training for numerical reasoning
  • Pix2Struct-based architecture
  • Chart derendering (image to data table)
  • Chart QA and summarization

Example usage:

How It Works

MATCHA is designed specifically for understanding charts and plots, combining math-aware pre-training with visual encoding to extract data, answer questions, and summarize chart content.

Reference: “MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering” (ACL 2023) https://arxiv.org/abs/2212.09662

Constructors

Constructor Summary
MATCHA(NeuralNetworkArchitecture<T> architecture, MATCHAOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a MATCHA model using native layers for training and inference.
MATCHA(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, MATCHAOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a MATCHA model using a pre-trained ONNX model for inference.

Properties

Property Summary
DocumentType SupportedDocumentTypes { get; }
IReadOnlyList<string> SupportedChartTypes { get; } Gets the supported chart types.
bool RequiresOCR { get; }
bool SupportsBorderedTables { get; }
bool SupportsBorderlessTables { get; }
bool SupportsMergedCells { get; }
int ExpectedImageSize { get; }
int MaxPatchesPerImage { get; } Gets the maximum patches per image.

Methods

Method Summary
Dictionary<string, DocumentQAResult<T>> ExtractFields(Tensor<T> documentImage, IEnumerable<string> fieldPrompts)
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question)
DocumentQAResult<T> AnswerQuestion(Tensor<T> documentImage, string question, int maxAnswerLength, double temperature)
IEnumerable<DocumentQAResult<T>> AnswerQuestions(Tensor<T> documentImage, IEnumerable<string> questions)
IEnumerable<List<List<string>>> ExtractTableContent(Tensor<T> documentImage)
IEnumerable<TableRegion<T>> DetectTables(Tensor<T> documentImage)
ModelMetadata<T> GetModelMetadata()
TableStructureResult<T> RecognizeStructure(Tensor<T> tableImage)
Tensor<T> EncodeDocument(Tensor<T> documentImage)
Tensor<T> ForwardForTraining(Tensor<T> input)
string ExportTables(Tensor<T> documentImage, TableExportFormat format)
string GetModelSummary()
void ValidateInputShape(Tensor<T> documentImage)