GOTOCR2<T>

Models & Types · AiDotNet.VisionLanguage.Document

GOT-OCR2: 580M unified OCR model for text, tables, charts, equations, and music scores.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → GOTOCR2<T>

Implements: IDocumentUnderstandingModel<T>

For Beginners

GOT-OCR2 is a unified OCR model that handles text, tables, charts, equations, and music scores. Default values follow the original paper settings.

How It Works

GOT-OCR2 (StepFun, 2024) is a 580M unified end-to-end OCR model that handles diverse visual text including plain text, tables, charts, mathematical equations, and music scores. A SAM ViTDet-B encoder reads a 1024 px page. Two stride-2 convolutions reduce it to 256 image tokens, which fill the image slots of a ChatML prompt for the Qwen-0.5B decoder. The decoder then generates the text directly, with no separate detection or recognition stage.

References:

  • Paper: “General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model” (StepFun, 2024)

Example

// Create a GOT-OCR2 model for unified end-to-end OCR
// handling text, tables, charts, equations, and music scores
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.TextGeneration,
    inputHeight: 1024, inputWidth: 1024, inputDepth: 3, outputSize: 151860);
            
// ONNX inference mode with pre-trained model
var model = new GOTOCR2<double>(architecture, "gotocr2.onnx");
            
// Native SAM ViTDet + Qwen model; read a page
var trainModel = new GOTOCR2<double>(architecture, new GOTOCR2Options());
var pageImage = Tensor<double>.CreateRandom(new[] { 1, 3, 1024, 1024 });
string text = trainModel.ReadText(pageImage);

Constructors

Constructor Summary
GOTOCR2(NeuralNetworkArchitecture<T> architecture, GOTOCR2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ITokenizer tokenizer) Creates a trainable GOT-OCR2.
GOTOCR2(NeuralNetworkArchitecture<T> architecture, string modelPath, GOTOCR2Options options, ITokenizer tokenizer) Creates a trainable GOT-OCR2.

Properties

Property Summary
bool IsOcrFree { get; }
int DecoderEmbeddingDim { get; } The decoder width, which the projected image tokens share.
int EmbeddingDimension { get; }
int ImageTokenCount { get; } Number of image tokens: the patch grid reduced 4x per side by the projector.
int MaxGenerationLength { get; } Most tokens one generation produces.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> AnswerDocumentQuestion(Tensor<T> documentImage, string question) GOT-OCR2 is trained for reading, not question answering.
Tensor<T> EncodeImage(Tensor<T> image) The page’s projected image tokens [ImageTokenCount, DecoderDim].
Tensor<T> ExtractText(Tensor<T> documentImage) Reads the page under GOT-OCR2’s OCR query and returns the generated token ids.
Tensor<T> ForwardForTraining(Tensor<T> input) The training-mode counterpart of PredictCore: next-token logits [1, vocab] at the end of the OCR prompt for the preprocessed page.
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Greedily reads the page and returns the generated token ids, ending with <|im_end|> when one was produced.
Tensor<T> PredictTokens(Tensor<T> image, IReadOnlyList<int> answer, string prompt) Teacher-forced logits [T, vocab].
string ReadText(Tensor<T> documentImage, string prompt) Reads the page and decodes the answer to text.
void TrainOcr(Tensor<T> image, IReadOnlyList<int> answerIds, string prompt) One teacher-forced step that teaches the model to read image as answerIds.