GOTOCR2<T>
Models & Types · AiDotNet.VisionLanguage.Document
GOT-OCR2: 580M unified OCR model for text, tables, charts, equations, and music scores.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → GOTOCR2<T>
Implements: IDocumentUnderstandingModel<T>
For Beginners
GOT-OCR2 is a unified OCR model that handles text, tables, charts, equations, and music scores. Default values follow the original paper settings.
How It Works
GOT-OCR2 (StepFun, 2024) is a 580M unified end-to-end OCR model that handles diverse visual text including plain text, tables, charts, mathematical equations, and music scores. A SAM ViTDet-B encoder reads a 1024 px page. Two stride-2 convolutions reduce it to 256 image tokens, which fill the image slots of a ChatML prompt for the Qwen-0.5B decoder. The decoder then generates the text directly, with no separate detection or recognition stage.
References:
- Paper: “General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model” (StepFun, 2024)
Example
// Create a GOT-OCR2 model for unified end-to-end OCR
// handling text, tables, charts, equations, and music scores
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.TextGeneration,
inputHeight: 1024, inputWidth: 1024, inputDepth: 3, outputSize: 151860);
// ONNX inference mode with pre-trained model
var model = new GOTOCR2<double>(architecture, "gotocr2.onnx");
// Native SAM ViTDet + Qwen model; read a page
var trainModel = new GOTOCR2<double>(architecture, new GOTOCR2Options());
var pageImage = Tensor<double>.CreateRandom(new[] { 1, 3, 1024, 1024 });
string text = trainModel.ReadText(pageImage);
Constructors
| Constructor | Summary |
|---|---|
GOTOCR2(NeuralNetworkArchitecture<T> architecture, GOTOCR2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ITokenizer tokenizer) |
Creates a trainable GOT-OCR2. |
GOTOCR2(NeuralNetworkArchitecture<T> architecture, string modelPath, GOTOCR2Options options, ITokenizer tokenizer) |
Creates a trainable GOT-OCR2. |
Properties
| Property | Summary |
|---|---|
bool IsOcrFree { get; } |
|
int DecoderEmbeddingDim { get; } |
The decoder width, which the projected image tokens share. |
int EmbeddingDimension { get; } |
|
int ImageTokenCount { get; } |
Number of image tokens: the patch grid reduced 4x per side by the projector. |
int MaxGenerationLength { get; } |
Most tokens one generation produces. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> AnswerDocumentQuestion(Tensor<T> documentImage, string question) |
GOT-OCR2 is trained for reading, not question answering. |
Tensor<T> EncodeImage(Tensor<T> image) |
The page’s projected image tokens [ImageTokenCount, DecoderDim]. |
Tensor<T> ExtractText(Tensor<T> documentImage) |
Reads the page under GOT-OCR2’s OCR query and returns the generated token ids. |
Tensor<T> ForwardForTraining(Tensor<T> input) |
The training-mode counterpart of PredictCore: next-token logits [1, vocab] at the end of the OCR prompt for the preprocessed page. |
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Greedily reads the page and returns the generated token ids, ending with <|im_end|> when one was produced. |
Tensor<T> PredictTokens(Tensor<T> image, IReadOnlyList<int> answer, string prompt) |
Teacher-forced logits [T, vocab]. |
string ReadText(Tensor<T> documentImage, string prompt) |
Reads the page and decodes the answer to text. |
void TrainOcr(Tensor<T> image, IReadOnlyList<int> answerIds, string prompt) |
One teacher-forced step that teaches the model to read image as answerIds. |