DocPedia<T>

Models & Types · AiDotNet.VisionLanguage.Document

DocPedia: frequency-domain document understanding model.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → DocPedia<T>

Implements: IDocumentUnderstandingModel<T>

For Beginners

DocPedia is a document understanding model that processes documents in the frequency domain for text and layout analysis. Default values follow the original paper settings.

How It Works

DocPedia (2024) unleashes the power of large multimodal models for document understanding by operating in the frequency domain. It converts document images into frequency representations via DCT transforms, enabling the model to capture both structural layout patterns and fine text details without requiring explicit OCR preprocessing.

References:

  • Paper: “DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain” (2024)

Example

// Create a DocPedia model for frequency-domain document understanding
// with DCT-based visual encoding for text and layout analysis
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new DocPedia<double>(architecture, "docpedia.onnx");
            
// Training mode with native layers
var trainModel = new DocPedia<double>(architecture, new DocPediaOptions());

Constructors

Constructor Summary
DocPedia(NeuralNetworkArchitecture<T> architecture, DocPediaOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
DocPedia(NeuralNetworkArchitecture<T> architecture, string modelPath, DocPediaOptions options)

Properties

Property Summary
bool IsOcrFree { get; }
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> AnswerDocumentQuestion(Tensor<T> documentImage, string question)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> ExtractText(Tensor<T> documentImage)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text from a document image using DocPedia’s frequency-domain pipeline.