Florence2<T>

Models & Types · AiDotNet.VisionLanguage.Encoders

Florence-2 unified vision foundation model for captioning, detection, grounding, and OCR.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Florence2<T>

Implements: IVisualEncoder<T>

For Beginners

Florence-2 from Microsoft is a lightweight vision model (0.23B-0.77B parameters) that handles many tasks through text prompts — captioning, object detection, grounding, OCR, and segmentation — all in a single unified model. It uses DaViT (Dual Attention ViT) as its vision encoder and generates structured text output for each task. Default values follow the original paper settings.

How It Works

Florence-2 (Xiao et al., Microsoft 2024) is a lightweight sequence-to-sequence vision model (0.23B-0.77B) handling multiple tasks through a unified prompt-based approach. It uses DaViT (Dual Attention ViT) as the vision encoder and a multi-task decoder for captioning, detection, grounding, OCR, and segmentation.

References:

  • Paper: “Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks” (Xiao et al., 2024)

Example

// Create a Florence-2 model for unified multi-task vision understanding
// handling captioning, detection, grounding, OCR, and segmentation
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.TextGeneration,
    inputHeight: 768, inputWidth: 768, inputDepth: 3, outputSize: 51289);
            
// ONNX inference mode with pre-trained model
var model = new Florence2<double>(architecture, "florence2.onnx");
            
// Native DaViT + BART model; read the text on a page
var trainModel = new Florence2<double>(architecture, new Florence2Options());
var pageImage = Tensor<double>.CreateRandom(new[] { 1, 3, 768, 768 });
var answer = trainModel.RunTask(pageImage, Florence2Task.Ocr);

Constructors

Constructor Summary
Florence2(NeuralNetworkArchitecture<T> architecture, Florence2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ITokenizer tokenizer) Creates a trainable Florence-2.
Florence2(NeuralNetworkArchitecture<T> architecture, string modelPath, Florence2Options options, ITokenizer tokenizer) Creates a trainable Florence-2.

Properties

Property Summary
int EmbeddingDimension { get; }
int ImageTokenCount { get; } Number of image tokens spliced into the encoder: the mean token plus one per stride-32 position.

Methods

Method Summary
Florence2TaskResult RunTask(Tensor<T> image, Florence2Task task, string taskInput) Runs task and parses the answer.
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image) Encodes an image into its projected image tokens [1 + h*w, EmbeddingDim]: the vision half of the model, before the BART encoder.
Tensor<T> GenerateFromImage(Tensor<T> image, Florence2Task? task, string taskInput) Greedily generates the answer to task and returns its token ids, ending with EOS when one was produced within MaxOutputTokens.
Tensor<T> PredictTokens(Tensor<T> image, IReadOnlyList<int> decoderInput, Florence2Task? task, string taskInput) Teacher-forced decoder logits [T, vocab] for decoderInput under task’s prompt.
void TrainCaption(Tensor<T> image, IReadOnlyList<int> targetIds, Florence2Task? task, string taskInput) One teacher-forced step that teaches the model to answer task with targetIds.