Dragonfly<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

Dragonfly: multi-resolution visual encoding VLM for fine-grained understanding.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Dragonfly<T>

Implements: IInstructionTunedVLM<T>

For Beginners

Dragonfly is a vision-language model that looks at images at multiple zoom levels — like how you might first glance at a whole photo, then zoom into interesting parts for more detail. Its multi-resolution zoom module automatically identifies important regions and processes them at higher resolution while keeping the overall scene context at lower resolution. This approach is more efficient than processing the entire image at high resolution, while still capturing fine details where they matter. Default values follow the original paper settings.

How It Works

Dragonfly (Together AI, 2024) introduces multi-resolution visual encoding that processes images at multiple zoom levels simultaneously. It uses a “zoom-in” module that identifies important regions of the image and processes them at higher resolution, while maintaining a global view at lower resolution. This multi-resolution approach gives the model both broad scene understanding and fine-grained detail recognition without requiring extremely high-resolution processing of the entire image.

References:

  • Paper: “Dragonfly: Multi-Resolution Zoom Supercharges Large Visual-Language Model” (Together AI, 2024)

Example

// Create a Dragonfly model for multi-resolution visual understanding
// using zoom-in modules for fine-grained detail recognition
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new Dragonfly<double>(architecture, "dragonfly.onnx");
            
// Training mode with native layers
var trainModel = new Dragonfly<double>(architecture, new DragonflyOptions());

Constructors

Constructor Summary
Dragonfly(NeuralNetworkArchitecture<T> architecture, DragonflyOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Dragonfly(NeuralNetworkArchitecture<T> architecture, string modelPath, DragonflyOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using Dragonfly’s multi-resolution visual encoding with zoom-and-select.