Fuyu<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

Fuyu: no vision encoder; raw patches directly into transformer.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Fuyu<T>

Implements: IInstructionTunedVLM<T>

For Beginners

Fuyu takes the simplest possible approach to combining vision and language: instead of using a separate vision encoder like CLIP or ViT, it feeds raw image patches directly into the transformer decoder. Each image patch is simply projected to the model’s hidden dimension with a linear layer and treated like a text token. This means the same transformer processes both image and text tokens, making the architecture extremely simple. The trade-off is that it needs more compute to process images compared to models with dedicated vision encoders, but it avoids any information loss from pre-processing. Default values follow the original paper settings.

How It Works

Fuyu (Adept, 2023) takes a radically simple approach to multimodal processing by feeding raw image patches directly into the transformer decoder without a separate vision encoder. Patches are linearly projected to the decoder dimension and interleaved with text tokens.

References:

  • Paper: “Fuyu-8B: A Multimodal Architecture for AI Agents” (Adept, 2023)

Example

// Create a Fuyu model for encoder-free multimodal processing
// feeding raw image patches directly into the transformer
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new Fuyu<double>(architecture, "fuyu.onnx");
            
// Training mode with native layers
var trainModel = new Fuyu<double>(architecture, new FuyuOptions());

Constructors

Constructor Summary
Fuyu(NeuralNetworkArchitecture<T> architecture, FuyuOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Fuyu(NeuralNetworkArchitecture<T> architecture, string modelPath, FuyuOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using Fuyu’s direct patch-to-transformer architecture.