Fuyu<T>
Models & Types · AiDotNet.VisionLanguage.InstructionTuned
Fuyu: no vision encoder; raw patches directly into transformer.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Fuyu<T>
Implements: IInstructionTunedVLM<T>
For Beginners
Fuyu takes the simplest possible approach to combining vision and language: instead of using a separate vision encoder like CLIP or ViT, it feeds raw image patches directly into the transformer decoder. Each image patch is simply projected to the model’s hidden dimension with a linear layer and treated like a text token. This means the same transformer processes both image and text tokens, making the architecture extremely simple. The trade-off is that it needs more compute to process images compared to models with dedicated vision encoders, but it avoids any information loss from pre-processing. Default values follow the original paper settings.
How It Works
Fuyu (Adept, 2023) takes a radically simple approach to multimodal processing by feeding raw image patches directly into the transformer decoder without a separate vision encoder. Patches are linearly projected to the decoder dimension and interleaved with text tokens.
References:
- Paper: “Fuyu-8B: A Multimodal Architecture for AI Agents” (Adept, 2023)
Example
// Create a Fuyu model for encoder-free multimodal processing
// feeding raw image patches directly into the transformer
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new Fuyu<double>(architecture, "fuyu.onnx");
// Training mode with native layers
var trainModel = new Fuyu<double>(architecture, new FuyuOptions());
Constructors
| Constructor | Summary |
|---|---|
Fuyu(NeuralNetworkArchitecture<T> architecture, FuyuOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
Fuyu(NeuralNetworkArchitecture<T> architecture, string modelPath, FuyuOptions options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
|
string LanguageModelName { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using Fuyu’s direct patch-to-transformer architecture. |