Gpt4VisionNeuralNetwork<T>

Models & Types · AiDotNet.NeuralNetworks

GPT-4V-style neural network that combines vision understanding with large language model capabilities.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → Gpt4VisionNeuralNetwork<T>

Implements: IGpt4VisionModel<T>

For Beginners

GPT-4 Vision combines a visual understanding system with a language model, letting you ask questions about images and receive natural language answers. It first processes an image through a vision encoder (extracting features like objects, text, and spatial relationships), then feeds those features into a language model that generates human-readable responses. This enables tasks like describing images, reading text in photos, and answering visual questions.

How It Works

This implementation provides a vision-language model that can understand images and generate text responses, similar to GPT-4V, LLaVA, or other vision-language models.

Architecture Overview:

  1. Vision Encoder: ViT-based encoder to extract visual features
  2. Vision-Language Projector: Maps visual features to LLM embedding space
  3. Language Model: Transformer decoder for text generation
  4. Multi-modal Attention: Allows text to attend to visual features

Example

var options = new Gpt4VisionOptions { };
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 336, 336 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new Gpt4VisionNeuralNetwork<float>(
        architecture,
        visionEncoderPath: "vision_encoder.onnx", languageModelPath: "language_model.onnx",
        tokenizer: CharacterTokenizer.CreateAscii()))
    .Build(trainX, trainY);
var output = result.Predict(image);

Constructors

Constructor Summary
Gpt4VisionNeuralNetwork(NeuralNetworkArchitecture<T> architecture, ITokenizer tokenizer, Gpt4VisionOptions options, ILossFunction<T> lossFunction) Creates a GPT-4 Vision network using native layers (for training or when ONNX is not available).
Gpt4VisionNeuralNetwork(NeuralNetworkArchitecture<T> architecture, string visionEncoderPath, string languageModelPath, ITokenizer tokenizer, Gpt4VisionOptions options, ILossFunction<T> lossFunction) Creates a GPT-4 Vision network using pretrained ONNX models.

Properties

Property Summary
IReadOnlyList<string> SupportedDetailLevels { get; }
ValueTuple<int, int> MaxImageResolution { get; }
int ContextWindowSize { get; }
int EmbeddingDimension { get; }
int ImageEmbeddingDimension { get; }
int ImageSize { get; }
int MaxImagesPerRequest { get; }
int MaxSequenceLength { get; }
int TextEmbeddingDimension { get; }

Methods

Method Summary
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, IEnumerable<string> labels)
Dictionary<string, T> ZeroShotClassify(double[] imageData, IEnumerable<string> labels)
Dictionary<string, ValueTuple<bool, T>> SafetyCheck(Tensor<T> image)
IEnumerable<ValueTuple<int, T>> RetrieveImages(string query, IEnumerable<Tensor<T>> images, int topK)
IEnumerable<ValueTuple<int, T>> RetrieveTexts(Tensor<T> image, IEnumerable<string> texts, int topK)
IEnumerable<ValueTuple<string, T, int, int, int, int>> DetectObjects(Tensor<T> image, string objectQuery)
IEnumerable<Vector<T>> GetImageEmbeddings(IEnumerable<Tensor<T>> images)
IEnumerable<Vector<T>> GetTextEmbeddings(IEnumerable<string> texts)
Matrix<T> EncodeImageBatch(IEnumerable<double[]> imageDataBatch)
Matrix<T> EncodeTextBatch(IEnumerable<string> texts)
ModelMetadata<T> GetModelMetadata()
T ComputeSimilarity(Tensor<T> image, string text)
T ComputeSimilarity(Vector<T> textEmbedding, Vector<T> imageEmbedding)
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> GetAttentionMap(Tensor<T> image, string prompt)
ValueTuple<Dictionary<string, T>, IEnumerable<string>> EvaluateImageQuality(Tensor<T> image)
ValueTuple<string, Dictionary<string, object>, string> AnalyzeChart(Tensor<T> chartImage)
ValueTuple<string, Dictionary<string, object>> ExtractText(Tensor<T> image, bool preserveLayout)
ValueTuple<string, T> AnswerVisualQuestion(Tensor<T> image, string question)
ValueTuple<string, string> VisualReasoning(Tensor<T> image, string reasoningTask, string question)
Vector<T> EncodeImage(double[] imageData)
Vector<T> EncodeText(string text)
Vector<T> GetImageEmbedding(Tensor<T> image)
Vector<T> GetTextEmbedding(string text)
string AnalyzeDocument(Tensor<T> documentImage, string analysisType, string additionalPrompt)
string Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage, int maxTokens)
string CompareImages(Tensor<T> image1, Tensor<T> image2, string comparisonType)
string DescribeImage(Tensor<T> image, string style, string detailLevel)
string ExtractStructuredData(Tensor<T> image, string schema)
string Generate(Tensor<T> image, string prompt, int maxTokens, double temperature)
string GenerateCodeFromUI(Tensor<T> uiScreenshot, string targetFramework, string additionalInstructions)
string GenerateEditInstructions(Tensor<T> image, string editRequest)
string GenerateFromMultipleImages(IEnumerable<Tensor<T>> images, string prompt, int maxTokens, double temperature)
string GenerateStory(Tensor<T> image, string genre, string length)