Gpt4VisionNeuralNetwork<T>
Models & Types · AiDotNet.NeuralNetworks
GPT-4V-style neural network that combines vision understanding with large language model capabilities.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → Gpt4VisionNeuralNetwork<T>
Implements: IGpt4VisionModel<T>
For Beginners
GPT-4 Vision combines a visual understanding system with a language model, letting you ask questions about images and receive natural language answers. It first processes an image through a vision encoder (extracting features like objects, text, and spatial relationships), then feeds those features into a language model that generates human-readable responses. This enables tasks like describing images, reading text in photos, and answering visual questions.
How It Works
This implementation provides a vision-language model that can understand images and generate text responses, similar to GPT-4V, LLaVA, or other vision-language models.
Architecture Overview:
- Vision Encoder: ViT-based encoder to extract visual features
- Vision-Language Projector: Maps visual features to LLM embedding space
- Language Model: Transformer decoder for text generation
- Multi-modal Attention: Allows text to attend to visual features
Example
var options = new Gpt4VisionOptions { };
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 336, 336 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new Gpt4VisionNeuralNetwork<float>(
architecture,
visionEncoderPath: "vision_encoder.onnx", languageModelPath: "language_model.onnx",
tokenizer: CharacterTokenizer.CreateAscii()))
.Build(trainX, trainY);
var output = result.Predict(image);
Constructors
| Constructor | Summary |
|---|---|
Gpt4VisionNeuralNetwork(NeuralNetworkArchitecture<T> architecture, ITokenizer tokenizer, Gpt4VisionOptions options, ILossFunction<T> lossFunction) |
Creates a GPT-4 Vision network using native layers (for training or when ONNX is not available). |
Gpt4VisionNeuralNetwork(NeuralNetworkArchitecture<T> architecture, string visionEncoderPath, string languageModelPath, ITokenizer tokenizer, Gpt4VisionOptions options, ILossFunction<T> lossFunction) |
Creates a GPT-4 Vision network using pretrained ONNX models. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedDetailLevels { get; } |
|
ValueTuple<int, int> MaxImageResolution { get; } |
|
int ContextWindowSize { get; } |
|
int EmbeddingDimension { get; } |
|
int ImageEmbeddingDimension { get; } |
|
int ImageSize { get; } |
|
int MaxImagesPerRequest { get; } |
|
int MaxSequenceLength { get; } |
|
int TextEmbeddingDimension { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, IEnumerable<string> labels) |
|
Dictionary<string, T> ZeroShotClassify(double[] imageData, IEnumerable<string> labels) |
|
Dictionary<string, ValueTuple<bool, T>> SafetyCheck(Tensor<T> image) |
|
IEnumerable<ValueTuple<int, T>> RetrieveImages(string query, IEnumerable<Tensor<T>> images, int topK) |
|
IEnumerable<ValueTuple<int, T>> RetrieveTexts(Tensor<T> image, IEnumerable<string> texts, int topK) |
|
IEnumerable<ValueTuple<string, T, int, int, int, int>> DetectObjects(Tensor<T> image, string objectQuery) |
|
IEnumerable<Vector<T>> GetImageEmbeddings(IEnumerable<Tensor<T>> images) |
|
IEnumerable<Vector<T>> GetTextEmbeddings(IEnumerable<string> texts) |
|
Matrix<T> EncodeImageBatch(IEnumerable<double[]> imageDataBatch) |
|
Matrix<T> EncodeTextBatch(IEnumerable<string> texts) |
|
ModelMetadata<T> GetModelMetadata() |
|
T ComputeSimilarity(Tensor<T> image, string text) |
|
T ComputeSimilarity(Vector<T> textEmbedding, Vector<T> imageEmbedding) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> GetAttentionMap(Tensor<T> image, string prompt) |
|
ValueTuple<Dictionary<string, T>, IEnumerable<string>> EvaluateImageQuality(Tensor<T> image) |
|
ValueTuple<string, Dictionary<string, object>, string> AnalyzeChart(Tensor<T> chartImage) |
|
ValueTuple<string, Dictionary<string, object>> ExtractText(Tensor<T> image, bool preserveLayout) |
|
ValueTuple<string, T> AnswerVisualQuestion(Tensor<T> image, string question) |
|
ValueTuple<string, string> VisualReasoning(Tensor<T> image, string reasoningTask, string question) |
|
Vector<T> EncodeImage(double[] imageData) |
|
Vector<T> EncodeText(string text) |
|
Vector<T> GetImageEmbedding(Tensor<T> image) |
|
Vector<T> GetTextEmbedding(string text) |
|
string AnalyzeDocument(Tensor<T> documentImage, string analysisType, string additionalPrompt) |
|
string Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage, int maxTokens) |
|
string CompareImages(Tensor<T> image1, Tensor<T> image2, string comparisonType) |
|
string DescribeImage(Tensor<T> image, string style, string detailLevel) |
|
string ExtractStructuredData(Tensor<T> image, string schema) |
|
string Generate(Tensor<T> image, string prompt, int maxTokens, double temperature) |
|
string GenerateCodeFromUI(Tensor<T> uiScreenshot, string targetFramework, string additionalInstructions) |
|
string GenerateEditInstructions(Tensor<T> image, string editRequest) |
|
string GenerateFromMultipleImages(IEnumerable<Tensor<T>> images, string prompt, int maxTokens, double temperature) |
|
string GenerateStory(Tensor<T> image, string genre, string length) |