UnifiedMultimodalNetwork<T>

Models & Types · AiDotNet.NeuralNetworks

Unified multimodal network that handles text, images, audio, and video in a single architecture with cross-modal attention and any-to-any generation.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → UnifiedMultimodalNetwork<T>

Implements: IUnifiedMultimodalModel<T>

For Beginners

This network handles text, images, audio, and video all in one architecture. Instead of needing separate models for each type of data, it processes all modalities through shared transformer layers with cross-modal attention. It can both understand any combination of inputs and generate outputs in any modality, enabling tasks like describing a video, generating images from text, or answering questions about audio clips.

Example

var image = Tensor<float>.CreateRandom(new[] { 1, 3, 224, 224 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new UnifiedMultimodalNetwork<float>())
    .Build(trainX, trainY);
var output = result.Predict(image);

Constructors

Constructor Summary
UnifiedMultimodalNetwork()
UnifiedMultimodalNetwork(NeuralNetworkArchitecture<T> architecture, UnifiedMultimodalNetworkOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, int? seed) Initializes a new instance of the UnifiedMultimodalNetwork.

Properties

Property Summary
IReadOnlyList<ModalityType> SupportedInputModalities { get; }
IReadOnlyList<ModalityType> SupportedOutputModalities { get; }
bool SupportsStreaming { get; }
int EmbeddingDimension { get; }
int MaxSequenceLength { get; }

Methods

Method Summary
Dictionary<ModalityType, ValueTuple<bool, IEnumerable<string>>> SafetyCheck(IEnumerable<MultimodalInput<T>> inputs)
IEnumerable<MultimodalOutput<T>> GenerateInterleaved(IEnumerable<MultimodalInput<T>> inputs, IEnumerable<ValueTuple<ModalityType, int>> outputSpec)
IEnumerable<ValueTuple<int, T, ModalityType>> Retrieve(MultimodalInput<T> query, IEnumerable<MultimodalInput<T>> database, int topK)
IEnumerable<ValueTuple<string, T, ModalityType, object>> Detect(IEnumerable<MultimodalInput<T>> inputs, string targetDescription)
Matrix<T> AlignTemporally(IEnumerable<MultimodalInput<T>> inputs)
Matrix<T> EncodeSequence(IEnumerable<MultimodalInput<T>> inputs)
ModelMetadata<T> GetModelMetadata()
MultimodalOutput<T> Edit(MultimodalInput<T> original, string editInstructions)
MultimodalOutput<T> FewShotLearn(IEnumerable<ValueTuple<IEnumerable<MultimodalInput<T>>, MultimodalOutput<T>>> examples, IEnumerable<MultimodalInput<T>> query)
MultimodalOutput<T> Generate(IEnumerable<MultimodalInput<T>> inputs, ModalityType outputModality, int maxLength)
MultimodalOutput<T> Summarize(IEnumerable<MultimodalInput<T>> inputs, ModalityType outputModality, int maxLength)
MultimodalOutput<T> Translate(MultimodalInput<T> input, ModalityType targetModality)
T ComputeSimilarity(MultimodalInput<T> input1, MultimodalInput<T> input2)
Tensor<T> ForwardForTraining(Tensor<T> input) Training-mode forward — same architecture as Predict but routes every reshape through Reshape so the gradient tape stays connected end-to-end.
Tensor<T> GenerateAudio(IEnumerable<MultimodalInput<T>> inputs, double durationSeconds, int sampleRate)
Tensor<T> GenerateImage(IEnumerable<MultimodalInput<T>> inputs, int width, int height)
Tensor<T> GetCrossModalAttention(IEnumerable<MultimodalInput<T>> inputs)
ValueTuple<string, Dictionary<string, IEnumerable<T>>> Compare(IEnumerable<MultimodalInput<T>> inputs, IEnumerable<string> comparisonCriteria)
ValueTuple<string, IEnumerable<string>> Reason(IEnumerable<MultimodalInput<T>> inputs, string task)
ValueTuple<string, T> AnswerQuestion(IEnumerable<MultimodalInput<T>> context, string question)
Vector<T> Encode(MultimodalInput<T> input)
Vector<T> Fuse(IEnumerable<MultimodalInput<T>> inputs, string fusionStrategy)
string Chat(IEnumerable<ValueTuple<string, IEnumerable<MultimodalInput<T>>>> conversationHistory, IEnumerable<MultimodalInput<T>> newInputs, int maxTokens)
string GenerateText(IEnumerable<MultimodalInput<T>> inputs, string prompt, int maxTokens, double temperature)