UnifiedMultimodalNetwork<T>
Models & Types · AiDotNet.NeuralNetworks
Unified multimodal network that handles text, images, audio, and video in a single architecture with cross-modal attention and any-to-any generation.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → UnifiedMultimodalNetwork<T>
Implements: IUnifiedMultimodalModel<T>
For Beginners
This network handles text, images, audio, and video all in one architecture. Instead of needing separate models for each type of data, it processes all modalities through shared transformer layers with cross-modal attention. It can both understand any combination of inputs and generate outputs in any modality, enabling tasks like describing a video, generating images from text, or answering questions about audio clips.
Example
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 224, 224 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new UnifiedMultimodalNetwork<float>())
.Build(trainX, trainY);
var output = result.Predict(image);
Constructors
| Constructor | Summary |
|---|---|
UnifiedMultimodalNetwork() |
|
UnifiedMultimodalNetwork(NeuralNetworkArchitecture<T> architecture, UnifiedMultimodalNetworkOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, int? seed) |
Initializes a new instance of the UnifiedMultimodalNetwork. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<ModalityType> SupportedInputModalities { get; } |
|
IReadOnlyList<ModalityType> SupportedOutputModalities { get; } |
|
bool SupportsStreaming { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxSequenceLength { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<ModalityType, ValueTuple<bool, IEnumerable<string>>> SafetyCheck(IEnumerable<MultimodalInput<T>> inputs) |
|
IEnumerable<MultimodalOutput<T>> GenerateInterleaved(IEnumerable<MultimodalInput<T>> inputs, IEnumerable<ValueTuple<ModalityType, int>> outputSpec) |
|
IEnumerable<ValueTuple<int, T, ModalityType>> Retrieve(MultimodalInput<T> query, IEnumerable<MultimodalInput<T>> database, int topK) |
|
IEnumerable<ValueTuple<string, T, ModalityType, object>> Detect(IEnumerable<MultimodalInput<T>> inputs, string targetDescription) |
|
Matrix<T> AlignTemporally(IEnumerable<MultimodalInput<T>> inputs) |
|
Matrix<T> EncodeSequence(IEnumerable<MultimodalInput<T>> inputs) |
|
ModelMetadata<T> GetModelMetadata() |
|
MultimodalOutput<T> Edit(MultimodalInput<T> original, string editInstructions) |
|
MultimodalOutput<T> FewShotLearn(IEnumerable<ValueTuple<IEnumerable<MultimodalInput<T>>, MultimodalOutput<T>>> examples, IEnumerable<MultimodalInput<T>> query) |
|
MultimodalOutput<T> Generate(IEnumerable<MultimodalInput<T>> inputs, ModalityType outputModality, int maxLength) |
|
MultimodalOutput<T> Summarize(IEnumerable<MultimodalInput<T>> inputs, ModalityType outputModality, int maxLength) |
|
MultimodalOutput<T> Translate(MultimodalInput<T> input, ModalityType targetModality) |
|
T ComputeSimilarity(MultimodalInput<T> input1, MultimodalInput<T> input2) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
Training-mode forward — same architecture as Predict but routes every reshape through Reshape so the gradient tape stays connected end-to-end. |
Tensor<T> GenerateAudio(IEnumerable<MultimodalInput<T>> inputs, double durationSeconds, int sampleRate) |
|
Tensor<T> GenerateImage(IEnumerable<MultimodalInput<T>> inputs, int width, int height) |
|
Tensor<T> GetCrossModalAttention(IEnumerable<MultimodalInput<T>> inputs) |
|
ValueTuple<string, Dictionary<string, IEnumerable<T>>> Compare(IEnumerable<MultimodalInput<T>> inputs, IEnumerable<string> comparisonCriteria) |
|
ValueTuple<string, IEnumerable<string>> Reason(IEnumerable<MultimodalInput<T>> inputs, string task) |
|
ValueTuple<string, T> AnswerQuestion(IEnumerable<MultimodalInput<T>> context, string question) |
|
Vector<T> Encode(MultimodalInput<T> input) |
|
Vector<T> Fuse(IEnumerable<MultimodalInput<T>> inputs, string fusionStrategy) |
|
string Chat(IEnumerable<ValueTuple<string, IEnumerable<MultimodalInput<T>>>> conversationHistory, IEnumerable<MultimodalInput<T>> newInputs, int maxTokens) |
|
string GenerateText(IEnumerable<MultimodalInput<T>> inputs, string prompt, int maxTokens, double temperature) |