IUnifiedMultimodalModel<T>

Interfaces · AiDotNet.Interfaces

Defines the contract for unified multimodal models that handle multiple modalities in a single architecture, similar to GPT-4o, Gemini, or Meta’s CM3Leon.

Known implementations: IAutoregressiveMultimodalModel<T>, UnifiedMultimodalNetwork<T>

For Beginners

One model that can see, hear, read, and create!

Key capabilities:

  • Any-to-any generation: Text → Image, Image → Text, Audio → Text, etc.
  • Interleaved understanding: Process mixed sequences of text, images, audio
  • Cross-modal reasoning: Answer questions using information from multiple sources
  • Unified embeddings: All modalities share a common representation space

Architecture concepts:

  1. Modality Encoders: Specialized encoders for each input type
  2. Unified Transformer: Core model that processes all modalities
  3. Modality Decoders: Generate outputs in any modality
  4. Cross-Attention: Allow modalities to attend to each other

How It Works

Unified multimodal models represent the next generation of AI systems that can seamlessly process and generate content across multiple modalities (text, image, audio, video) within a single unified architecture.

Properties

Property Summary
IReadOnlyList<ModalityType> SupportedInputModalities { get; } Gets the supported input modalities.
IReadOnlyList<ModalityType> SupportedOutputModalities { get; } Gets the supported output modalities.
bool SupportsStreaming { get; } Gets whether the model supports streaming generation.
int EmbeddingDimension { get; } Gets the unified embedding dimension.
int MaxSequenceLength { get; } Gets the maximum sequence length for interleaved inputs.

Methods

Method Summary
Dictionary<ModalityType, ValueTuple<bool, IEnumerable<string>>> SafetyCheck(IEnumerable<MultimodalInput<T>> inputs) Checks content for safety across all modalities.
IEnumerable<MultimodalOutput<T>> GenerateInterleaved(IEnumerable<MultimodalInput<T>> inputs, IEnumerable<ValueTuple<ModalityType, int>> outputSpec) Generates an interleaved sequence of multiple modalities.
IEnumerable<ValueTuple<int, T, ModalityType>> Retrieve(MultimodalInput<T> query, IEnumerable<MultimodalInput<T>> database, int topK) Retrieves the most similar items from a database given a query.
IEnumerable<ValueTuple<string, T, ModalityType, object>> Detect(IEnumerable<MultimodalInput<T>> inputs, string targetDescription) Detects and localizes objects/events across modalities.
Matrix<T> AlignTemporally(IEnumerable<MultimodalInput<T>> inputs) Aligns content across modalities temporally.
Matrix<T> EncodeSequence(IEnumerable<MultimodalInput<T>> inputs) Encodes multiple interleaved inputs into a sequence of embeddings.
MultimodalOutput<T> Edit(MultimodalInput<T> original, string editInstructions) Edits multimodal content based on instructions.
MultimodalOutput<T> FewShotLearn(IEnumerable<ValueTuple<IEnumerable<MultimodalInput<T>>, MultimodalOutput<T>>> examples, IEnumerable<MultimodalInput<T>> query) Performs in-context learning from multimodal examples.
MultimodalOutput<T> Generate(IEnumerable<MultimodalInput<T>> inputs, ModalityType outputModality, int maxLength) Generates output in the specified modality given multimodal inputs.
MultimodalOutput<T> Summarize(IEnumerable<MultimodalInput<T>> inputs, ModalityType outputModality, int maxLength) Summarizes multimodal content.
MultimodalOutput<T> Translate(MultimodalInput<T> input, ModalityType targetModality) Translates content between modalities.
T ComputeSimilarity(MultimodalInput<T> input1, MultimodalInput<T> input2) Computes cross-modal similarity between inputs.
Tensor<T> GenerateAudio(IEnumerable<MultimodalInput<T>> inputs, double durationSeconds, int sampleRate) Generates audio from multimodal inputs.
Tensor<T> GenerateImage(IEnumerable<MultimodalInput<T>> inputs, int width, int height) Generates an image from multimodal inputs.
Tensor<T> GetCrossModalAttention(IEnumerable<MultimodalInput<T>> inputs) Gets attention weights showing cross-modal relationships.
ValueTuple<string, Dictionary<string, IEnumerable<T>>> Compare(IEnumerable<MultimodalInput<T>> inputs, IEnumerable<string> comparisonCriteria) Compares multiple multimodal inputs and provides analysis.
ValueTuple<string, IEnumerable<string>> Reason(IEnumerable<MultimodalInput<T>> inputs, string task) Performs reasoning across multiple modalities.
ValueTuple<string, T> AnswerQuestion(IEnumerable<MultimodalInput<T>> context, string question) Answers a question using multimodal context.
Vector<T> Encode(MultimodalInput<T> input) Encodes any supported modality into the unified embedding space.
Vector<T> Fuse(IEnumerable<MultimodalInput<T>> inputs, string fusionStrategy) Fuses multiple modality inputs into a unified representation.
string Chat(IEnumerable<ValueTuple<string, IEnumerable<MultimodalInput<T>>>> conversationHistory, IEnumerable<MultimodalInput<T>> newInputs, int maxTokens) Conducts a multi-turn conversation with multimodal inputs.
string GenerateText(IEnumerable<MultimodalInput<T>> inputs, string prompt, int maxTokens, double temperature) Generates text response from multimodal inputs.