IAutoregressiveMultimodalModel<T>

Interfaces · AiDotNet.Interfaces

Defines the contract for autoregressive multimodal generation models that can generate tokens from any modality in an interleaved fashion.

Implements: IUnifiedMultimodalModel<T>

How It Works

This interface represents models like CM3Leon, Chameleon, or similar that use a unified vocabulary across all modalities and generate content token-by-token regardless of modality.

Properties

Property Summary
IReadOnlyDictionary<ModalityType, int> ModalityTokenCounts { get; } Gets the number of tokens reserved for each modality.
int VocabularySize { get; } Gets the vocabulary size (includes all modality tokens).

Methods

Method Summary
IEnumerable<MultimodalOutput<T>> Detokenize(IEnumerable<int> tokenIds) Detokenizes token IDs back to multimodal outputs.
IEnumerable<int> Tokenize(MultimodalInput<T> input) Tokenizes input into unified vocabulary tokens.
T ComputeLoss(IEnumerable<MultimodalInput<T>> inputs, IEnumerable<MultimodalOutput<T>> targets) Computes the loss for next-token prediction.
ValueTuple<int, ModalityType> GenerateNextToken(IEnumerable<MultimodalInput<T>> context, double temperature) Generates the next token given the context.
Vector<T> GetNextTokenLogits(IEnumerable<MultimodalInput<T>> context) Gets token probabilities for next position.