IAutoregressiveMultimodalModel<T>
Interfaces · AiDotNet.Interfaces
Defines the contract for autoregressive multimodal generation models that can generate tokens from any modality in an interleaved fashion.
Implements: IUnifiedMultimodalModel<T>
How It Works
This interface represents models like CM3Leon, Chameleon, or similar that use a unified vocabulary across all modalities and generate content token-by-token regardless of modality.
Properties
| Property | Summary |
|---|---|
IReadOnlyDictionary<ModalityType, int> ModalityTokenCounts { get; } |
Gets the number of tokens reserved for each modality. |
int VocabularySize { get; } |
Gets the vocabulary size (includes all modality tokens). |
Methods
| Method | Summary |
|---|---|
IEnumerable<MultimodalOutput<T>> Detokenize(IEnumerable<int> tokenIds) |
Detokenizes token IDs back to multimodal outputs. |
IEnumerable<int> Tokenize(MultimodalInput<T> input) |
Tokenizes input into unified vocabulary tokens. |
T ComputeLoss(IEnumerable<MultimodalInput<T>> inputs, IEnumerable<MultimodalOutput<T>> targets) |
Computes the loss for next-token prediction. |
ValueTuple<int, ModalityType> GenerateNextToken(IEnumerable<MultimodalInput<T>> context, double temperature) |
Generates the next token given the context. |
Vector<T> GetNextTokenLogits(IEnumerable<MultimodalInput<T>> context) |
Gets token probabilities for next position. |