MultimodalDataset<T>

Models & Types · AiDotNet.Data.Multimodal

A dataset of multimodal samples for training models that process multiple data types simultaneously.

For Beginners

Use this when your model needs to process multiple types of data together, like image captioning (image + text) or audio-visual tasks (audio + video).

How It Works

MultimodalDataset stores collections of multimodal samples and provides batching, shuffling, and splitting capabilities. Each sample can contain any combination of modalities (image, text, audio, etc.).

Constructors

Constructor Summary
MultimodalDataset()

Properties

Property Summary
MultimodalSample<T> Item { get; }
int Count { get; } Gets the total number of samples in the dataset.

Methods

Method Summary
HashSet<ModalityType> GetPresentModalities() Gets all unique modality types present across all samples.
HashSet<string> GetPresentKeys() Gets all unique modality keys present across all samples.
IEnumerable<ValueTuple<int, int>> GetBatchIndices(int batchSize) Iterates over the dataset in batches.
Tensor<T> GetLabelBatch(int startIndex, int batchSize) Gets a batch of label tensors.
Tensor<T> GetModalityBatch(int startIndex, int batchSize, string modalityKey) Gets a batch of modality tensors by key, stacked along a new batch dimension.
ValueTuple<MultimodalDataset<T>, MultimodalDataset<T>, MultimodalDataset<T>> Split(double trainRatio, double validationRatio, int? seed) Splits the dataset into train, validation, and test sets.
void Add(MultimodalSample<T> sample) Adds a multimodal sample to the dataset.
void AddRange(IEnumerable<MultimodalSample<T>> samples) Adds multiple multimodal samples to the dataset.
void Shuffle(int? seed) Shuffles the dataset using the specified random seed.