DataLoaders

Helpers & Utilities · AiDotNet.Data.Loaders

Static factory class for creating data loaders with beginner-friendly methods.

For Beginners

This is your starting point for loading data into AiDotNet! Choose the method that matches your data format:

Common Patterns:

// From arrays (simplest for small datasets)
var features = new double[,] { { 1.0, 2.0 }, { 3.0, 4.0 } };
var labels = new double[] { 0.0, 1.0 };
var arrayLoader = DataLoaders.FromArrays(features, labels);

// From Matrix and Vector (most common for ML)
var featureMatrix = new Matrix<double>(features);
var labelVector = new Vector<double>(labels);
var matrixLoader = DataLoaders.FromMatrixVector(featureMatrix, labelVector);

// From Tensors (for deep learning)
var inputTensor = Tensor<double>.CreateRandom(2, 2);
var outputTensor = Tensor<double>.CreateRandom(2, 1);
var tensorLoader = DataLoaders.FromTensors(inputTensor, outputTensor);

All loaders support:

  • Batching: loader.BatchSize = 32;
  • Shuffling: loader.Shuffle();
  • Splitting: var (train, val, test) = loader.Split();

How It Works

DataLoaders provides the easiest way to create data loaders for common scenarios. It follows a factory pattern with static methods that handle type inference and common configurations automatically.

Methods

Method Summary
Ade20kDataLoader<T> Ade20k(Ade20kDataLoaderOptions options) Creates an ADE20K semantic segmentation data loader (~25K images, 150 classes).
AudioFileDataset<T> AudioFiles(AudioFileDatasetOptions options) Creates an audio file dataset loader that reads WAV/PCM audio files from directories.
AudioSetDataLoader<T> AudioSet(AudioSetDataLoaderOptions options) Creates an AudioSet large-scale audio event dataset loader (2M+ clips, 527 categories).
BigEarthNetDataLoader<T> BigEarthNet(BigEarthNetDataLoaderOptions options) Creates a BigEarthNet multi-label remote sensing dataset loader (590K Sentinel-2 patches, 19 or 43 classes).
CelebADataLoader<T> CelebA(CelebADataLoaderOptions options) Creates a CelebA face attributes data loader (~200K images, 40 binary attributes).
CheXpertDataLoader<T> CheXpert(CheXpertDataLoaderOptions options) Creates a CheXpert chest radiograph dataset loader (224K images, 14 observations with uncertainty).
ChestXray14DataLoader<T> ChestXray14(ChestXray14DataLoaderOptions options) Creates a NIH Chest X-ray 14 multi-label classification dataset loader (112K images, 14 diseases).
Cifar100DataLoader<T> Cifar100(Cifar100DataLoaderOptions options) Creates a CIFAR-100 dataset loader (50k train / 10k test, 32x32 RGB, 100 classes).
Cifar10DataLoader<T> Cifar10(Cifar10DataLoaderOptions options) Creates a CIFAR-10 dataset loader (50k train / 10k test, 32x32 RGB, 10 classes).
CocoDetectionDataLoader<T> CocoDetection(CocoDetectionDataLoaderOptions options) Creates a COCO 2017 object detection dataset loader (118K train / 5K val, 80 categories).
CommonVoiceDataLoader<T> CommonVoice(CommonVoiceDataLoaderOptions options) Creates a Mozilla Common Voice multilingual speech dataset loader (19K+ hours, 100+ languages).
CsvStreamingDataLoader<T, TInput, TOutput> FromCsv(string filePath, Func<string, int, ValueTuple<TInput, TOutput>> lineParser, int batchSize, bool hasHeader, int prefetchCount) Creates a streaming data loader from a CSV file.
DocVqaDataLoader<T> DocVqa(DocVqaDataLoaderOptions options) Creates a DocVQA document visual question answering data loader.
Esc50DataLoader<T> Esc50(Esc50DataLoaderOptions options) Creates an ESC-50 environmental sound classification dataset loader (2000 clips, 50 classes).
EuroSatDataLoader<T> EuroSat(EuroSatDataLoaderOptions options) Creates an EuroSAT land use/land cover classification dataset loader (27K patches, 64x64, 10 classes).
FMoWDataLoader<T> FMoW(FMoWDataLoaderOptions options) Creates an fMoW (Functional Map of the World) satellite imagery dataset loader (1M+ images, 62 categories).
FashionMnistDataLoader<T> FashionMnist(FashionMnistDataLoaderOptions options) Creates a Fashion-MNIST dataset loader (60k train / 10k test, 28x28 grayscale clothing images).
FileStreamingDataLoader<T, TInput, TOutput> FromDirectory(string directory, string filePattern, Func<string, CancellationToken, Task<ValueTuple<TInput, TOutput>>> fileProcessor, int batchSize, SearchOption searchOption, int prefetchCount, int numWorkers) Creates a streaming data loader from a directory of files.
FleursDataLoader<T> Fleurs(FleursDataLoaderOptions options) Creates a FLEURS multilingual speech benchmark loader (102 languages, ~12 hours per language).
Fsd50kDataLoader<T> Fsd50k(Fsd50kDataLoaderOptions options) Creates an FSD50K audio event dataset loader (51,197 clips, 200 sound event classes).
GigaSpeechDataLoader<T> GigaSpeech(GigaSpeechDataLoaderOptions options) Creates a GigaSpeech multi-domain English ASR dataset loader (up to 10K hours).
GlueDataLoader<T> Glue(GlueDataLoaderOptions options) Creates a GLUE benchmark data loader (9 NLU tasks for evaluating language understanding).
Hmdb51DataLoader<T> Hmdb51(Hmdb51DataLoaderOptions options) Creates an HMDB-51 video action recognition data loader (51 classes, 6.8K clips).
INaturalistDataLoader<T> INaturalist(INaturalistDataLoaderOptions options) Creates an iNaturalist species classification dataset loader (~2.7M images, 10,000 species).
ImageFolderDataset<T> ImageFolder(ImageFolderDatasetOptions options) Creates an image folder dataset loader that reads images from a directory structure where each subdirectory name is a class label.
ImageNet1kDataLoader<T> ImageNet1k(ImageNet1kDataLoaderOptions options) Creates an ImageNet-1K (ILSVRC 2012) dataset loader (~1.28M train / 50K val, 1000 classes).
ImageNet21kDataLoader<T> ImageNet21k(ImageNet21kDataLoaderOptions options) Creates an ImageNet-21K dataset loader (~14.2M images, 21,841 categories).
Imdb50kDataLoader<T> Imdb50k(Imdb50kDataLoaderOptions options) Creates an IMDB 50k movie review sentiment dataset loader (25k train / 25k test).
InMemoryDataLoader<T, Matrix<T>, Matrix<T>> FromMatrices(Matrix<T> features, Matrix<T> labels) Creates a data loader from a feature Matrix and label Matrix (for multi-output regression).
InMemoryDataLoader<T, Matrix<T>, Vector<T>> Empty() Creates an empty data loader placeholder (useful for meta-learning or custom scenarios).
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromArrays(T[] features, T[] labels) Creates a data loader from 1D feature array (single feature) and 1D label array.
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromArrays(T[] features, T[] labels) Creates a data loader from 1D feature array (single feature) and 1D label array.
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromArrays(T[][] features, T[] labels) Creates a data loader from 1D feature array (single feature) and 1D label array.
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromMatrix(Matrix<T> features) Creates a data loader from a feature Matrix only (for unsupervised learning like clustering).
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromMatrixVector(Matrix<T> features, Vector<T> labels) Creates a data loader from a feature Matrix and label Vector.
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromTextDocuments(string[] documents, T[] labels, CountVectorizer<T> vectorizer) Creates a data loader from text documents using a Hashing vectorizer with array labels.
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromTextDocuments(string[] documents, T[] labels, HashingVectorizer<T> vectorizer) Creates a data loader from text documents using a Hashing vectorizer with array labels.
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromTextDocuments(string[] documents, T[] labels, ITextVectorizer<T> vectorizer) Creates a data loader from text documents using a Hashing vectorizer with array labels.
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromTextDocuments(string[] documents, T[] labels, TfidfVectorizer<T> vectorizer) Creates a data loader from text documents using a Hashing vectorizer with array labels.
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromTextDocuments(string[] documents, Vector<T> labels, CountVectorizer<T> vectorizer) Creates a data loader from text documents using a Hashing vectorizer with array labels.
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromTextDocuments(string[] documents, Vector<T> labels, HashingVectorizer<T> vectorizer) Creates a data loader from text documents using a Hashing vectorizer with array labels.
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromTextDocuments(string[] documents, Vector<T> labels, ITextVectorizer<T> vectorizer) Creates a data loader from text documents using a Hashing vectorizer with array labels.
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromTextDocuments(string[] documents, Vector<T> labels, TfidfVectorizer<T> vectorizer) Creates a data loader from text documents using a Hashing vectorizer with array labels.
InMemoryDataLoader<T, Matrix<T>, Vector<T>> WithBatchSize(Matrix<T> features, Vector<T> labels, int batchSize) Creates a data loader with pre-configured batch size.
InMemoryDataLoader<T, Tensor<T>, Tensor<T>> FromTensors(Tensor<T> features, Tensor<T> labels) Creates a data loader from input and output Tensors.
InMemoryDataLoader<T, Tensor<T>, Vector<T>> FromTensorVector(Tensor<T> features, Vector<T> labels) Creates a data loader from a Tensor of features and a Vector of labels.
JsonlDataLoader<T> FromJsonl(string filePath, Func<JObject, ValueTuple<Tensor<T>, Tensor<T>>> recordParser, int batchSize, string textField, string labelField, int shuffleBufferSize) Creates a typed JSONL data loader for reading multiple JSONL files.
JsonlDataLoader<T> FromJsonl(string[] filePaths, Func<JObject, ValueTuple<Tensor<T>, Tensor<T>>> recordParser, int batchSize, string textField, string labelField, int shuffleBufferSize) Creates a typed JSONL data loader for reading multiple JSONL files.
Kinetics400DataLoader<T> Kinetics400(Kinetics400DataLoaderOptions options) Creates a Kinetics-400 video action recognition data loader (400 classes, ~300K clips).
KittiDataLoader<T> Kitti(KittiDataLoaderOptions options) Creates a KITTI 3D object detection data loader (LiDAR point clouds with bounding boxes).
LeafFederatedDataLoader<T> FromLeafFederatedJsonFiles(string trainFilePath, string testFilePath, LeafFederatedDatasetLoadOptions options) Creates a LEAF federated data loader from LEAF benchmark JSON files.
LibriSpeechDataLoader<T> LibriSpeech(LibriSpeechDataLoaderOptions options) Creates a LibriSpeech ASR dataset loader (~1000 hours of 16kHz English speech).
MaestroDataLoader<T> Maestro(MaestroDataLoaderOptions options) Creates a MAESTRO piano performance dataset loader (~200 hours, aligned MIDI and audio).
MnistDataLoader<T> Mnist(MnistDataLoaderOptions options) Creates a MNIST handwritten digit dataset loader (60k train / 10k test, 28x28 grayscale).
ModelNet40ClassificationDataLoader<T> ModelNet40Classification(ModelNet40ClassificationDataLoaderOptions options) Creates a ModelNet40 classification data loader.
MultimodalDataset<T> Multimodal() Creates a new empty multimodal dataset for building vision-language or multi-modal training samples.
MultimodalDataset<T> Multimodal(IEnumerable<MultimodalSample<T>> samples) Creates a multimodal dataset pre-populated with samples.
Musdb18DataLoader<T> Musdb18(Musdb18DataLoaderOptions options) Creates a MUSDB18 music source separation dataset loader (150 tracks, 4 stems).
NuScenesDataLoader<T> NuScenes(NuScenesDataLoaderOptions options) Creates a nuScenes 3D object detection data loader (LiDAR point clouds, 23 classes).
OpenImagesDataLoader<T> OpenImages(OpenImagesDataLoaderOptions options) Creates an Open Images V7 object detection dataset loader (~9M images, 600 categories).
ParquetDataLoader<T> FromParquet(ParquetDataLoaderOptions options) Creates a Parquet file data loader for reading columnar data from Apache Parquet files.
PascalVocDataLoader<T> PascalVoc(PascalVocDataLoaderOptions options) Creates a Pascal VOC object detection dataset loader (20 categories, XML annotations).
Places365DataLoader<T> Places365(Places365DataLoaderOptions options) Creates a Places365 scene recognition dataset loader (1.8M train / 36.5K val, 365 scene categories).
ProteinDataLoader<T> Protein(ProteinDataLoaderOptions options) Creates a protein structure graph classification data loader.
PubLayNetDataLoader<T> PubLayNet(PubLayNetDataLoaderOptions options) Creates a PubLayNet document layout analysis data loader (~360K images, 5 categories).
Qm9DataLoader<T> Qm9(Qm9DataLoaderOptions options) Creates a QM9 molecular property prediction data loader (~134K molecules, 19 properties).
RetinalFundusDataLoader<T> RetinalFundus(RetinalFundusDataLoaderOptions options) Creates a Retinal Fundus diabetic retinopathy grading dataset loader (5 severity levels).
ScanNetSemanticSegmentationDataLoader<T> ScanNetSemanticSegmentation(ScanNetSemanticSegmentationDataLoaderOptions options) Creates a ScanNet semantic segmentation data loader.
SemanticKittiDataLoader<T> SemanticKitti(SemanticKittiDataLoaderOptions options) Creates a SemanticKITTI per-point semantic segmentation data loader (28 classes).
ShapeNetCorePartSegmentationDataLoader<T> ShapeNetCorePartSegmentation(ShapeNetCorePartSegmentationDataLoaderOptions options) Creates a ShapeNetCore part segmentation data loader.
ShardedStreamingDataLoader<T> FromShards(string[] shardPaths, Func<byte[], ValueTuple<Tensor<T>, Tensor<T>>> recordParser, int batchSize, ShardedStreamingDatasetOptions options) Creates a typed sharded streaming data loader for deterministic, resumable reading, compatible with AiModelBuilder.ConfigureDataLoader().
SkinLesionDataLoader<T> SkinLesion(SkinLesionDataLoaderOptions options) Creates an ISIC Skin Lesion classification dataset loader (~25K images, 8 diagnostic categories).
SomethingSomethingV2DataLoader<T> SomethingSomethingV2(SomethingSomethingV2DataLoaderOptions options) Creates a Something-Something V2 video understanding data loader (174 classes, ~220K clips).
SquadDataLoader<T> Squad(SquadDataLoaderOptions options) Creates a SQuAD reading comprehension data loader (100K+ question-answer pairs).
StatefulDataLoader<T, TInput, TOutput> WithCheckpointing(InputOutputDataLoaderBase<T, TInput, TOutput> loader) Wraps any data loader with stateful checkpointing support for mid-epoch resume.
StreamingDataLoader<T, TInput, TOutput> Streaming(int sampleCount, Func<int, CancellationToken, Task<ValueTuple<TInput, TOutput>>> sampleReader, int batchSize, int prefetchCount, int numWorkers) Creates a streaming data loader that reads samples on-demand.
StreamingTextDataset<T> StreamingText(StreamingTextDatasetOptions options) Creates a streaming text dataset for LLM pre-training on large text corpora.
SuperGlueDataLoader<T> SuperGlue(SuperGlueDataLoaderOptions options) Creates a SuperGLUE benchmark data loader (8 advanced NLU tasks).
TemporalGraphDataLoader<T> TemporalGraph(TemporalGraphDataLoaderOptions options) Creates a temporal graph data loader for dynamic link prediction.
TransformedDataLoader<T> WithTransforms(InputOutputDataLoaderBase<T, Tensor<T>, Tensor<T>> loader, ITransform<T[], T[]> transform) Wraps a data loader with a composable transform pipeline applied to features.
Ucf101DataLoader<T> Ucf101(Ucf101DataLoaderOptions options) Creates a UCF-101 video action recognition data loader (101 classes, 13.3K clips).
UrbanSound8kDataLoader<T> UrbanSound8k(UrbanSound8kDataLoaderOptions options) Creates an UrbanSound8K urban sound classification dataset loader (8732 clips, 10 classes).
VideoFrameDataset<T> VideoFrames(VideoFrameDatasetOptions options) Creates a video frame dataset loader that extracts frames from video directories.
VoxPopuliDataLoader<T> VoxPopuli(VoxPopuliDataLoaderOptions options) Creates a VoxPopuli multilingual speech dataset loader (400K+ hours, 23 languages).
WaymoDataLoader<T> Waymo(WaymoDataLoaderOptions options) Creates a Waymo Open Dataset 3D detection data loader (LiDAR point clouds).
WebDatasetDataLoader<T> FromWebDataset(string tarPath, Func<Dictionary<string, byte[]>, ValueTuple<Tensor<T>, Tensor<T>>> sampleParser, int batchSize, WebDatasetOptions options) Creates a typed WebDataset data loader for a single TAR archive.
WebDatasetDataLoader<T> FromWebDataset(string[] tarPaths, Func<Dictionary<string, byte[]>, ValueTuple<Tensor<T>, Tensor<T>>> sampleParser, int batchSize, WebDatasetOptions options) Creates a typed WebDataset data loader for a single TAR archive.
WikiText103DataLoader<T> WikiText103(WikiText103DataLoaderOptions options) Creates a WikiText-103 language modeling data loader (100M+ tokens from Wikipedia).
Wikidata5mDataLoader<T> Wikidata5m(Wikidata5mDataLoaderOptions options) Creates a Wikidata5M knowledge graph link prediction data loader (~5M entities, 21M triplets).
ZincDataLoader<T> Zinc(ZincDataLoaderOptions options) Creates a ZINC molecular dataset data loader (~250K drug-like molecules).