DataLoaders
Helpers & Utilities · AiDotNet.Data.Loaders
Static factory class for creating data loaders with beginner-friendly methods.
For Beginners
This is your starting point for loading data into AiDotNet! Choose the method that matches your data format:
Common Patterns:
// From arrays (simplest for small datasets)
var features = new double[,] { { 1.0, 2.0 }, { 3.0, 4.0 } };
var labels = new double[] { 0.0, 1.0 };
var arrayLoader = DataLoaders.FromArrays(features, labels);
// From Matrix and Vector (most common for ML)
var featureMatrix = new Matrix<double>(features);
var labelVector = new Vector<double>(labels);
var matrixLoader = DataLoaders.FromMatrixVector(featureMatrix, labelVector);
// From Tensors (for deep learning)
var inputTensor = Tensor<double>.CreateRandom(2, 2);
var outputTensor = Tensor<double>.CreateRandom(2, 1);
var tensorLoader = DataLoaders.FromTensors(inputTensor, outputTensor);
All loaders support:
- Batching:
loader.BatchSize = 32; - Shuffling:
loader.Shuffle(); - Splitting:
var (train, val, test) = loader.Split();
How It Works
DataLoaders provides the easiest way to create data loaders for common scenarios. It follows a factory pattern with static methods that handle type inference and common configurations automatically.
Methods
| Method | Summary |
|---|---|
Ade20kDataLoader<T> Ade20k(Ade20kDataLoaderOptions options) |
Creates an ADE20K semantic segmentation data loader (~25K images, 150 classes). |
AudioFileDataset<T> AudioFiles(AudioFileDatasetOptions options) |
Creates an audio file dataset loader that reads WAV/PCM audio files from directories. |
AudioSetDataLoader<T> AudioSet(AudioSetDataLoaderOptions options) |
Creates an AudioSet large-scale audio event dataset loader (2M+ clips, 527 categories). |
BigEarthNetDataLoader<T> BigEarthNet(BigEarthNetDataLoaderOptions options) |
Creates a BigEarthNet multi-label remote sensing dataset loader (590K Sentinel-2 patches, 19 or 43 classes). |
CelebADataLoader<T> CelebA(CelebADataLoaderOptions options) |
Creates a CelebA face attributes data loader (~200K images, 40 binary attributes). |
CheXpertDataLoader<T> CheXpert(CheXpertDataLoaderOptions options) |
Creates a CheXpert chest radiograph dataset loader (224K images, 14 observations with uncertainty). |
ChestXray14DataLoader<T> ChestXray14(ChestXray14DataLoaderOptions options) |
Creates a NIH Chest X-ray 14 multi-label classification dataset loader (112K images, 14 diseases). |
Cifar100DataLoader<T> Cifar100(Cifar100DataLoaderOptions options) |
Creates a CIFAR-100 dataset loader (50k train / 10k test, 32x32 RGB, 100 classes). |
Cifar10DataLoader<T> Cifar10(Cifar10DataLoaderOptions options) |
Creates a CIFAR-10 dataset loader (50k train / 10k test, 32x32 RGB, 10 classes). |
CocoDetectionDataLoader<T> CocoDetection(CocoDetectionDataLoaderOptions options) |
Creates a COCO 2017 object detection dataset loader (118K train / 5K val, 80 categories). |
CommonVoiceDataLoader<T> CommonVoice(CommonVoiceDataLoaderOptions options) |
Creates a Mozilla Common Voice multilingual speech dataset loader (19K+ hours, 100+ languages). |
CsvStreamingDataLoader<T, TInput, TOutput> FromCsv(string filePath, Func<string, int, ValueTuple<TInput, TOutput>> lineParser, int batchSize, bool hasHeader, int prefetchCount) |
Creates a streaming data loader from a CSV file. |
DocVqaDataLoader<T> DocVqa(DocVqaDataLoaderOptions options) |
Creates a DocVQA document visual question answering data loader. |
Esc50DataLoader<T> Esc50(Esc50DataLoaderOptions options) |
Creates an ESC-50 environmental sound classification dataset loader (2000 clips, 50 classes). |
EuroSatDataLoader<T> EuroSat(EuroSatDataLoaderOptions options) |
Creates an EuroSAT land use/land cover classification dataset loader (27K patches, 64x64, 10 classes). |
FMoWDataLoader<T> FMoW(FMoWDataLoaderOptions options) |
Creates an fMoW (Functional Map of the World) satellite imagery dataset loader (1M+ images, 62 categories). |
FashionMnistDataLoader<T> FashionMnist(FashionMnistDataLoaderOptions options) |
Creates a Fashion-MNIST dataset loader (60k train / 10k test, 28x28 grayscale clothing images). |
FileStreamingDataLoader<T, TInput, TOutput> FromDirectory(string directory, string filePattern, Func<string, CancellationToken, Task<ValueTuple<TInput, TOutput>>> fileProcessor, int batchSize, SearchOption searchOption, int prefetchCount, int numWorkers) |
Creates a streaming data loader from a directory of files. |
FleursDataLoader<T> Fleurs(FleursDataLoaderOptions options) |
Creates a FLEURS multilingual speech benchmark loader (102 languages, ~12 hours per language). |
Fsd50kDataLoader<T> Fsd50k(Fsd50kDataLoaderOptions options) |
Creates an FSD50K audio event dataset loader (51,197 clips, 200 sound event classes). |
GigaSpeechDataLoader<T> GigaSpeech(GigaSpeechDataLoaderOptions options) |
Creates a GigaSpeech multi-domain English ASR dataset loader (up to 10K hours). |
GlueDataLoader<T> Glue(GlueDataLoaderOptions options) |
Creates a GLUE benchmark data loader (9 NLU tasks for evaluating language understanding). |
Hmdb51DataLoader<T> Hmdb51(Hmdb51DataLoaderOptions options) |
Creates an HMDB-51 video action recognition data loader (51 classes, 6.8K clips). |
INaturalistDataLoader<T> INaturalist(INaturalistDataLoaderOptions options) |
Creates an iNaturalist species classification dataset loader (~2.7M images, 10,000 species). |
ImageFolderDataset<T> ImageFolder(ImageFolderDatasetOptions options) |
Creates an image folder dataset loader that reads images from a directory structure where each subdirectory name is a class label. |
ImageNet1kDataLoader<T> ImageNet1k(ImageNet1kDataLoaderOptions options) |
Creates an ImageNet-1K (ILSVRC 2012) dataset loader (~1.28M train / 50K val, 1000 classes). |
ImageNet21kDataLoader<T> ImageNet21k(ImageNet21kDataLoaderOptions options) |
Creates an ImageNet-21K dataset loader (~14.2M images, 21,841 categories). |
Imdb50kDataLoader<T> Imdb50k(Imdb50kDataLoaderOptions options) |
Creates an IMDB 50k movie review sentiment dataset loader (25k train / 25k test). |
InMemoryDataLoader<T, Matrix<T>, Matrix<T>> FromMatrices(Matrix<T> features, Matrix<T> labels) |
Creates a data loader from a feature Matrix and label Matrix (for multi-output regression). |
InMemoryDataLoader<T, Matrix<T>, Vector<T>> Empty() |
Creates an empty data loader placeholder (useful for meta-learning or custom scenarios). |
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromArrays(T[] features, T[] labels) |
Creates a data loader from 1D feature array (single feature) and 1D label array. |
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromArrays(T[] features, T[] labels) |
Creates a data loader from 1D feature array (single feature) and 1D label array. |
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromArrays(T[][] features, T[] labels) |
Creates a data loader from 1D feature array (single feature) and 1D label array. |
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromMatrix(Matrix<T> features) |
Creates a data loader from a feature Matrix only (for unsupervised learning like clustering). |
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromMatrixVector(Matrix<T> features, Vector<T> labels) |
Creates a data loader from a feature Matrix and label Vector. |
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromTextDocuments(string[] documents, T[] labels, CountVectorizer<T> vectorizer) |
Creates a data loader from text documents using a Hashing vectorizer with array labels. |
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromTextDocuments(string[] documents, T[] labels, HashingVectorizer<T> vectorizer) |
Creates a data loader from text documents using a Hashing vectorizer with array labels. |
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromTextDocuments(string[] documents, T[] labels, ITextVectorizer<T> vectorizer) |
Creates a data loader from text documents using a Hashing vectorizer with array labels. |
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromTextDocuments(string[] documents, T[] labels, TfidfVectorizer<T> vectorizer) |
Creates a data loader from text documents using a Hashing vectorizer with array labels. |
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromTextDocuments(string[] documents, Vector<T> labels, CountVectorizer<T> vectorizer) |
Creates a data loader from text documents using a Hashing vectorizer with array labels. |
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromTextDocuments(string[] documents, Vector<T> labels, HashingVectorizer<T> vectorizer) |
Creates a data loader from text documents using a Hashing vectorizer with array labels. |
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromTextDocuments(string[] documents, Vector<T> labels, ITextVectorizer<T> vectorizer) |
Creates a data loader from text documents using a Hashing vectorizer with array labels. |
InMemoryDataLoader<T, Matrix<T>, Vector<T>> FromTextDocuments(string[] documents, Vector<T> labels, TfidfVectorizer<T> vectorizer) |
Creates a data loader from text documents using a Hashing vectorizer with array labels. |
InMemoryDataLoader<T, Matrix<T>, Vector<T>> WithBatchSize(Matrix<T> features, Vector<T> labels, int batchSize) |
Creates a data loader with pre-configured batch size. |
InMemoryDataLoader<T, Tensor<T>, Tensor<T>> FromTensors(Tensor<T> features, Tensor<T> labels) |
Creates a data loader from input and output Tensors. |
InMemoryDataLoader<T, Tensor<T>, Vector<T>> FromTensorVector(Tensor<T> features, Vector<T> labels) |
Creates a data loader from a Tensor of features and a Vector of labels. |
JsonlDataLoader<T> FromJsonl(string filePath, Func<JObject, ValueTuple<Tensor<T>, Tensor<T>>> recordParser, int batchSize, string textField, string labelField, int shuffleBufferSize) |
Creates a typed JSONL data loader for reading multiple JSONL files. |
JsonlDataLoader<T> FromJsonl(string[] filePaths, Func<JObject, ValueTuple<Tensor<T>, Tensor<T>>> recordParser, int batchSize, string textField, string labelField, int shuffleBufferSize) |
Creates a typed JSONL data loader for reading multiple JSONL files. |
Kinetics400DataLoader<T> Kinetics400(Kinetics400DataLoaderOptions options) |
Creates a Kinetics-400 video action recognition data loader (400 classes, ~300K clips). |
KittiDataLoader<T> Kitti(KittiDataLoaderOptions options) |
Creates a KITTI 3D object detection data loader (LiDAR point clouds with bounding boxes). |
LeafFederatedDataLoader<T> FromLeafFederatedJsonFiles(string trainFilePath, string testFilePath, LeafFederatedDatasetLoadOptions options) |
Creates a LEAF federated data loader from LEAF benchmark JSON files. |
LibriSpeechDataLoader<T> LibriSpeech(LibriSpeechDataLoaderOptions options) |
Creates a LibriSpeech ASR dataset loader (~1000 hours of 16kHz English speech). |
MaestroDataLoader<T> Maestro(MaestroDataLoaderOptions options) |
Creates a MAESTRO piano performance dataset loader (~200 hours, aligned MIDI and audio). |
MnistDataLoader<T> Mnist(MnistDataLoaderOptions options) |
Creates a MNIST handwritten digit dataset loader (60k train / 10k test, 28x28 grayscale). |
ModelNet40ClassificationDataLoader<T> ModelNet40Classification(ModelNet40ClassificationDataLoaderOptions options) |
Creates a ModelNet40 classification data loader. |
MultimodalDataset<T> Multimodal() |
Creates a new empty multimodal dataset for building vision-language or multi-modal training samples. |
MultimodalDataset<T> Multimodal(IEnumerable<MultimodalSample<T>> samples) |
Creates a multimodal dataset pre-populated with samples. |
Musdb18DataLoader<T> Musdb18(Musdb18DataLoaderOptions options) |
Creates a MUSDB18 music source separation dataset loader (150 tracks, 4 stems). |
NuScenesDataLoader<T> NuScenes(NuScenesDataLoaderOptions options) |
Creates a nuScenes 3D object detection data loader (LiDAR point clouds, 23 classes). |
OpenImagesDataLoader<T> OpenImages(OpenImagesDataLoaderOptions options) |
Creates an Open Images V7 object detection dataset loader (~9M images, 600 categories). |
ParquetDataLoader<T> FromParquet(ParquetDataLoaderOptions options) |
Creates a Parquet file data loader for reading columnar data from Apache Parquet files. |
PascalVocDataLoader<T> PascalVoc(PascalVocDataLoaderOptions options) |
Creates a Pascal VOC object detection dataset loader (20 categories, XML annotations). |
Places365DataLoader<T> Places365(Places365DataLoaderOptions options) |
Creates a Places365 scene recognition dataset loader (1.8M train / 36.5K val, 365 scene categories). |
ProteinDataLoader<T> Protein(ProteinDataLoaderOptions options) |
Creates a protein structure graph classification data loader. |
PubLayNetDataLoader<T> PubLayNet(PubLayNetDataLoaderOptions options) |
Creates a PubLayNet document layout analysis data loader (~360K images, 5 categories). |
Qm9DataLoader<T> Qm9(Qm9DataLoaderOptions options) |
Creates a QM9 molecular property prediction data loader (~134K molecules, 19 properties). |
RetinalFundusDataLoader<T> RetinalFundus(RetinalFundusDataLoaderOptions options) |
Creates a Retinal Fundus diabetic retinopathy grading dataset loader (5 severity levels). |
ScanNetSemanticSegmentationDataLoader<T> ScanNetSemanticSegmentation(ScanNetSemanticSegmentationDataLoaderOptions options) |
Creates a ScanNet semantic segmentation data loader. |
SemanticKittiDataLoader<T> SemanticKitti(SemanticKittiDataLoaderOptions options) |
Creates a SemanticKITTI per-point semantic segmentation data loader (28 classes). |
ShapeNetCorePartSegmentationDataLoader<T> ShapeNetCorePartSegmentation(ShapeNetCorePartSegmentationDataLoaderOptions options) |
Creates a ShapeNetCore part segmentation data loader. |
ShardedStreamingDataLoader<T> FromShards(string[] shardPaths, Func<byte[], ValueTuple<Tensor<T>, Tensor<T>>> recordParser, int batchSize, ShardedStreamingDatasetOptions options) |
Creates a typed sharded streaming data loader for deterministic, resumable reading, compatible with AiModelBuilder.ConfigureDataLoader(). |
SkinLesionDataLoader<T> SkinLesion(SkinLesionDataLoaderOptions options) |
Creates an ISIC Skin Lesion classification dataset loader (~25K images, 8 diagnostic categories). |
SomethingSomethingV2DataLoader<T> SomethingSomethingV2(SomethingSomethingV2DataLoaderOptions options) |
Creates a Something-Something V2 video understanding data loader (174 classes, ~220K clips). |
SquadDataLoader<T> Squad(SquadDataLoaderOptions options) |
Creates a SQuAD reading comprehension data loader (100K+ question-answer pairs). |
StatefulDataLoader<T, TInput, TOutput> WithCheckpointing(InputOutputDataLoaderBase<T, TInput, TOutput> loader) |
Wraps any data loader with stateful checkpointing support for mid-epoch resume. |
StreamingDataLoader<T, TInput, TOutput> Streaming(int sampleCount, Func<int, CancellationToken, Task<ValueTuple<TInput, TOutput>>> sampleReader, int batchSize, int prefetchCount, int numWorkers) |
Creates a streaming data loader that reads samples on-demand. |
StreamingTextDataset<T> StreamingText(StreamingTextDatasetOptions options) |
Creates a streaming text dataset for LLM pre-training on large text corpora. |
SuperGlueDataLoader<T> SuperGlue(SuperGlueDataLoaderOptions options) |
Creates a SuperGLUE benchmark data loader (8 advanced NLU tasks). |
TemporalGraphDataLoader<T> TemporalGraph(TemporalGraphDataLoaderOptions options) |
Creates a temporal graph data loader for dynamic link prediction. |
TransformedDataLoader<T> WithTransforms(InputOutputDataLoaderBase<T, Tensor<T>, Tensor<T>> loader, ITransform<T[], T[]> transform) |
Wraps a data loader with a composable transform pipeline applied to features. |
Ucf101DataLoader<T> Ucf101(Ucf101DataLoaderOptions options) |
Creates a UCF-101 video action recognition data loader (101 classes, 13.3K clips). |
UrbanSound8kDataLoader<T> UrbanSound8k(UrbanSound8kDataLoaderOptions options) |
Creates an UrbanSound8K urban sound classification dataset loader (8732 clips, 10 classes). |
VideoFrameDataset<T> VideoFrames(VideoFrameDatasetOptions options) |
Creates a video frame dataset loader that extracts frames from video directories. |
VoxPopuliDataLoader<T> VoxPopuli(VoxPopuliDataLoaderOptions options) |
Creates a VoxPopuli multilingual speech dataset loader (400K+ hours, 23 languages). |
WaymoDataLoader<T> Waymo(WaymoDataLoaderOptions options) |
Creates a Waymo Open Dataset 3D detection data loader (LiDAR point clouds). |
WebDatasetDataLoader<T> FromWebDataset(string tarPath, Func<Dictionary<string, byte[]>, ValueTuple<Tensor<T>, Tensor<T>>> sampleParser, int batchSize, WebDatasetOptions options) |
Creates a typed WebDataset data loader for a single TAR archive. |
WebDatasetDataLoader<T> FromWebDataset(string[] tarPaths, Func<Dictionary<string, byte[]>, ValueTuple<Tensor<T>, Tensor<T>>> sampleParser, int batchSize, WebDatasetOptions options) |
Creates a typed WebDataset data loader for a single TAR archive. |
WikiText103DataLoader<T> WikiText103(WikiText103DataLoaderOptions options) |
Creates a WikiText-103 language modeling data loader (100M+ tokens from Wikipedia). |
Wikidata5mDataLoader<T> Wikidata5m(Wikidata5mDataLoaderOptions options) |
Creates a Wikidata5M knowledge graph link prediction data loader (~5M entities, 21M triplets). |
ZincDataLoader<T> Zinc(ZincDataLoaderOptions options) |
Creates a ZINC molecular dataset data loader (~250K drug-like molecules). |