ActiveLearningQueryStrategy |
Selects the most informative unlabeled samples for annotation using active learning. |
ActiveLearningSampler<T> |
A sampler for active learning that selects the most informative samples for labeling. |
Ade20kDataLoader<T> |
Loads the ADE20K semantic segmentation dataset. |
AgNewsDataLoader<T> |
Loads the AG News topic classification dataset (4 classes: World, Sports, Business, Sci/Tech). |
ArcDataLoader<T> |
Loads the AI2 Reasoning Challenge (ARC) multiple-choice science QA benchmark (Clark et al. |
ArrowDataset<T> |
Provides read/write access to datasets in a custom binary columnar format inspired by Apache Arrow IPC. |
AsyncDataPipeline<T> |
Provides async data pipeline operations with prefetching support. |
AudioFileDataset<T> |
Loads audio files from directories for audio classification and processing tasks. |
AudioSetDataLoader<T> |
Loads the AudioSet large-scale audio event dataset (2M+ 10-second clips, 527 categories). |
BalancedEpisodicDataLoader<T, TInput, TOutput> |
Provides balanced episodic task sampling that ensures equal class representation across multiple tasks. |
BigEarthNetDataLoader<T> |
Loads the BigEarthNet multi-label remote sensing dataset (590K Sentinel-2 patches, 19 or 43 classes). |
BucketBatchSampler |
Groups sequences by length into buckets, then batches within each bucket to minimize padding. |
CacheInfo |
Information about the current state of a pipeline cache. |
CachingDataLoader<TKey, TValue> |
Wraps data loading with an in-memory cache to avoid redundant I/O. |
Caltech101DataLoader<T> |
Loads the Caltech-101 image classification dataset (Fei-Fei et al. |
CelebADataLoader<T> |
Loads the CelebA face attributes dataset. |
CheXpertDataLoader<T> |
Loads the CheXpert chest radiograph dataset (224K images, 14 observations with uncertainty). |
CheckpointData |
Data stored in a mid-epoch checkpoint. |
ChestXray14DataLoader<T> |
Loads the NIH Chest X-ray 14 multi-label classification dataset (112K images, 14 disease labels). |
Cifar100DataLoader<T> |
Loads the CIFAR-100 image classification dataset (50k train / 10k test, 32x32 RGB, 100 fine / 20 coarse classes). |
Cifar10DataLoader<T> |
Loads the CIFAR-10 image classification dataset (50k train / 10k test, 32x32 RGB, 10 classes). |
CitationNetworkLoader<T> |
Loads citation network datasets (Cora, CiteSeer, PubMed) for node classification. |
CityscapesDataLoader<T> |
Loads the Cityscapes semantic-segmentation dataset (Cordts et al. |
CnnDailyMailDataLoader<T> |
Loads the CNN/DailyMail abstractive-summarization dataset v3.0.0 (Hermann et al. |
CocoDetectionDataLoader<T> |
Loads the COCO 2017 object detection dataset (118K train / 5K val, 80 categories). |
CommonVoiceDataLoader<T> |
Loads the Mozilla Common Voice multilingual speech dataset (19K+ hours, 100+ languages). |
Compose<T> |
Chains multiple transforms of the same type into a single transform. |
CoresetSelector |
Selects a representative coreset from a dataset using distance-based strategies. |
CsvDataLoader<T> |
Loads supervised learning data from CSV files into Matrix/Vector format. |
CsvStreamingDataLoader<T, TInput, TOutput> |
A streaming data loader that reads from a CSV file line by line. |
CurriculumDataScheduler |
Schedules training data presentation order and pacing for curriculum learning. |
CurriculumEpisodicDataLoader<T, TInput, TOutput> |
Provides curriculum-based episodic task sampling that progressively increases task difficulty during training. |
CurriculumSampler<T> |
A sampler that implements curriculum learning by progressively introducing harder samples. |
DataPipeline<T> |
Provides TensorFlow-style data pipeline operations for transforming and processing data. |
DataPruner |
Prunes (removes) training samples based on difficulty/importance scores. |
DatasetDistiller |
Performs dataset distillation to synthesize a compact representative dataset. |
DatasetMixer<T> |
Blends multiple datasets by weight ratios, producing mixed batches for curriculum or domain mixing. |
DefaultCollateFunction<T> |
Stacks equal-size tensors into a batch tensor along dimension 0. |
DenseSampler |
Dense sampling: uniformly samples frames at regular intervals from the video. |
DistributedBucketSampler |
Combines distributed partitioning with bucket batching for efficient distributed NLP training. |
DistributedSampler |
Partitions dataset indices across N ranks for distributed (multi-GPU/multi-node) training. |
DistributedStreamingDataLoader<T, TInput, TOutput> |
Wraps any IStreamingDataLoader to partition each global batch deterministically across distributed ranks, supporting both homogeneous and heterogeneous per-rank micro-batch sizes as well as exact sample-offset fast-forwarding across world… |
DocVqaDataLoader<T> |
Loads the DocVQA document visual question answering dataset. |
DomainMixingSampler |
Samples from multiple data domains with configurable mixing ratios for multi-domain LLM training. |
DtdDataLoader<T> |
Loads the Describable Textures Dataset (DTD; Cimpoi et al. |
DynamicBatchSampler |
Creates batches that fit a maximum number of tokens/elements rather than a fixed number of samples. |
ElasticDistributedSampler |
Distributed sampler that evenly divides data across multiple workers with elastic scaling support. |
EnvironmentDataLoader<T> |
Data loader for reinforcement learning that wraps an environment for experience collection. |
Enwik8DataLoader<T> |
Loads the enwik8 character-level Wikipedia language modeling benchmark (first 100M bytes of an English Wikipedia XML dump). |
Esc50DataLoader<T> |
Loads the ESC-50 environmental sound classification dataset (2000 clips, 50 classes). |
EuroSatDataLoader<T> |
Loads the EuroSAT land use/land cover classification dataset (27K patches, 64x64 RGB, 10 classes). |
ExactHashDeduplicator |
Detects exact duplicate documents using cryptographic hashing (SHA-256). |
FMoWDataLoader<T> |
Loads the Functional Map of the World (fMoW) satellite imagery dataset (1M+ images, 62 categories). |
FashionMnistDataLoader<T> |
Loads the Fashion-MNIST clothing classification dataset (60k train / 10k test, 28x28 grayscale). |
FileStreamingDataLoader<T, TInput, TOutput> |
A streaming data loader that reads from files in a directory. |
FleursDataLoader<T> |
Loads the FLEURS multilingual speech benchmark (102 languages, ~12 hours per language). |
Flowers102DataLoader<T> |
Loads the Oxford Flowers-102 dataset (Nilsback & Zisserman 2008) — 102 fine-grained flower species. |
Food101DataLoader<T> |
Loads the Food-101 fine-grained classification dataset (Bossard et al. |
Fsd50kDataLoader<T> |
Loads the FSD50K audio event dataset (51,197 clips, 200 sound event classes). |
GigaSpeechDataLoader<T> |
Loads the GigaSpeech multi-domain English ASR dataset (10K hours from audiobooks, podcasts, YouTube). |
GlueDataLoader<T> |
Loads GLUE benchmark sub-tasks (CoLA, SST-2, MRPC, QQP, STS-B, MNLI, QNLI, RTE, WNLI). |
GraphClassificationTask<T> |
Represents a graph classification task where the goal is to classify entire graphs. |
GraphData<T> |
Represents a single graph with nodes, edges, features, and optional labels. |
GraphGenerationTask<T> |
Represents a graph generation task where the goal is to generate new valid graphs. |
Gsm8kDataLoader<T> |
Loads the GSM8K grade-school math word-problem dataset (Cobbe et al. |
GtzanDataLoader<T> |
Loads the GTZAN music genre classification dataset (Tzanetakis & Cook 2002). |
Hdf5Dataset<T> |
Provides read/write access to datasets in a custom binary format for named multidimensional arrays, inspired by the HDF5 data model. |
HellaswagDataLoader<T> |
Loads the HellaSwag 4-way commonsense NLI benchmark (Zellers et al. |
HeuristicTextFilter |
Filters text documents using simple heuristic rules for quality assessment. |
Hmdb51DataLoader<T> |
Loads the HMDB51 action recognition dataset (6,766 clips, 51 classes). |
HumanEvalDataLoader<T> |
Loads the HumanEval Python code-generation benchmark (Chen et al. |
INaturalistDataLoader<T> |
Loads the iNaturalist species classification dataset (~2.7M images, 10,000 species in 2021 version). |
IdentityTransform<T> |
A no-op transform that passes input through unchanged. |
ImageClassificationDataset<T> |
An in-memory image classification dataset with an optional composable transform pipeline. |
ImageFolderDataset<T> |
Loads images from a directory structure where each subdirectory is a class label. |
ImageNet1kDataLoader<T> |
Loads the ImageNet-1K (ILSVRC 2012) image classification dataset (~1.28M train / 50K val, 1000 classes). |
ImageNet21kDataLoader<T> |
Loads the ImageNet-21K dataset (~14.2M images, 21,841 categories from the full WordNet hierarchy). |
ImageQualityFilter |
Filters images based on resolution, aspect ratio, and pixel statistics. |
Imdb50kDataLoader<T> |
Loads the IMDB 50k movie review sentiment analysis dataset (25k train / 25k test, binary classification). |
ImportanceSampler<T> |
A sampler that implements importance sampling for variance reduction. |
InMemoryDataLoader<T, TInput, TOutput> |
A simple in-memory data loader for supervised learning data. |
InterleavedDataset<T> |
A dataset where each sample is an interleaved sequence of modality segments, such as alternating image-text-image-text in vision-language models. |
InterleavedSegment<T> |
A single segment within an interleaved sequence. |
InterleavedSequence<T> |
A single interleaved sequence containing ordered segments of different modalities. |
JsonlDataLoader<T> |
A typed data loader facade that wraps JsonlStreamingLoader and implements StreamingDataLoaderBase for IDataLoader compliance. |
Kinetics400DataLoader<T> |
Loads the Kinetics-400 human action recognition dataset (~300K clips, 400 classes). |
KittiDataLoader<T> |
Loads the KITTI 3D object detection dataset (LiDAR point clouds with 3D bounding boxes). |
LambdaTransform<TInput, TOutput> |
Wraps a Func delegate as an ITransform. |
LanguageIdFilter |
Filters documents based on detected language using character n-gram profiles. |
LeafFederatedDataLoader<T> |
Data loader that reads LEAF benchmark JSON splits and exposes both aggregated (X, Y) data and per-client partitions. |
LibriSpeechDataLoader<T> |
Loads the LibriSpeech automatic speech recognition dataset (~1000 hours of 16kHz English speech). |
LinkPredictionTask<T> |
Represents a link prediction task where the goal is to predict missing or future edges. |
LjSpeechDataLoader<T> |
Loads the LJSpeech 1.1 single-speaker TTS corpus (Ito & Johnson 2017). |
LmdbDataset<T> |
Provides read-only access to datasets stored in a custom binary key-value format inspired by LMDB (Lightning Memory-Mapped Database). |
M4DatasetLoader<T> |
Loads time series datasets from the M4 Competition for benchmarking forecasting models. |
M4TimeSeries<T> |
Represents a single time series from the M4 Competition. |
MaestroDataLoader<T> |
Loads the MAESTRO piano performance dataset (~200 hours, aligned MIDI and audio). |
MathDataLoader<T> |
Loads the Hendrycks MATH benchmark — competition math problems (Hendrycks et al. |
MbppDataLoader<T> |
Loads the MBPP (Mostly Basic Python Problems) benchmark — 1,000 entry-level Python coding problems with natural-language descriptions and unit tests (Austin et al. |
MemoryMappedDataset<T> |
Memory-mapped dataset access for efficient I/O on large binary datasets. |
MemoryMappedStreamingDataLoader<T, TInput, TOutput> |
A streaming data loader that uses memory-mapped files for efficient random access to large binary datasets. |
MetaLearningTask<T, TInput, TOutput> |
Represents a single meta-learning task for few-shot learning, containing support and query sets. |
MidEpochCheckpointer |
Saves and restores training state mid-epoch for fault tolerance. |
MinHashDeduplicator |
Detects near-duplicate documents using MinHash with Locality-Sensitive Hashing (LSH). |
MinMaxScaleTransform<T> |
Scales values to a target range using min-max normalization. |
MmluDataLoader<T> |
Loads MMLU — Massive Multitask Language Understanding (Hendrycks et al. |
MnistDataLoader<T> |
Loads the MNIST handwritten digit classification dataset (60k train / 10k test, 28x28 grayscale). |
ModalitySample<T> |
Represents a single modality’s data within a multimodal sample. |
ModelNet40ClassificationDataLoader<T> |
Loads the ModelNet40 point cloud classification dataset. |
MolecularDatasetLoader<T> |
Loads molecular graph datasets (ZINC, QM9) for graph-level property prediction and generation. |
MultiSourceMixer<TItem> |
Mixes multiple data sources with configurable weights for multi-domain training. |
MultimodalDataset<T> |
A dataset of multimodal samples for training models that process multiple data types simultaneously. |
MultimodalSample<T> |
A collection of modality samples representing one multimodal data point. |
Musdb18DataLoader<T> |
Loads the MUSDB18 music source separation dataset (150 tracks, 4 stems: vocals, drums, bass, other). |
NodeClassificationTask<T> |
Represents a node classification task where the goal is to predict labels for individual nodes. |
NormalizeTransform<T> |
Normalizes an array of values using mean and standard deviation: (x - mean) / std. |
NsynthDataLoader<T> |
Loads the NSynth Neural Synth dataset (Engel et al. |
NuScenesDataLoader<T> |
Loads the nuScenes dataset (LiDAR point clouds with 3D bounding box annotations). |
OGBDatasetLoader<T> |
Loads datasets from the Open Graph Benchmark (OGB) for standardized evaluation. |
OneHotEncodeTransform<T> |
Converts a class index (integer label) to a one-hot encoded vector. |
OpenImagesDataLoader<T> |
Loads the Open Images V7 object detection dataset (~9M images, 600 categories). |
OxfordPetsDataLoader<T> |
Loads the Oxford-IIIT Pet dataset (Parkhi et al. |
PackedSequenceBatch<T> |
Represents a batch of packed (non-padded) variable-length sequences. |
PackedSequenceCollateFunction<T> |
Packs variable-length sequences into a contiguous tensor without padding, along with sequence lengths for reconstruction. |
PaddingCollateFunction<T> |
Pads variable-length sequences to the maximum length in the batch, then stacks them. |
ParallelBatchLoader<TBatch> |
Provides parallel batch loading with multiple workers for improved throughput. |
ParquetDataLoader<T> |
Reads tabular data from Apache Parquet columnar files using the Parquet.Net library. |
PascalVocDataLoader<T> |
Loads the Pascal VOC object detection dataset (20 categories, XML annotations). |
PennTreebankDataLoader<T> |
Loads the Penn Treebank language modeling dataset (Mikolov-preprocessed split). |
PerplexityFilter |
Filters documents based on perplexity scores from a simple n-gram language model. |
Places365DataLoader<T> |
Loads the Places365 scene recognition dataset (1.8M train / 36.5K val, 256x256 RGB, 365 scene categories). |
PrefetchDataLoader<TBatch> |
Wraps a batch-producing function with asynchronous prefetching for pipelined data loading. |
ProteinDataLoader<T> |
Loads protein structure datasets as flattened feature/label tensors for graph-level classification. |
PubLayNetDataLoader<T> |
Loads the PubLayNet document layout analysis dataset. |
Qm9DataLoader<T> |
Thin wrapper around MolecularDatasetLoader for the QM9 dataset. |
RandomSampler |
A sampler that randomly shuffles the dataset indices each epoch. |
RetinalFundusDataLoader<T> |
Loads retinal fundus photography datasets for diabetic retinopathy detection (5-class grading). |
ScanNetSemanticSegmentationDataLoader<T> |
Loads the ScanNet semantic segmentation dataset. |
SelfPacedSampler<T> |
A sampler that implements self-paced learning with automatic difficulty adjustment. |
SemanticDeduplicator |
Detects semantic duplicates using embedding cosine similarity. |
SemanticKittiDataLoader<T> |
Loads the SemanticKITTI dataset (per-point semantic labels for LiDAR point clouds). |
SequencePackingCollateFunction<T> |
Packs multiple variable-length sequences into fixed-length blocks for efficient LLM training. |
SequentialSampler |
A sampler that returns indices in sequential order without shuffling. |
ShapeNetCorePartSegmentationDataLoader<T> |
Loads the ShapeNetCore part segmentation dataset. |
ShardedStreamingDataLoader<T> |
A typed data loader facade that wraps ShardedStreamingDataset and implements StreamingDataLoaderBase for IDataLoader compliance. |
SkinLesionDataLoader<T> |
Loads the ISIC Skin Lesion classification dataset (~25K images, 8 diagnostic categories). |
SlowFastSampler |
SlowFast sampling: produces two sets of frame indices at different temporal resolutions for SlowFast networks (Feichtenhofer et al., 2019). |
SnapshotPipeline<T> |
Persists an entire processed pipeline to disk for fast reload across epochs, with automatic invalidation when source data or pipeline configuration changes. |
SomethingSomethingV2DataLoader<T> |
Loads the Something-Something V2 action recognition dataset (220K clips, 174 classes). |
SpecAugmentTransform<T> |
Applies SpecAugment data augmentation (Park et al., 2019) to spectrogram tensors. |
SpectrogramTransform<T> |
Transforms raw audio waveform tensors into Mel spectrogram representations. |
SpeechCommandsDataLoader<T> |
Loads the Google Speech Commands v2 dataset (~65K clips, 35 words, 16kHz). |
SquadDataLoader<T> |
Loads the SQuAD question answering dataset (100K+ Q&A pairs on Wikipedia articles). |
StandardScaleTransform<T> |
Applies Z-score normalization: (x - mean) / std, computing mean and std from a reference dataset. |
StanfordCarsDataLoader<T> |
Loads the Stanford Cars dataset (Krause et al. |
StatefulDataLoader<T, TInput, TOutput> |
Wraps any InMemoryDataLoader with checkpoint/resume support. |
Stl10DataLoader<T> |
Loads the STL-10 image classification dataset (Coates et al. |
StratifiedBatchSampler |
A batch sampler that ensures each batch contains samples from all classes. |
StratifiedEpisodicDataLoader<T, TInput, TOutput> |
Provides stratified episodic task sampling that maintains dataset class proportions across tasks. |
StratifiedSampler |
A sampler that ensures each class is represented proportionally in each epoch. |
StreamingDataLoader<T, TInput, TOutput> |
A data loader that streams data from disk or other sources without loading all data into memory. |
StreamingTextDataset<T> |
A streaming text dataset that lazily reads and tokenizes text files for language model training. |
SubsetSampler |
A sampler that returns a subset of indices. |
SuperGlueDataLoader<T> |
Loads SuperGLUE benchmark sub-tasks (BoolQ, CB, COPA, MultiRC, ReCoRD, RTE, WiC, WSC). |
SvhnDataLoader<T> |
Loads the SVHN (Street View House Numbers) Format-2 dataset (Netzer et al. |
TaskBatch<T, TInput, TOutput> |
Represents a batch of tasks for meta-learning with advanced batching strategies. |
TemporalGraphDataLoader<T> |
Loads temporal graph datasets (timestamped interactions for dynamic link prediction). |
TemporalJitterAugmentation<T> |
Applies temporal jitter to video frame sequences by randomly shifting frame indices. |
TemporalSegmentSampler |
Temporal Segment Networks (TSN) sampling: divides video into equal segments and samples one frame per segment. |
TextLineDataset<T> |
Streams text line-by-line from a file for language modeling tasks. |
TimitDataLoader<T> |
Loads the TIMIT acoustic-phonetic continuous-speech corpus (Garofolo et al. |
TinyImageNetDataLoader<T> |
Loads the Tiny ImageNet 200-class image-classification dataset (500 train + 50 val + 50 test per class at 64×64). |
TinyStoriesDataLoader<T> |
Loads the TinyStories synthetic LM corpus (Eldan & Li, 2023): ≈ 2.1M GPT-generated short stories with deliberately small vocabulary for small-scale language-model research. |
ToTensorTransform<T> |
Converts a flat array to a Tensor with the specified shape. |
TokenizedTextDataset<T> |
In-memory dataset of pre-tokenized text sequences for language model training. |
TransformedDataLoader<T> |
Wraps a data loader and applies a composable transform pipeline to feature data during batch extraction. |
TruthfulQaDataLoader<T> |
Loads the TruthfulQA benchmark (Lin et al. |
Ucf101DataLoader<T> |
Loads the UCF101 action recognition dataset (13,320 clips, 101 classes). |
UniformEpisodicDataLoader<T, TInput, TOutput> |
Provides uniform random episodic task sampling for N-way K-shot meta-learning scenarios. |
UrbanSound8kDataLoader<T> |
Loads the UrbanSound8K environmental sound dataset (8732 clips, 10 classes, 10 folds). |
VctkDataLoader<T> |
Loads the VCTK Corpus 0.92 multi-speaker TTS dataset. |
VideoFrameDataset<T> |
Loads videos represented as directories of sequentially numbered image frames. |
VoxPopuliDataLoader<T> |
Loads the VoxPopuli multilingual speech dataset from European Parliament recordings. |
WaymoDataLoader<T> |
Loads the Waymo Open Dataset (LiDAR point clouds with 3D bounding boxes). |
WebDatasetDataLoader<T> |
A typed data loader facade that wraps WebDataset and implements StreamingDataLoaderBase for IDataLoader compliance. |
WeightedSampler<T> |
A sampler that samples indices based on their weights. |
WikiText103DataLoader<T> |
Loads the WikiText-103 language modeling dataset (100M+ tokens from Wikipedia articles). |
WikiText2DataLoader<T> |
Loads the WikiText-2 language modeling dataset (≈ 2M tokens train, ≈ 245k val, ≈ 281k test from Wikipedia articles). |
Wikidata5mDataLoader<T> |
Loads Wikidata5M knowledge graph triplets as tensor features and labels. |
XSumDataLoader<T> |
Loads the XSum extreme abstractive-summarization dataset (Narayan et al. |
ZincDataLoader<T> |
Thin wrapper around MolecularDatasetLoader for the ZINC dataset. |