DistributedSampler

Models & Types · AiDotNet.Data.Sampling

Partitions dataset indices across N ranks for distributed (multi-GPU/multi-node) training.

Inheritance: DataSamplerBase → DistributedSampler

How It Works

Each rank sees a disjoint 1/N subset of the data. Optionally pads the dataset so all ranks receive exactly the same number of samples (required when using AllReduce-style gradient synchronization). Equivalent to PyTorch’s DistributedSampler.

At each epoch, the sampler shuffles the full dataset deterministically using the epoch number as the seed modifier, ensuring all ranks agree on the same shuffle order.

Constructors

Constructor Summary
DistributedSampler(int datasetSize, int numReplicas, int rank, bool shuffle, bool dropLast, int? seed) Creates a new distributed sampler.

Properties

Property Summary
int Length { get; }
int NumSamplesPerRank { get; set; } Number of samples assigned to this rank.