DistributedStreamingDataLoader<T, TInput, TOutput>
Models & Types · AiDotNet.Data.Loaders
Wraps any IStreamingDataLoader to partition each global batch deterministically
across distributed ranks, supporting both homogeneous and heterogeneous per-rank micro-batch sizes as well as
exact sample-offset fast-forwarding across world-size changes on checkpoint resume.
Implements: IStreamingDataLoader<T, TInput, TOutput>
For Beginners
If GPU 0 (12 GB) uses BatchSize = 2 and GPU 1 (24 GB) uses BatchSize = 6,
DistributedStreamingDataLoader automatically forms a global batch of 8
non-overlapping samples per step—feeding samples 0..1 to GPU 0 and samples 2..7 to GPU 1—without
duplicating data or hanging on incomplete tail batches.
How It Works
At the start of enumeration, ranks exchange their configured local BatchSize values
(B_0, B_1, ..., B_{W-1}) via AllGather so every rank knows
GlobalBatchSize (sum_r B_r). For every global step, the underlying sample stream is
partitioned contiguously across ranks 0..W-1, ensuring the union of per-rank micro-batches equals a
single-process run configured with BatchSize = GlobalBatchSize and the same shuffle seed.
Constructors
| Constructor | Summary |
|---|---|
DistributedStreamingDataLoader(IStreamingDataLoader<T, TInput, TOutput> innerLoader, ICommunicationBackend<T> backend, int? localBatchSize) |
Initializes a new DistributedStreamingDataLoader around an existing streaming loader. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<int> RankBatchSizes { get; } |
Gets the per-rank micro-batch sizes ([B_0, ..., B_{W-1}]) synchronized across the world. |
IStreamingDataLoader<T, TInput, TOutput> InnerLoader { get; } |
Gets the underlying unwrapped streaming data loader. |
bool IsLoaded { get; } |
|
double Progress { get; } |
|
int BatchCount { get; } |
|
int BatchSize { get; set; } |
|
int CurrentBatchIndex { get; } |
|
int CurrentIndex { get; } |
|
int GlobalBatchSize { get; } |
Gets the synchronized global batch size across all ranks (sum_r RankBatchSizes[r]). |
int NumWorkers { get; } |
|
int PrefetchCount { get; } |
|
int Rank { get; } |
Gets this process’s rank in [0, WorldSize - 1]. |
int SampleCount { get; } |
|
int TotalCount { get; } |
|
int WorldSize { get; } |
Gets the number of participating distributed processes. |
long LastBatchGlobalSampleCount { get; } |
Gets the total number of global samples consumed across all ranks in the most recently yielded step. |
string Description { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
IAsyncEnumerable<ValueTuple<TInput[], TOutput[]>> GetBatchesAsync(bool shuffle, bool dropLast, int? seed, CancellationToken cancellationToken) |
|
IEnumerable<ValueTuple<TInput[], TOutput[]>> GetBatches(bool shuffle, bool dropLast, int? seed) |
|
Task LoadAsync(CancellationToken cancellationToken) |
|
void Reset() |
|
void SkipSamplesOnNextEnumeration(long samplesToSkip) |
Schedules an exact global sample count to skip at the start of the next GetBatches or GetBatchesAsync call (used when resuming mid-epoch from a checkpoint across arbitrary world sizes). |
void Unload() |