DistributedStreamingDataLoader<T, TInput, TOutput>

Models & Types · AiDotNet.Data.Loaders

Wraps any IStreamingDataLoader to partition each global batch deterministically across distributed ranks, supporting both homogeneous and heterogeneous per-rank micro-batch sizes as well as exact sample-offset fast-forwarding across world-size changes on checkpoint resume.

Implements: IStreamingDataLoader<T, TInput, TOutput>

For Beginners

If GPU 0 (12 GB) uses BatchSize = 2 and GPU 1 (24 GB) uses BatchSize = 6, DistributedStreamingDataLoader automatically forms a global batch of 8 non-overlapping samples per step—feeding samples 0..1 to GPU 0 and samples 2..7 to GPU 1—without duplicating data or hanging on incomplete tail batches.

How It Works

At the start of enumeration, ranks exchange their configured local BatchSize values (B_0, B_1, ..., B_{W-1}) via AllGather so every rank knows GlobalBatchSize (sum_r B_r). For every global step, the underlying sample stream is partitioned contiguously across ranks 0..W-1, ensuring the union of per-rank micro-batches equals a single-process run configured with BatchSize = GlobalBatchSize and the same shuffle seed.

Constructors

Constructor Summary
DistributedStreamingDataLoader(IStreamingDataLoader<T, TInput, TOutput> innerLoader, ICommunicationBackend<T> backend, int? localBatchSize) Initializes a new DistributedStreamingDataLoader around an existing streaming loader.

Properties

Property Summary
IReadOnlyList<int> RankBatchSizes { get; } Gets the per-rank micro-batch sizes ([B_0, ..., B_{W-1}]) synchronized across the world.
IStreamingDataLoader<T, TInput, TOutput> InnerLoader { get; } Gets the underlying unwrapped streaming data loader.
bool IsLoaded { get; }
double Progress { get; }
int BatchCount { get; }
int BatchSize { get; set; }
int CurrentBatchIndex { get; }
int CurrentIndex { get; }
int GlobalBatchSize { get; } Gets the synchronized global batch size across all ranks (sum_r RankBatchSizes[r]).
int NumWorkers { get; }
int PrefetchCount { get; }
int Rank { get; } Gets this process’s rank in [0, WorldSize - 1].
int SampleCount { get; }
int TotalCount { get; }
int WorldSize { get; } Gets the number of participating distributed processes.
long LastBatchGlobalSampleCount { get; } Gets the total number of global samples consumed across all ranks in the most recently yielded step.
string Description { get; }
string Name { get; }

Methods

Method Summary
IAsyncEnumerable<ValueTuple<TInput[], TOutput[]>> GetBatchesAsync(bool shuffle, bool dropLast, int? seed, CancellationToken cancellationToken)
IEnumerable<ValueTuple<TInput[], TOutput[]>> GetBatches(bool shuffle, bool dropLast, int? seed)
Task LoadAsync(CancellationToken cancellationToken)
void Reset()
void SkipSamplesOnNextEnumeration(long samplesToSkip) Schedules an exact global sample count to skip at the start of the next GetBatches or GetBatchesAsync call (used when resuming mid-epoch from a checkpoint across arbitrary world sizes).
void Unload()