StatefulDataLoader<T, TInput, TOutput>
Models & Types · AiDotNet.Data.Loaders
Wraps any InMemoryDataLoader with checkpoint/resume support.
Inheritance: DataLoaderBase<T> → StatefulDataLoader<T, TInput, TOutput>
Implements: IInputOutputDataLoader<T, TInput, TOutput>, IStatefulDataLoader<T>
For Beginners
Wrap your data loader with this class to enable saving and restoring the exact position during training:
How It Works
Inspired by PyTorch’s StatefulDataLoader (torchdata 2025), this wrapper adds mid-epoch checkpointing to any in-memory data loader. The state can be serialized and restored for fault-tolerant training on large datasets.
Constructors
| Constructor | Summary |
|---|---|
StatefulDataLoader(InputOutputDataLoaderBase<T, TInput, TOutput> inner) |
Creates a stateful wrapper around an existing data loader. |
Properties
| Property | Summary |
|---|---|
TInput Features { get; } |
|
TOutput Labels { get; } |
|
bool HasNext { get; } |
|
bool IsShuffled { get; } |
|
int Epoch { get; } |
Gets the current epoch number. |
int FeatureCount { get; } |
|
int OutputDimension { get; } |
|
int TotalCount { get; } |
|
string Description { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
DataLoaderCheckpoint GetState() |
|
IAsyncEnumerable<ValueTuple<TInput, TOutput>> GetBatchesAsync(int? batchSize, bool shuffle, bool dropLast, int? seed, int prefetchCount, CancellationToken cancellationToken) |
|
IEnumerable<ValueTuple<TInput, TOutput>> GetBatches(int? batchSize, bool shuffle, bool dropLast, int? seed) |
|
ValueTuple<IInputOutputDataLoader<T, TInput, TOutput>, IInputOutputDataLoader<T, TInput, TOutput>, IInputOutputDataLoader<T, TInput, TOutput>> Split(double trainRatio, double validationRatio, int? seed) |
|
ValueTuple<TInput, TOutput> GetNextBatch() |
|
bool TryGetNextBatch(ValueTuple<TInput, TOutput> batch) |
|
void LoadState(DataLoaderCheckpoint state) |
|
void OnEpochStart() |
Called when starting a new epoch. |
void Shuffle(int? seed) |
|
void Unshuffle() |