StatefulDataLoader<T, TInput, TOutput>

Models & Types · AiDotNet.Data.Loaders

Wraps any InMemoryDataLoader with checkpoint/resume support.

Inheritance: DataLoaderBase<T> → StatefulDataLoader<T, TInput, TOutput>

Implements: IInputOutputDataLoader<T, TInput, TOutput>, IStatefulDataLoader<T>

For Beginners

Wrap your data loader with this class to enable saving and restoring the exact position during training:

How It Works

Inspired by PyTorch’s StatefulDataLoader (torchdata 2025), this wrapper adds mid-epoch checkpointing to any in-memory data loader. The state can be serialized and restored for fault-tolerant training on large datasets.

Constructors

Constructor Summary
StatefulDataLoader(InputOutputDataLoaderBase<T, TInput, TOutput> inner) Creates a stateful wrapper around an existing data loader.

Properties

Property Summary
TInput Features { get; }
TOutput Labels { get; }
bool HasNext { get; }
bool IsShuffled { get; }
int Epoch { get; } Gets the current epoch number.
int FeatureCount { get; }
int OutputDimension { get; }
int TotalCount { get; }
string Description { get; }
string Name { get; }

Methods

Method Summary
DataLoaderCheckpoint GetState()
IAsyncEnumerable<ValueTuple<TInput, TOutput>> GetBatchesAsync(int? batchSize, bool shuffle, bool dropLast, int? seed, int prefetchCount, CancellationToken cancellationToken)
IEnumerable<ValueTuple<TInput, TOutput>> GetBatches(int? batchSize, bool shuffle, bool dropLast, int? seed)
ValueTuple<IInputOutputDataLoader<T, TInput, TOutput>, IInputOutputDataLoader<T, TInput, TOutput>, IInputOutputDataLoader<T, TInput, TOutput>> Split(double trainRatio, double validationRatio, int? seed)
ValueTuple<TInput, TOutput> GetNextBatch()
bool TryGetNextBatch(ValueTuple<TInput, TOutput> batch)
void LoadState(DataLoaderCheckpoint state)
void OnEpochStart() Called when starting a new epoch.
void Shuffle(int? seed)
void Unshuffle()