ShardedStreamingDataLoader<T>
Models & Types · AiDotNet.Data.Formats
A typed data loader facade that wraps ShardedStreamingDataset and implements
StreamingDataLoaderBase for IDataLoader compliance.
Inheritance: DataLoaderBase<T> → StreamingDataLoaderBase<T, Tensor<T>, Tensor<T>> → ShardedStreamingDataLoader<T>
For Beginners
Use this when you want to load sharded binary datasets and
feed them into AiModelBuilder.ConfigureDataLoader(). You provide a parser function
that knows how to decode each binary record into tensors for training.
How It Works
This loader eagerly reads all records from the underlying sharded binary files during loading,
caches them in memory, and provides index-based access via a user-supplied parser delegate
that converts raw byte[] records into typed tensor pairs.
Constructors
| Constructor | Summary |
|---|---|
ShardedStreamingDataLoader(string[] shardPaths, Func<byte[], ValueTuple<Tensor<T>, Tensor<T>>> recordParser, int batchSize, ShardedStreamingDatasetOptions options) |
Creates a new ShardedStreamingDataLoader. |
Properties
| Property | Summary |
|---|---|
int SampleCount { get; } |
|
string Name { get; } |