ShardedStreamingDataLoader<T>

Models & Types · AiDotNet.Data.Formats

A typed data loader facade that wraps ShardedStreamingDataset and implements StreamingDataLoaderBase for IDataLoader compliance.

Inheritance: DataLoaderBase<T> → StreamingDataLoaderBase<T, Tensor<T>, Tensor<T>> → ShardedStreamingDataLoader<T>

For Beginners

Use this when you want to load sharded binary datasets and feed them into AiModelBuilder.ConfigureDataLoader(). You provide a parser function that knows how to decode each binary record into tensors for training.

How It Works

This loader eagerly reads all records from the underlying sharded binary files during loading, caches them in memory, and provides index-based access via a user-supplied parser delegate that converts raw byte[] records into typed tensor pairs.

Constructors

Constructor Summary
ShardedStreamingDataLoader(string[] shardPaths, Func<byte[], ValueTuple<Tensor<T>, Tensor<T>>> recordParser, int batchSize, ShardedStreamingDatasetOptions options) Creates a new ShardedStreamingDataLoader.

Properties

Property Summary
int SampleCount { get; }
string Name { get; }