TextLineDataset<T>
Models & Types ยท AiDotNet.Data.Text
Streams text line-by-line from a file for language modeling tasks.
Inheritance: DataLoaderBase<T> โ TextLineDataset<T>
For Beginners
This loader is for large text files that you want to process line by line. Each line becomes one sample in your dataset.
How It Works
Reads a text file line by line, providing each line as a string sample. Extends DataLoaderBase for lifecycle management while exposing line-reading as an async enumerable.
Constructors
| Constructor | Summary |
|---|---|
TextLineDataset(string filePath, bool skipEmptyLines, int batchSize) |
Creates a new TextLineDataset that reads from the specified file. |
Properties
| Property | Summary |
|---|---|
int TotalCount { get; } |
|
string Description { get; } |
|
string FilePath { get; } |
Gets the file path being read. |
string Name { get; } |
Methods
| Method | Summary |
|---|---|
IAsyncEnumerable<string> ReadLinesAsync(CancellationToken cancellationToken) |
Reads all lines from the text file. |
IAsyncEnumerable<string[]> ReadBatchesAsync(int? batchSize, CancellationToken cancellationToken) |
Reads lines in batches. |