TextLineDataset<T>

Models & Types ยท AiDotNet.Data.Text

Streams text line-by-line from a file for language modeling tasks.

Inheritance: DataLoaderBase<T> โ†’ TextLineDataset<T>

For Beginners

This loader is for large text files that you want to process line by line. Each line becomes one sample in your dataset.

How It Works

Reads a text file line by line, providing each line as a string sample. Extends DataLoaderBase for lifecycle management while exposing line-reading as an async enumerable.

Constructors

Constructor Summary
TextLineDataset(string filePath, bool skipEmptyLines, int batchSize) Creates a new TextLineDataset that reads from the specified file.

Properties

Property Summary
int TotalCount { get; }
string Description { get; }
string FilePath { get; } Gets the file path being read.
string Name { get; }

Methods

Method Summary
IAsyncEnumerable<string> ReadLinesAsync(CancellationToken cancellationToken) Reads all lines from the text file.
IAsyncEnumerable<string[]> ReadBatchesAsync(int? batchSize, CancellationToken cancellationToken) Reads lines in batches.