ArrowDataset<T>

Models & Types · AiDotNet.Data.Formats

Provides read/write access to datasets in a custom binary columnar format inspired by Apache Arrow IPC.

Implements: IDisposable

How It Works

Important: This is NOT a native Apache Arrow implementation and is NOT compatible with Arrow IPC files created by PyArrow, the Arrow C++ library, or other Arrow tools. It is a custom columnar format designed for efficient column-wise access in AiDotNet ML pipelines. For native Arrow interop, use the Apache.Arrow NuGet package.

The on-disk format:

  • Header: [magic: “ARRW” 4 bytes] [version: 4 bytes] [numRows: 4 bytes] [numColumns: 4 bytes]
  • Column table: for each column: [nameLen: 4][name: bytes][elementType: 4][elementsPerRow: 4][dataOffset: 8][dataLength: 8]
  • Data section: raw values for each column (doubles, 8 bytes each)

Constructors

Constructor Summary
ArrowDataset(ArrowDatasetOptions options) Opens an Arrow dataset file.

Properties

Property Summary
IReadOnlyCollection<string> ColumnNames { get; } Gets the column names.
int NumRows { get; } Gets the total number of rows in the dataset.

Methods

Method Summary
T[] ReadColumn(string columnName) Reads an entire column as a flat array.
T[] ReadColumnSlice(string columnName, int startRow, int numRows) Reads a batch of rows from the specified columns.
ValueTuple<Tensor<T>, Tensor<T>> ReadBatch(int startRow, int numRows) Reads feature and label columns as tensors.
void Dispose()
void WriteFile(string filePath, IReadOnlyDictionary<string, ValueTuple<T[], int>> columns, int numRows) Writes a dataset to Arrow format.