ArrowDataset<T>
Models & Types · AiDotNet.Data.Formats
Provides read/write access to datasets in a custom binary columnar format inspired by Apache Arrow IPC.
Implements: IDisposable
How It Works
Important: This is NOT a native Apache Arrow implementation and is NOT compatible with Arrow IPC files created by PyArrow, the Arrow C++ library, or other Arrow tools. It is a custom columnar format designed for efficient column-wise access in AiDotNet ML pipelines. For native Arrow interop, use the Apache.Arrow NuGet package.
The on-disk format:
- Header: [magic: “ARRW” 4 bytes] [version: 4 bytes] [numRows: 4 bytes] [numColumns: 4 bytes]
- Column table: for each column: [nameLen: 4][name: bytes][elementType: 4][elementsPerRow: 4][dataOffset: 8][dataLength: 8]
- Data section: raw values for each column (doubles, 8 bytes each)
Constructors
| Constructor | Summary |
|---|---|
ArrowDataset(ArrowDatasetOptions options) |
Opens an Arrow dataset file. |
Properties
| Property | Summary |
|---|---|
IReadOnlyCollection<string> ColumnNames { get; } |
Gets the column names. |
int NumRows { get; } |
Gets the total number of rows in the dataset. |
Methods
| Method | Summary |
|---|---|
T[] ReadColumn(string columnName) |
Reads an entire column as a flat array. |
T[] ReadColumnSlice(string columnName, int startRow, int numRows) |
Reads a batch of rows from the specified columns. |
ValueTuple<Tensor<T>, Tensor<T>> ReadBatch(int startRow, int numRows) |
Reads feature and label columns as tensors. |
void Dispose() |
|
void WriteFile(string filePath, IReadOnlyDictionary<string, ValueTuple<T[], int>> columns, int numRows) |
Writes a dataset to Arrow format. |