ParquetDataLoader<T>
Models & Types · AiDotNet.Data.Formats
Reads tabular data from Apache Parquet columnar files using the Parquet.Net library.
Inheritance: DataLoaderBase<T> → InputOutputDataLoaderBase<T, Tensor<T>, Tensor<T>> → ParquetDataLoader<T>
For Beginners
Parquet files are commonly produced by Apache Spark, Pandas, and other data processing frameworks. This loader reads them directly into AiDotNet for training.
How It Works
Parquet is an efficient columnar storage format widely used in data engineering and ML pipelines. This loader reads Parquet files and converts numeric columns into tensors for training. Supports all standard Parquet compression codecs (Snappy, GZIP, ZSTD, LZ4, Brotli) and encodings.
Constructors
| Constructor | Summary |
|---|---|
ParquetDataLoader(ParquetDataLoaderOptions options) |
Creates a new Parquet data loader. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> ResolvedFeatureColumns { get; set; } |
Gets the column names used as features. |
int FeatureCount { get; } |
|
int OutputDimension { get; } |
|
int TotalCount { get; } |
|
string Description { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
ValueTuple<IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>> Split(double trainRatio, double validationRatio, int? seed) |