ParquetDataLoader<T>

Models & Types · AiDotNet.Data.Formats

Reads tabular data from Apache Parquet columnar files using the Parquet.Net library.

Inheritance: DataLoaderBase<T> → InputOutputDataLoaderBase<T, Tensor<T>, Tensor<T>> → ParquetDataLoader<T>

For Beginners

Parquet files are commonly produced by Apache Spark, Pandas, and other data processing frameworks. This loader reads them directly into AiDotNet for training.

How It Works

Parquet is an efficient columnar storage format widely used in data engineering and ML pipelines. This loader reads Parquet files and converts numeric columns into tensors for training. Supports all standard Parquet compression codecs (Snappy, GZIP, ZSTD, LZ4, Brotli) and encodings.

Constructors

Constructor Summary
ParquetDataLoader(ParquetDataLoaderOptions options) Creates a new Parquet data loader.

Properties

Property Summary
IReadOnlyList<string> ResolvedFeatureColumns { get; set; } Gets the column names used as features.
int FeatureCount { get; }
int OutputDimension { get; }
int TotalCount { get; }
string Description { get; }
string Name { get; }

Methods

Method Summary
ValueTuple<IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>> Split(double trainRatio, double validationRatio, int? seed)