DataPreparationPipeline<T>
Models & Types · AiDotNet.Preprocessing.DataPreparation
Chains multiple row operations into a sequential data preparation pipeline.
For Beginners
Before training a model, you might want to:
- Remove outliers (unusual data points that could confuse the model)
- Add synthetic samples (SMOTE) to balance classes
- Split data into train/validation/test sets
This pipeline handles all these operations in sequence, making sure your features and labels stay properly aligned.
How It Works
DataPreparationPipeline handles operations that change the number of rows in a dataset, such as outlier removal, data augmentation (SMOTE), and data splitting. These operations must process both features (X) and labels (y) together to maintain alignment.
Data Preparation vs Data Preprocessing:
- Data Preparation (this pipeline): Changes the NUMBER of rows - outlier removal,
augmentation, splitting. Only happens during training.
- Data Preprocessing: Transforms values WITHOUT changing row count - scaling,
encoding. Happens during both training and prediction.
Pipeline Flow:
Usage: Users interact with this through AiModelBuilder.ConfigureDataPreparation().
Constructors
| Constructor | Summary |
|---|---|
DataPreparationPipeline() |
Creates a new empty data preparation pipeline. |
Properties
| Property | Summary |
|---|---|
IDataSplitter<T> Splitter { get; } |
Gets the configured data splitter, or null if none is set. |
IReadOnlyList<ValueTuple<string, IRowOperation<T>>> Operations { get; } |
Gets the named operations in the pipeline. |
bool HasSplitter { get; } |
Gets whether this pipeline has a splitter configured. |
bool IsFitted { get; } |
Gets whether this pipeline has been fitted to data. |
int Count { get; } |
Gets the number of operations in the pipeline. |
Methods
| Method | Summary |
|---|---|
DataPreparationPipeline<T> Add(IRowOperation<T> operation) |
Adds a row operation to the pipeline. |
DataPreparationPipeline<T> Add(string name, IRowOperation<T> operation) |
Adds a named row operation to the pipeline. |
DataPreparationPipeline<T> WithKFold(int k, bool shuffle, int randomSeed) |
Configures K-Fold cross-validation. |
DataPreparationPipeline<T> WithSplitter(IDataSplitter<T> splitter) |
Configures a custom data splitter for this pipeline. |
DataPreparationPipeline<T> WithStratifiedKFold(int k, bool shuffle, int randomSeed) |
Configures Stratified K-Fold cross-validation that preserves class distribution. |
DataPreparationPipeline<T> WithStratifiedSplit(double testSize, bool shuffle, int randomSeed) |
Configures a stratified train/test split for classification. |
DataPreparationPipeline<T> WithTimeSeriesSplit(int nSplits, int gap) |
Configures a time series split with expanding window (no shuffling). |
DataPreparationPipeline<T> WithTrainTestSplit(double testSize, bool shuffle, int randomSeed) |
Configures a simple train/test split. |
DataPreparationPipeline<T> WithTrainValTestSplit(double trainSize, double validationSize, bool shuffle, int randomSeed) |
Configures a three-way train/validation/test split. |
DataSplitResult<T> FitResampleAndSplit(Matrix<T> X, Vector<T> y) |
Applies row operations and then splits the data. |
IEnumerable<DataSplitResult<T>> FitResampleAndGetSplits(Matrix<T> X, Vector<T> y) |
Applies row operations and returns multiple splits (for cross-validation). |
IEnumerable<TensorSplitResult<T>> FitResampleAndGetTensorSplits(Tensor<T> X, Tensor<T> y) |
Applies row operations and returns multiple tensor splits (for cross-validation). |
IRowOperation<T> GetOperation(string name) |
Gets an operation by name. |
TensorSplitResult<T> FitResampleAndSplitTensor(Tensor<T> X, Tensor<T> y) |
Applies row operations and then splits the tensor data. |
ValueTuple<Matrix<T>, Vector<T>> FitResample(Matrix<T> X, Vector<T> y) |
Fits all operations and applies row modifications to the data. |
ValueTuple<Tensor<T>, Tensor<T>> FitResampleTensor(Tensor<T> X, Tensor<T> y) |
Fits all operations and applies row modifications to tensor data. |
string GetSummary() |
Gets a summary of the pipeline operations and splitter. |
void Clear() |
Clears all operations and splitter from the pipeline. |