DataPipeline<T>

Models & Types · AiDotNet.Data.Pipeline

Provides TensorFlow-style data pipeline operations for transforming and processing data.

Implements: IEnumerable<T>

For Beginners

Data pipelines let you build complex data processing workflows step by step. Each operation transforms the data in some way:

How It Works

DataPipeline provides a fluent API for building data processing pipelines similar to TensorFlow’s tf.data API. Operations are lazily evaluated and can be chained together.

Constructors

Constructor Summary
DataPipeline(Func<IEnumerable<T>> sourceFactory) Initializes a new instance of the DataPipeline class.

Methods

Method Summary
AsyncDataPipeline<T> Prefetch(int bufferSize) Prefetches elements in the background for improved performance.
AsyncDataPipeline<TResult> MapAsync(Func<T, CancellationToken, Task<TResult>> selector, int maxConcurrency) Applies an async transformation function to each element.
DataPipeline<T> Cache() Caches elements in memory for faster subsequent iterations.
DataPipeline<T> Concat(DataPipeline<T> other) Concatenates another pipeline to this one.
DataPipeline<T> Filter(Func<T, bool> predicate) Filters elements based on a predicate.
DataPipeline<T> ForEach(Action<T> action) Applies a side effect to each element without modifying it.
DataPipeline<T> From(IBatchIterable<T> source) Creates a new DataPipeline from a batch iterable.
DataPipeline<T> From(IEnumerable<T> source) Creates a new DataPipeline from a batch iterable.
DataPipeline<T> Interleave(DataPipeline<T>[] pipelines) Interleaves multiple pipelines.
DataPipeline<T> Repeat(int? count) Repeats the pipeline indefinitely or a specified number of times.
DataPipeline<T> Shuffle(int bufferSize, int? seed) Shuffles elements using a buffer.
DataPipeline<T> Skip(int count) Skips the first N elements.
DataPipeline<T> Take(int count) Takes only the first N elements.
DataPipeline<TElement> Flatten(Func<T, IEnumerable<TElement>> selector) Flattens a pipeline by extracting enumerables from each element.
DataPipeline<TResult> Map(Func<T, TResult> selector) Applies a transformation function to each element.
DataPipeline<T[]> Batch(int batchSize, bool dropRemainder) Groups elements into batches.
DataPipeline<ValueTuple<T, TOther>> Zip(DataPipeline<TOther> other) Zips this pipeline with another to create pairs.
IEnumerator<T> GetEnumerator()