DataPipeline<T>
Models & Types · AiDotNet.Data.Pipeline
Provides TensorFlow-style data pipeline operations for transforming and processing data.
Implements: IEnumerable<T>
For Beginners
Data pipelines let you build complex data processing workflows step by step. Each operation transforms the data in some way:
How It Works
DataPipeline provides a fluent API for building data processing pipelines similar to TensorFlow’s tf.data API. Operations are lazily evaluated and can be chained together.
Constructors
| Constructor | Summary |
|---|---|
DataPipeline(Func<IEnumerable<T>> sourceFactory) |
Initializes a new instance of the DataPipeline class. |
Methods
| Method | Summary |
|---|---|
AsyncDataPipeline<T> Prefetch(int bufferSize) |
Prefetches elements in the background for improved performance. |
AsyncDataPipeline<TResult> MapAsync(Func<T, CancellationToken, Task<TResult>> selector, int maxConcurrency) |
Applies an async transformation function to each element. |
DataPipeline<T> Cache() |
Caches elements in memory for faster subsequent iterations. |
DataPipeline<T> Concat(DataPipeline<T> other) |
Concatenates another pipeline to this one. |
DataPipeline<T> Filter(Func<T, bool> predicate) |
Filters elements based on a predicate. |
DataPipeline<T> ForEach(Action<T> action) |
Applies a side effect to each element without modifying it. |
DataPipeline<T> From(IBatchIterable<T> source) |
Creates a new DataPipeline from a batch iterable. |
DataPipeline<T> From(IEnumerable<T> source) |
Creates a new DataPipeline from a batch iterable. |
DataPipeline<T> Interleave(DataPipeline<T>[] pipelines) |
Interleaves multiple pipelines. |
DataPipeline<T> Repeat(int? count) |
Repeats the pipeline indefinitely or a specified number of times. |
DataPipeline<T> Shuffle(int bufferSize, int? seed) |
Shuffles elements using a buffer. |
DataPipeline<T> Skip(int count) |
Skips the first N elements. |
DataPipeline<T> Take(int count) |
Takes only the first N elements. |
DataPipeline<TElement> Flatten(Func<T, IEnumerable<TElement>> selector) |
Flattens a pipeline by extracting enumerables from each element. |
DataPipeline<TResult> Map(Func<T, TResult> selector) |
Applies a transformation function to each element. |
DataPipeline<T[]> Batch(int batchSize, bool dropRemainder) |
Groups elements into batches. |
DataPipeline<ValueTuple<T, TOther>> Zip(DataPipeline<TOther> other) |
Zips this pipeline with another to create pairs. |
IEnumerator<T> GetEnumerator() |