SnapshotPipeline<T>

Models & Types · AiDotNet.Data.Pipeline

Persists an entire processed pipeline to disk for fast reload across epochs, with automatic invalidation when source data or pipeline configuration changes.

For Beginners

Imagine you have a pipeline that reads images, resizes them, and normalizes pixel values. On the first run, this is slow. SnapshotPipeline saves the processed results so that the second run is instant:

How It Works

Inspired by TensorFlow’s tf.data snapshot pattern. After the first epoch processes data through expensive transforms (decode, resize, augment, tokenize), the entire output is saved to disk. Subsequent epochs load the cached result directly, skipping all preprocessing.

Cache invalidation is automatic: the pipeline configuration and source data metadata are hashed, and the cache is rebuilt when the hash changes.

Constructors

Constructor Summary
SnapshotPipeline(DataPipeline<Tensor<T>> sourcePipeline, DiskCacheOptions options, string pipelineId) Creates a new snapshot pipeline.

Properties

Property Summary
bool IsCacheValid { get; } Gets whether the cache is currently valid and up-to-date.
string ActiveCacheDirectory { get; } Gets the path to the active cache directory, or null if no cache exists.

Methods

Method Summary
CacheInfo GetCacheInfo() Gets information about the current cache state.
DataPipeline<Tensor<T>> GetCachedPipeline(CancellationToken cancellationToken) Gets a pipeline that reads from cache if available, or processes and caches the source pipeline.
void CleanupCache() Cleans up old cache entries across all pipeline caches based on the eviction policy.
void InvalidateCache() Invalidates and removes the cache.
void RebuildCache(CancellationToken cancellationToken) Forces a rebuild of the cache, even if a valid cache exists.