IDataVersionControl<T>

Interfaces · AiDotNet.Interfaces

Defines the contract for data version control systems that track dataset changes over time.

Known implementations: DataVersionControlBase<T>

How It Works

A data version control system manages versions of datasets used for training and evaluating models, ensuring reproducibility and traceability.

For Beginners: Think of data version control like Git, but for your datasets instead of code. Just like Git tracks changes to your code, data version control tracks changes to your data:

  • Records what data was used to train each model
  • Lets you go back to previous versions of datasets
  • Helps reproduce experiments with exact same data
  • Tracks where data came from and how it was transformed

Common scenarios include:

  • Dataset updates (new examples added, errors corrected)
  • Data preprocessing changes (different normalization, feature engineering)
  • Train/validation/test splits that need to be reproduced
  • Tracking data lineage for compliance

Why data version control matters:

  • Models trained on different data versions perform differently
  • Reproducing results requires exact same data
  • Debugging requires knowing what data was used
  • Compliance and auditing need data traceability
  • Collaboration requires shared understanding of data versions

Methods

Method Summary
DatasetComparison<T> CompareDatasetVersions(string datasetName, string version1Hash, string version2Hash) Compares two dataset versions to see what changed.
DatasetLineage GetDatasetLineage(string datasetName, string versionHash) Gets the lineage information for a dataset version.
DatasetSnapshot GetDatasetSnapshot(string snapshotName) Retrieves a dataset snapshot.
DatasetStatistics<T> GetDatasetStatistics(string datasetName, string versionHash) Gets statistics about a dataset version.
DatasetVersion<T> GetDatasetByTag(string datasetName, string tag) Gets a dataset version by its tag.
DatasetVersion<T> GetDatasetForRun(string runId) Gets the dataset version used by a specific training run.
DatasetVersion<T> GetDatasetVersion(string datasetName, string versionHash) Retrieves a specific version of a dataset.
DatasetVersion<T> GetLatestDatasetVersion(string datasetName) Gets the latest version of a dataset.
List<DatasetVersionInfo<T>> ListDatasetVersions(string datasetName) Lists all versions of a dataset.
List<string> GetRunsUsingDataset(string datasetName, string versionHash) Gets all training runs that used a specific dataset version.
List<string> ListDatasets(string filter, Dictionary<string, string> tags) Lists all tracked datasets.
bool VerifyDatasetIntegrity(string datasetName, string versionHash, string currentDataPath) Verifies that a dataset hasn’t been modified by comparing its hash.
string ComputeDatasetHash(string dataPath) Computes and stores a hash of the dataset for integrity verification.
string CreateDatasetSnapshot(string snapshotName, Dictionary<string, string> datasets, string description) Creates a snapshot of multiple related datasets together.
string CreateDatasetVersion(string datasetName, string dataPath, string description, Dictionary<string, object> metadata, Dictionary<string, string> tags) Creates a new dataset version.
void DeleteDatasetVersion(string datasetName, string versionHash) Deletes a specific dataset version.
void LinkDatasetToRun(string datasetName, string versionHash, string runId, string modelId) Links a dataset version to a model training run.
void RecordDatasetLineage(string datasetName, string versionHash, DatasetLineage lineage) Records metadata about how a dataset was created or transformed.
void TagDatasetVersion(string datasetName, string versionHash, string tag) Tags a dataset version for easy reference.