IDataVersionControl<T>
Interfaces · AiDotNet.Interfaces
Defines the contract for data version control systems that track dataset changes over time.
Known implementations: DataVersionControlBase<T>
How It Works
A data version control system manages versions of datasets used for training and evaluating models, ensuring reproducibility and traceability.
For Beginners: Think of data version control like Git, but for your datasets instead of code. Just like Git tracks changes to your code, data version control tracks changes to your data:
- Records what data was used to train each model
- Lets you go back to previous versions of datasets
- Helps reproduce experiments with exact same data
- Tracks where data came from and how it was transformed
Common scenarios include:
- Dataset updates (new examples added, errors corrected)
- Data preprocessing changes (different normalization, feature engineering)
- Train/validation/test splits that need to be reproduced
- Tracking data lineage for compliance
Why data version control matters:
- Models trained on different data versions perform differently
- Reproducing results requires exact same data
- Debugging requires knowing what data was used
- Compliance and auditing need data traceability
- Collaboration requires shared understanding of data versions
Methods
| Method | Summary |
|---|---|
DatasetComparison<T> CompareDatasetVersions(string datasetName, string version1Hash, string version2Hash) |
Compares two dataset versions to see what changed. |
DatasetLineage GetDatasetLineage(string datasetName, string versionHash) |
Gets the lineage information for a dataset version. |
DatasetSnapshot GetDatasetSnapshot(string snapshotName) |
Retrieves a dataset snapshot. |
DatasetStatistics<T> GetDatasetStatistics(string datasetName, string versionHash) |
Gets statistics about a dataset version. |
DatasetVersion<T> GetDatasetByTag(string datasetName, string tag) |
Gets a dataset version by its tag. |
DatasetVersion<T> GetDatasetForRun(string runId) |
Gets the dataset version used by a specific training run. |
DatasetVersion<T> GetDatasetVersion(string datasetName, string versionHash) |
Retrieves a specific version of a dataset. |
DatasetVersion<T> GetLatestDatasetVersion(string datasetName) |
Gets the latest version of a dataset. |
List<DatasetVersionInfo<T>> ListDatasetVersions(string datasetName) |
Lists all versions of a dataset. |
List<string> GetRunsUsingDataset(string datasetName, string versionHash) |
Gets all training runs that used a specific dataset version. |
List<string> ListDatasets(string filter, Dictionary<string, string> tags) |
Lists all tracked datasets. |
bool VerifyDatasetIntegrity(string datasetName, string versionHash, string currentDataPath) |
Verifies that a dataset hasn’t been modified by comparing its hash. |
string ComputeDatasetHash(string dataPath) |
Computes and stores a hash of the dataset for integrity verification. |
string CreateDatasetSnapshot(string snapshotName, Dictionary<string, string> datasets, string description) |
Creates a snapshot of multiple related datasets together. |
string CreateDatasetVersion(string datasetName, string dataPath, string description, Dictionary<string, object> metadata, Dictionary<string, string> tags) |
Creates a new dataset version. |
void DeleteDatasetVersion(string datasetName, string versionHash) |
Deletes a specific dataset version. |
void LinkDatasetToRun(string datasetName, string versionHash, string runId, string modelId) |
Links a dataset version to a model training run. |
void RecordDatasetLineage(string datasetName, string versionHash, DatasetLineage lineage) |
Records metadata about how a dataset was created or transformed. |
void TagDatasetVersion(string datasetName, string versionHash, string tag) |
Tags a dataset version for easy reference. |