JsonLinesProposalProvenanceSink
Models & Types ยท AiDotNet.Evolution.Programs.Provenance
Writes provenance records as JSON Lines into a directory of crash-safe, immutable segment files.
Implements: IDisposable, IProposalProvenanceSink
For Beginners
This writes the notes about each AI conversation to disk, one JSON object per line,
in small files inside a folder you choose. Small files are used instead of one big one so that a crash or a
power cut can never leave you with a corrupted half-line. Point ProposalProvenanceReader at the same
folder afterwards to read everything back. Remember to dispose the sink (or await FlushAsync) when the
run finishes, or the last few notes stay in memory.
How It Works
Records are buffered and written in batches. Each batch becomes one new segment, written with the same idiom the checkpoint store uses: serialize into a temporary file in the target directory, flush it all the way to disk, then move it into place under its final name. A crash therefore leaves every finished segment intact and at worst discards the batch that was in flight โ never a half-written line that would poison a reader.
Segments are never rewritten, which is what makes this affordable. Appending to one growing file cannot be made crash-safe without either rewriting the whole file (quadratic in the number of records) or accepting torn writes; a new immutable file per batch gives durability at linear cost, and a reader simply reads the segments in name order. Segment numbering resumes from the highest existing segment, so restarting a run into the same directory adds to the stream rather than overwriting it.
The sink never blocks a search. Writes happen under one lock so a single instance may be shared by concurrent
workers, and the operator that feeds it treats a failure here as a recording problem rather than a run-ending
one. Call FlushAsync or dispose the sink at the end of a run so the final partial batch reaches
disk.
Constructors
| Constructor | Summary |
|---|---|
JsonLinesProposalProvenanceSink(string directoryPath, ProposalProvenanceOptions options, string baseName) |
Initializes a segment-writing sink. |
Properties
| Property | Summary |
|---|---|
int PendingRecords { get; } |
Gets how many records are buffered and not yet on disk. |
int SegmentsWritten { get; } |
Gets how many segment files this instance has written. |
long RecordsWritten { get; } |
Gets how many records have been flushed to disk. |
string BaseName { get; } |
Gets the file-name stem shared by every segment. |
string DirectoryPath { get; } |
Gets the absolute directory segments are written into. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<string> EnumerateSegments(string directoryPath, string baseName) |
Lists the segment files belonging to a base name inside a directory, in read order. |
IReadOnlyList<string> GetSegmentPaths() |
Gets the segment files this directory currently holds, in read order. |
Task FlushAsync(CancellationToken cancellationToken) |
Writes every buffered record to a new segment. |
Task RecordAsync(ProposalProvenanceRecord record, CancellationToken cancellationToken) |
|
void Dispose() |
Flushes any buffered records and releases the sink. |
Fields
| Field | Summary |
|---|---|
string DefaultBaseName |
The file-name stem used when none is supplied. |
string SegmentExtension |
The extension every segment carries. |