DICE API Documentation/com.embabel.dice.ingestion/IngestionLedger

IngestionLedger

A lean deduplication ledger recording which content hashes have been seen.

The ledger lets a handler short-circuit re-ingestion of identical content before any extraction work runs. Hashes are caller-supplied identity, not a tamper-proof security control. Implementations may persist their record to deduplicate across sessions; DICE ships an in-memory default.

Inheritors

InMemoryIngestionLedger

Functions

forget

abstract fun forget(contentHash: String)

Release a previously recorded content hash so it is no longer treated as seen. Used to undo a claim when the work it guarded did not complete, keeping retries of failed content un-deduplicated.

record

abstract fun record(contentHash: String)

Record that content with the given hash has been ingested.

recordIfAbsent

open fun recordIfAbsent(contentHash: String): Boolean

Atomically claim a content hash: record it and report whether the claim was new. This collapses the check-then-record sequence into a single step so concurrent attempts to ingest identical content cannot both pass a separate seen-check before either records.

seen

abstract fun seen(contentHash: String): Boolean

Whether content with the given hash has already been recorded.