
DICE Release: 0.2.0
Built against embabel-agent: 1.5.2
© 2024-2026 Embabel Pty, Ltd
1. Overview
DICE — Domain-Integrated Context Engineering — is a knowledge substrate for JVM applications. It turns text into confidence-weighted natural-language propositions, resolves the entity mentions inside them to canonical identifiers, and then projects those propositions into whatever representation a task needs: a Neo4j graph, a Prolog fact base, vector embeddings, agent working memory, or a human-readable report.
Propositions are the single system of record. Everything else is a derived view.
1.1. What is DICE?
DICE extends context engineering by insisting that a domain model is what gives context its shape, and by treating LLM outputs — not just inputs — as things worth structuring.
Despite their seductive ability to work with natural language, LLMs become safer to use the more we add structure to inputs and outputs. DICE helps LLMs converse in the established language of our business and applications.
Domain objects are not mere structs. They not only provide typing, but define focused behaviour. In an agentic system, behaviour can be exposed to manually authored code and selectively exposed to LLMs as tools.
Context Engineering Needs Domain Understanding
1.1.1. Why domain integration matters
| Benefit | What it buys you |
|---|---|
Structured context |
Code fills the context window, rather than string concatenation. Less delicate, more repeatable. |
System integration |
Domain objects are the interface to systems you already run. |
Reuse |
A domain model captures business understanding once and is shared across agents. |
Persistence |
Structured query through SQL, Cypher and Prolog — not vector search alone. |
Testability |
Structure and encapsulation make behaviour assertable. |
Observability |
Debuggers and tracing tools understand typed objects. |
1.1.2. When to reach for DICE
Use DICE when your application needs to accumulate knowledge from text over time and then query it in more than one shape:
-
A chatbot that should remember what a user told it three sessions ago, with a record of where each belief came from and how strongly it is held.
-
A document pipeline that builds a queryable graph of entities and relationships, incrementally, without re-processing what it has already seen.
-
An agent that needs working memory scoped to a conversation, with stale beliefs decaying rather than piling up.
-
Anything that needs to answer "why do you believe that?" with evidence.
Do not reach for DICE when a vector store and a retrieval prompt are enough. The proposition model earns its cost when knowledge is revised — merged, reinforced, contradicted, superseded — not when it is only ever appended.
1.1.3. A worked example in production
Impromptu is a classical-music exploration chatbot that builds
a knowledge graph from conversation. It exercises most of the surface described in this guide:
PropositionPipeline for extraction, incremental analysis for streaming conversation, an
EscalatingEntityResolver with an LLM-backed candidate searcher, and Spring Boot integration with
asynchronous processing.
1.2. Architecture
DICE uses a proposition-based architecture inspired by the General User Models (GUM) research from Stanford and Microsoft. Like GUM, it constructs confidence-weighted propositions through a cycle of propose, retrieve and revise.
Natural-language propositions are the system of record. They accumulate evidence over time and project into typed views, each serving a different kind of question.
Text / Chunks
|
v
LLM extraction ......... concurrent, resolver-free
|
v
Entity resolution ........ always serial
|
v
+-------------------------+
| PROPOSITIONS | the system of record
| confidence + importance |
| + decay |
+-------------------------+
| | | | | \
v v v v v v
Vector Neo4j Prolog Memory Oracle Report
semantic graph rules agent NL Q&A rationale
retrieval traversal context + structure
1.2.1. The flow, in one pass
-
Chunk. Source material arrives as
Chunkobjects — fromdice-ingestion, from a conversation, or built directly by your code. -
Extract. A
PropositionExtractor(normallyLlmPropositionExtractor) turns each chunk into suggested propositions and the entity mentions inside them. This stage touches no resolver and is safe to run concurrently. -
Resolve. Each mention is matched against known entities or minted as a new one. This stage is always serial, so entity identity holds across chunks regardless of how extraction was dispatched.
-
Revise. If a
PropositionReviseris configured, new propositions are compared against what is already stored and classified as new, merged, reinforced or contradicted. -
Gate. Admission gates run over the result before anything is written.
-
Persist. The pipeline returns unsaved results. The caller decides the transaction boundary and calls
persist(…). Nothing is stored until it does. -
Project. Stored propositions are projected on demand into a graph, a Prolog fact base, agent memory, or a report.
The split at step 6 is deliberate: DICE never opens a transaction on your behalf.
1.2.2. Design notes
This guide describes how to use DICE. For why it behaves as it does — the decisions you cannot
recover by reading a single class — the design notes live in docs/design/ in the repository:
-
extraction-pipeline.md— the two-stage extract/resolve flow and why the pipeline returns unsaved results. -
proposition-lifecycle.md— trust scoring, source authority, supersession versus contradiction, decay instead of deletion. -
knowledge-hygiene.md— admission gates, mark-and-sweep reclamation, and the consolidation dream loop. -
consolidation-and-dream-loop.md— the pass abstraction and the four consolidation passes. -
reclamation-and-collector.md— mark-and-sweep internals, sweep policy, dry-run versus live, and the audit trail. -
graph-projection.md— edge lineage, projection outcomes, the stale cascade, and idempotent reconciliation. -
retrieval-and-discovery.md— store-agnostic graph queries, query-time authority filtering, and link discovery. -
durable-storage.md— backend selection, defence-in-depth dedup, two-phase save, and the decay tick. -
events.md— the domain-event model the store and pipeline emit.
1.3. Modules
DICE publishes five consumer-facing artifacts under the com.embabel.dice group. There is no
aggregator starter: pick the modules you need.
| Artifact | What it gives you | Pull it in when |
|---|---|---|
|
The whole domain: the |
Always. Everything else depends on it. |
|
A Drivine/Neo4j implementation of |
You want propositions in Neo4j rather than in memory. |
|
Spring Boot auto-configuration: backend selection from |
You are on Spring Boot and want beans wired for you. This is the usual entry point. |
|
Output projectors over propositions: rationale (why a fact is believed, with its evidence), structured reports, and surprising-link discovery. |
You need human-readable output. |
|
An ingestion SPI turning artifacts into chunks, with a content-hash ledger so the same source is not extracted twice. |
You are feeding files or documents in, rather than conversation turns. |
dice-integration-tests is test-only: it holds the cross-feature end-to-end harness and is not
published for consumption.
|
Dependency scope
|
1.4. Glossary
The rest of this guide assumes these terms.
- Proposition
-
A single natural-language claim, with a confidence, an importance, a decay factor, a status, a
ContextId, the entity mentions it contains, and links back to the source material it came from. The system of record. - ContextId
-
The scope every proposition belongs to — a user, a tenant, a conversation, a document set. It is a Kotlin value class; Java callers use
getContextIdValue(). Every query starts from aContextIdby design, so nothing accidentally loads the whole store. - Confidence and effective confidence
-
confidenceis what the LLM assigned at extraction time and never changes on its own.effectiveConfidence()folds in decay and is what you rank and filter on. - Decay
-
A 0–1 factor that erodes a proposition’s effective confidence as it ages, anchored on
contentRevised. Administrative changes — status, pinning, grounding — touchmetadataRevisedonly and never reset the clock. Decay replaces deletion: a belief fades rather than vanishing. - Mention
-
A span of text in a proposition that names an entity, before or after it has been matched to a canonical identifier.
- Entity resolution
-
Matching a mention to an existing entity or minting a new one. The shipped
EscalatingEntityResolverclimbs a ladder — exact match, heuristic, embedding, LLM verification, LLM bake-off — and stops at the cheapest level that succeeds. - Revision
-
Comparing a newly extracted proposition against what is already stored, and classifying it as new, merged, reinforced or contradicted.
- Projection
-
A derived, typed view over propositions: a graph, a Prolog fact base, agent memory, a report. A projection is always rebuildable from the propositions.
- Grounding
-
The link from a proposition back to the chunks it was extracted from, so "why do you believe this?" has an answer.
- Provenance
-
Richer evidence links than grounding: a
ProvenanceEntryrecords where a claim came from and, throughSourceLocator, how to get back to it. - Admission gate
-
A check that runs on pipeline output before the caller persists it — a confidence floor, a trust threshold, a projection-eligibility rule.
- Reclamation (the collector)
-
The mark-and-sweep pass that finds duplicate or stale propositions and decides each one’s fate, with an audit trail.
- Consolidation (the dream loop)
-
A composed set of passes — session consolidation, abstraction, contradiction resolution, decay sweep — that runs periodically to keep a store coherent.
- Authority
-
A tier assigned to a source, used by
AuthorityWeightedTrustScorerto weight what a claim from that source is worth, and available as a query-time filter.
2. Quickstart
One path, from an empty Spring Boot application to a printed report. No Neo4j, no Docker, nothing external to create. Every decision that could be made takes the default and links out.
At the end you will have extracted propositions from a paragraph of text, persisted them, queried them back, and rendered a report.
2.1. Dependencies
DICE publishes no aggregator starter, so the quickstart uses two coordinates: the
auto-configuration, which pulls dice and dice-storage transitively, and the report projectors.
<dependency>
<!-- Spring Boot auto-configuration: store backend, decay tick, collector. -->
<groupId>com.embabel.dice</groupId>
<artifactId>dice-storage-autoconfigure</artifactId>
<version>0.2.0</version>
</dependency>
<dependency>
<!-- Rationale, structured reports, link discovery. -->
<groupId>com.embabel.dice</groupId>
<artifactId>dice-report</artifactId>
<version>0.2.0</version>
</dependency>
You also need an embabel-agent runtime on the classpath. DICE declares embabel-agent-api and
embabel-agent-rag-core as provided, so the version your application brings is the version that
runs.
Snapshots and releases resolve from Embabel’s Artifactory:
<repositories>
<repository>
<id>embabel-snapshots</id>
<url>https://repo.embabel.com/artifactory/libs-snapshot</url>
<releases><enabled>false</enabled></releases>
<snapshots><enabled>true</enabled></snapshots>
</repository>
</repositories>
2.2. Configuration
The only thing you must configure is an LLM provider, and that is inherited from embabel-agent — set whatever API key your chosen provider expects. DICE’s own defaults need nothing:
embabel:
dice:
store:
type: in-memory # the default; stated here to be explicit
2.3. What auto-configuration gives you
With dice-storage-autoconfigure on the classpath and type left at in-memory, the following
beans exist, each @ConditionalOnMissingBean so your own bean always wins:
| Bean | What it is for |
|---|---|
|
|
|
Remembers which chunks have already been processed, so incremental analysis does not redo work. |
|
Lineage for projected edges, so a projection can be reconciled rather than rebuilt. |
|
The audit trail for reclamation runs. |
|
Materialises effective confidence and applies lifecycle transitions, hourly by default. |
Switching type to graph swaps the first four for their Drivine/Neo4j equivalents and changes
nothing else in your code. See Graph-backed storage.
2.4. Extract
Build a pipeline from an extractor, then run a chunk through it. The pipeline is immutable — every
with* method returns a new instance.
@Bean
PropositionPipeline propositionPipeline(PropositionExtractor extractor) {
return PropositionPipeline.withExtractor(extractor);
}
@Bean
LlmPropositionExtractor llmPropositionExtractor(Ai ai, PropositionRepository repository) {
return LlmPropositionExtractor
.withLlm(llmOptions)
.withAi(ai)
.withPropositionRepository(repository)
.withSchemaAdherence(SchemaAdherence.DEFAULT);
}
Then extract:
var context = new SourceAnalysisContext(
dataDictionary, // your domain schema
EscalatingEntityResolver.create(entityRepo), // how mentions become entities
new ContextId("quickstart")); // the scope everything lands in
var chunk = new Chunk("chunk-1",
"Ada Lovelace worked with Charles Babbage on the Analytical Engine.");
var results = pipeline.process(List.of(chunk), context);
results.getPropositions().forEach(p ->
System.out.printf("%.2f %s%n", p.getConfidence(), p.getText()));
You should see one or more claims printed with their confidence, something like:
0.92 Ada Lovelace worked with Charles Babbage.
0.88 Ada Lovelace worked on the Analytical Engine.
2.5. Persist
The pipeline stored nothing. It returned PersistablePropositions, and if you drop that on the
floor the work is discarded. Persist inside your own transaction:
@Transactional
public void store(PersistablePropositions results) {
results.persist(propositionRepository, namedEntityDataRepository);
}
persist saves only the entities actually referenced by the propositions being saved, saves the
propositions themselves — the revised set if revision was enabled, the raw set otherwise — and
creates the structural relationships linking chunks, propositions and entities.
2.6. Query
Queries always start from a ContextId. There is no PropositionQuery.create(), deliberately: it
would make "load everything" a one-liner.
// Everything in this context.
var all = propositionRepository.query(
PropositionQuery.againstContext(new ContextId("quickstart")));
// Only what is still strongly believed, best first.
var confident = propositionRepository.query(
PropositionQuery.againstContext(new ContextId("quickstart"))
.withMinEffectiveConfidence(0.7));
// Everything mentioning one entity.
var aboutAda = propositionRepository.query(
PropositionQuery.againstContext(new ContextId("quickstart"))
.withEntityId(adaId));
|
|
2.7. Report
Turn what you stored into something a person can read:
var report = reportProjector.report(confident, "What we know about Ada Lovelace");
System.out.println(report.summary());
2.8. Where to go next
-
Concepts — read the five concept pages in order. They define the vocabulary the rest of the guide uses.
-
How-to — task-shaped pages: extracting from documents, tuning resolution, querying, projecting to a graph, agent memory.
-
Features — the opt-in surfaces, each opening with the property or bean that switches it on.
-
Running in production — what changes when this stops being a quickstart.
3. Concepts
Five pages, meant to be read in order. Each uses terms the page before it defines.
3.1. Propositions
A Proposition is one claim, written in natural language, that DICE believes to some degree.
Everything else in DICE is derived from propositions, which means a proposition is the only thing
you must get right.
3.1.1. What a proposition carries
| Field | Meaning |
|---|---|
|
The claim itself, in natural language. This is the payload. |
|
The scope it belongs to. See Context and schema. |
|
What the extractor assigned when the claim was made. Never changes on its own. |
|
A 0–1 erosion factor applied as the claim ages. |
|
|
|
A pinned proposition is exempt from decay-driven reclamation. |
|
0 for a directly extracted claim; higher for abstractions built from other propositions. |
|
How many times independent evidence has restated this claim. |
|
The entity mentions in the text, resolved or not. |
|
The chunks this claim came from. |
|
Richer evidence links, each locatable through a |
|
Optional validity window, for claims that were true only for a period. |
|
Four separate clocks. The distinction matters — see below. |
3.1.2. Effective confidence is the number you use
confidence is a historical fact: it is what the LLM thought at extraction time. It is not what you
should rank or filter on.
double now = proposition.effectiveConfidence(); // decay applied, k = 2.0
double later = proposition.effectiveConfidenceAt(instant); // as of some other time
PropositionQuery exposes both: withMinConfidence(…) filters the raw number, and
withMinEffectiveConfidence(…) filters the decayed one. Reach for the effective form unless you
have a specific reason not to.
The decay rate multiplier k defaults to 2.0 and is configurable per query and, for the scheduled
tick, through embabel.dice.store.decay.k.
3.1.3. The four clocks
contentRevised is the decay anchor. It moves only when the claim itself changes.
Administrative changes — flipping status, pinning, attaching grounding — touch metadataRevised and
leave contentRevised alone. This is what stops housekeeping from silently making stale beliefs look
fresh. lastAccessed is separate again, and touchAccessed(…) on the store updates it in bulk
without disturbing either revision clock.
3.1.4. Status
PropositionStatus tracks a claim’s standing. ACTIVE is the normal case; the other states record
that a claim has been superseded, contradicted, or has decayed past usefulness. Transitions are
governed by a StatusTransitionPolicy, which is an SPI you can replace.
DICE decays rather than deletes. A claim that stops being believed keeps its history, its evidence, and its links, so "we used to think X, then Y arrived" remains answerable. Hard deletion is a separate, opt-in sweep — see Knowledge hygiene.
3.1.5. Evidence accumulation
When the same claim arrives again from a different source, DICE does not store a duplicate. It folds the new evidence in:
Proposition merged = existing.absorbEvidence(incoming);
absorbEvidence combines confidence, grounding, provenance and reinforceCount. The inverse,
withoutFoldedEvidence(…), exists so a retraction can be applied without losing the rest.
3.1.6. Immutability
Proposition is a Kotlin data class and every mutator returns a copy: withText, withStatus,
withConfidence, withPinned, withGrounding, withTemporal, withProvenance,
withResolvedMentions, withMetadataValue. Nothing mutates in place, so a proposition can be shared
across threads without ceremony.
Java callers get a Proposition.create(…) factory and getContextIdValue() for the value-class
ContextId.
var p = propositionRepository.query(
PropositionQuery.againstContext(new ContextId("quickstart"))).get(0);
System.out.printf("raw=%.2f effective=%.2f decay=%.2f reinforced=%d status=%s%n",
p.getConfidence(), p.effectiveConfidence(), p.getDecay(),
p.getReinforceCount(), p.getStatus());
Expect raw and effective confidence to be close on a freshly extracted claim, and to diverge as the decay tick runs against it.
3.2. Entity resolution
Extraction produces mentions — spans of text that name something. Resolution decides whether a mention refers to an entity you already know, or to one that should be minted.
Getting this wrong is the most expensive failure mode in a knowledge substrate: an unresolved duplicate splits everything you know about one thing into two half-graphs, and a wrong merge welds two things into one.
3.2.1. The escalating resolver
EscalatingEntityResolver climbs a ladder and stops at the cheapest rung that works. The rung it
stopped at is reported back as a ResolutionLevel:
| Level | How it matches | LLM call |
|---|---|---|
|
Exact name match against the repository. |
No |
|
Normalised name, then fuzzy and partial name matching. |
No |
|
High-confidence embedding similarity. |
No |
|
One candidate, verified yes/no by an LLM. |
Yes |
|
Several candidates, an LLM picks the best. |
Yes |
|
Nothing matched at any level. |
— |
Each attempt returns a LevelResult carrying the level, the resolution, a confidence, and how many
candidates were considered — which is what you look at when tuning.
Construct one with the shipped searcher chain:
// Full chain, including the vector searcher.
var resolver = EscalatingEntityResolver.create(entityRepository, candidateBakeoff);
// Same, minus the vector searcher — for stores with no vector index.
var resolver = EscalatingEntityResolver.withoutVector(entityRepository, candidateBakeoff);
Passing null for the bake-off means ambiguous cases mint a new entity rather than asking an LLM.
3.2.2. The searchers
Candidate recall is pluggable. CandidateSearcher implementations ship for exact name, normalised
name, partial name, fuzzy name, id, vector similarity, and AgenticCandidateSearcher, which lets an
LLM go looking. DefaultCandidateSearchers assembles the standard chains.
To add a strategy, implement CandidateSearcher and compose it into the resolver’s chain.
3.2.3. Composing resolvers
Resolvers compose, and the order is the policy:
| Resolver | Behaviour |
|---|---|
|
Resolves against entities you handed in on the context — the current user, say. Put it first. |
|
The ladder above, against the repository. |
|
Remembers what this run has already resolved, so chunk 7 recognises what chunk 2 minted. |
|
Tries each resolver in order and takes the first hit. |
|
Mints unconditionally. Useful in tests and in throwaway contexts. |
PropositionPipeline.process(…) wraps your resolver with an InMemoryEntityResolver
automatically — but only when mintNewEntities is on. With minting off the wrapper is skipped
deliberately: remembering an entity that will never be created would produce mentions resolved to
ids that do not exist.
3.2.4. Why resolution is serial
Extraction is stateless per chunk and can be dispatched concurrently. Resolution is not: it reads
and writes shared identity state, and running it concurrently is how you get two entities for one
thing. DICE keeps resolution serial regardless of the ExtractionExecutionStrategy in force, so
cross-chunk identity holds however extraction was dispatched. See
Concurrent extraction.
3.2.5. Mention filtering
Not every mention deserves resolution. A MentionFilter runs before resolution and drops
low-quality spans — vague references, overly long spans, anything the schema says cannot be an
entity. Filtering here is cheaper than resolving and then reclaiming.
Shipped filters include SchemaValidatedMentionFilter, the context-aware filters in
ContextAwareMentionFilters, and ObservableMentionFilter, which wraps another filter to report
what it rejected.
pipeline = pipeline.withMentionFilter(mentionFilter);
var chunk = new Chunk("c1",
"Ada Lovelace met Babbage in 1833. Lovelace later wrote the first algorithm.");
var results = pipeline.process(List.of(chunk), context);
results.getPropositions().stream()
.flatMap(p -> p.getMentions().stream())
.forEach(m -> System.out.println(m.getText() + " -> " + m.getResolvedId()));
"Ada Lovelace" and "Lovelace" should resolve to the same id. If they do not, that is the tuning problem Tune entity resolution exists for.
3.3. Storage and projections
Propositions are stored once and viewed many ways. This page covers the port they are stored through, and the views derived from them.
3.3.1. The store SPI
PropositionStore is the base port and is deliberately small: save, find by id, find by entity, find
by status, find by grounding, find by context, delete, count — plus one composable query.
PropositionRepository extends it with optional capability interfaces. A backend implements only
what it genuinely supports:
| Capability | What it adds |
|---|---|
|
Similarity search over proposition embeddings. |
|
Traversal over the proposition/entity graph. |
|
Native neighbourhood, path and lineage queries, plus the |
|
Queries over validity windows. |
Each capability interface carries default implementations expressed over the base primitives, so a backend that implements nothing extra still behaves correctly — just more slowly.
3.3.2. The shipped backends
| Backend | Where it lives | Use it for |
|---|---|---|
|
|
Development, tests, and single-process applications. Needs an |
|
|
A store that survives a restart without any infrastructure. |
|
|
Propositions inside an embabel-agent RAG-backed Neo4j. |
|
|
Production Neo4j through Drivine, with the vector index, decay in the database, and lineage. |
Selection is a property, not a code change — see Graph-backed storage.
3.3.3. Composable queries
PropositionQuery is a data class assembled through with* methods. The axes:
-
Scope:
contextId,entityId,anyEntityIds,allEntityIds -
Lifecycle:
statuses,pinned,minLevel,maxLevel -
Time:
createdAfter/Before,revisedAfter/Before,accessedAfter/Before -
Belief:
minConfidence,minEffectiveConfidence(witheffectiveConfidenceAsOfanddecayK),belowEffectiveConfidence,minImportance,minReinforceCount,minTrustScore -
Shape:
orderBy,limit
var stale = PropositionQuery.againstContext(contextId)
.withBelowEffectiveConfidence(0.2)
.withStatus(PropositionStatus.ACTIVE)
.withLimit(100);
Kotlin infix factories have Java-friendly withXxx and againstContext equivalents throughout.
3.3.4. Projections
A projection is a rebuildable, typed view. Nothing in a projection is authoritative — if it and the propositions disagree, the propositions win.
| Projection | What it produces |
|---|---|
Graph ( |
A typed entity-relationship graph. |
Prolog ( |
A tuProlog fact base you can run rules against. Experimental — see Prolog inference. |
Memory ( |
Propositions sorted into |
Report ( |
Human-readable output, including "why is this believed" with the evidence. In |
EventEmittingProjector and EventEmittingPropositionRepository are decorators that publish Spring
application events on projection and on save. Wire them in when you want something to react.
3.3.5. Retrieval
RetrievalRouter is the single entry point for retrieval, choosing a RetrievalMode rather than
making the caller pick between vector search, graph traversal and keyword overlap. Authority
filtering is applied at query time, and pushed down to the backend where the backend says it can
honour it.
System.out.println(propositionRepository.getClass().getSimpleName());
System.out.println("vector: " + (propositionRepository instanceof VectorSearchCapable));
System.out.println("graph: " + (propositionRepository instanceof GraphQueryCapable));
System.out.println("temporal: " + (propositionRepository instanceof TemporalQueryCapable));
On the quickstart’s in-memory default this tells you exactly which capabilities you have before you write a query that needs one.
3.4. Context and schema
Two things bound what DICE extracts and what it will return: the ContextId that scopes a
proposition, and the schema that tells extraction what the domain looks like.
3.4.1. ContextId
Every proposition belongs to exactly one ContextId. It is the primary scope, and the starting
point of every query:
var query = PropositionQuery.againstContext(new ContextId("user:42"));
There is no PropositionQuery.create(). That is a design decision, not an omission: without a
scope, "load everything" would be the shortest thing to write, and in a store that accumulates
indefinitely that is a production incident waiting to happen.
ContextId is a Kotlin value class, so it costs nothing at runtime. Java callers reach the string
through getContextIdValue(), on both Proposition and PropositionQuery.
What a context should represent is your decision. Common choices: one per user, one per tenant, one per conversation, one per document collection. Contexts do not nest; if you need a hierarchy, encode it in the value and query by prefix at your own layer.
3.4.2. SourceAnalysisContext
SourceAnalysisContext is the per-run configuration handed to the pipeline. It carries the schema,
the resolver, the ContextId, and everything else extraction needs to know:
| Field | Purpose |
|---|---|
|
The |
|
How mentions become entities. See Entity resolution. |
|
The scope everything extracted lands in. |
|
Entities the run should already recognise — |
|
Declared predicates, used by extraction and by relation-based graph projection. |
|
Extra values available to the extraction template. |
|
How to get back to the source material, for provenance. |
|
Whose point of view the text is written from. |
|
Whether unresolved mentions may create entities. Off by default. |
|
Properties stamped onto anything minted. |
It is a data class with with* builders, so a base context can be specialised per run:
var context = new SourceAnalysisContext(dataDictionary, resolver, contextId)
.withKnownEntities(KnownEntity.asCurrentUser(user))
.withRelations(relations)
.withMintNewEntities(true)
.withSourceLocator(sourceLocator);
3.4.3. The DataDictionary and SchemaAdherence
The DataDictionary is your domain’s schema: the entity types extraction should look for, and the
relationship metadata that names the predicates it may use. SchemaRegistry holds them.
SchemaAdherence decides how strictly extraction obeys it. Two independent switches:
| Constant | entities |
predicates |
Behaviour |
|---|---|---|---|
|
|
|
Only schema entity types, only schema predicates. Nothing outside the model gets in. |
|
|
|
Entity types locked to the schema; any predicate allowed. The usual choice. |
|
|
|
Schema types preferred but not required, on both axes. Use when you are still discovering the domain. |
Set it on the extractor:
LlmPropositionExtractor.withLlm(llmOptions)
.withAi(ai)
.withPropositionRepository(repository)
.withSchemaAdherence(SchemaAdherence.STRICT);
Strictness is the main lever on extraction noise. Start at DEFAULT; move to STRICT when you
trust the schema and want the extractor to stop inventing predicates; move to RELAXED only while
exploring.
3.4.4. Relations
Relations declares the predicates that may hold between entities. Extraction uses them to phrase
claims consistently, and RelationBasedGraphProjector uses them to decide which edges to create.
Declaring relations is what turns free-text claims into a graph you can traverse rather than a bag
of sentences.
var strict = new SourceAnalysisContext(dataDictionary, resolver, new ContextId("strict-test"));
// ... run the same chunk through an extractor built with STRICT and with RELAXED,
// and compare how many propositions and mentions survive.
The gap between the two is the size of the noise your schema is currently suppressing.
3.5. Knowledge hygiene
A store that only ever accumulates gets worse over time. DICE has three interventions, at three different points in the life of a claim: gates keep bad claims out, reclamation removes claims that stopped earning their place, and consolidation rewrites what is left into something more useful.
3.5.1. 1. Admission gates
Gates run on pipeline output before the caller persists anything. They are the cheapest intervention, because nothing has been written yet.
| Gate | What it rejects |
|---|---|
|
Claims below a raw confidence floor. |
|
Claims with too little evidence behind them. |
|
Claims whose trust score — source authority folded into confidence — is too low. |
|
Routes claims that duplicate something already stored, rather than admitting a second copy. |
|
Routes claims that contradict something already stored. |
|
Admits the claim but marks it as not yet worth projecting. |
Compose them with ExtractionGatePipeline, and wrap any of them in ObservableGate to see what is
being rejected and why. A gate you cannot observe is a gate you will eventually mis-tune.
3.5.2. 2. Reclamation — the collector
Reclamation is mark-and-sweep over what is already stored. CollectorRunner walks the store, a
CollectorStrategy marks candidates, and a SweepPolicy decides each marked proposition’s fate.
The vocabulary is small and worth knowing: a PropositionMark carries a MarkReason, and the
policy produces a SweepAction.
Two shipped strategies:
-
DecayCollectorStrategy— marks claims whose effective confidence has fallen far enough. -
DuplicateCollectorStrategy— marks claims that say the same thing as another.
The MultiSignalCollectorStrategy is the production duplicate matcher, blending several signals
rather than trusting any one of them. See Multi-signal collector.
Two things make reclamation safe to run: it has a dry-run mode that reports what it would do
without doing it, and every live run is recorded to a CollectorRecordStore, so a surprising result
is explainable after the fact. Run it dry first. Always.
Pinned propositions are exempt.
3.5.3. 3. Consolidation — the dream loop
Consolidation rewrites what survives into something more useful. DreamLoopOrchestrator composes
passes, and DefaultDreamLoopOrchestrator gates each pass on a threshold, so a quiet store does no
work.
The four shipped passes:
| Pass | What it does |
|---|---|
|
Folds a session’s episodic detail into durable claims. |
|
Builds higher- |
|
Resolves claims that contradict one another, adjusting status rather than deleting. |
|
Applies decay and the resulting lifecycle transitions. |
A run produces a DreamLoopReport describing what each pass did.
DefaultMemoryMaintenanceOrchestrator is the older four-step pipeline. Prefer the dream loop for
new work.
3.5.4. Decay, not deletion
Underneath all three sits the same principle: DICE fades beliefs rather than erasing them. Decay
lowers effective confidence, status records that a claim was superseded or contradicted, and the
evidence stays attached. Hard deletion of STALE propositions is available but off by default —
embabel.dice.store.decay.prune-stale is false — because most applications would rather answer
"why did you stop believing that?" than reclaim the rows.
// See what reclamation would do, without doing it.
var dryRun = collectorRunner.run(contextId, /* dryRun = */ true);
System.out.println(dryRun);
Read the marks and their reasons before you ever run this live.
4. How-to
Task-shaped pages, titled by what you want rather than by what the code is called. Each states its prerequisites first.
4.1. Extract knowledge from documents
Prerequisites: dice, dice-ingestion, a configured LLM, and a PropositionRepository.
4.1.1. Ingest, then extract
dice-ingestion turns artifacts into chunks and keeps a ledger so the same source is not extracted
twice.
| Type | Role |
|---|---|
|
Turns one artifact into chunks. |
|
One thing that was ingested, with its content hash. |
|
A set of artifacts ingested together. |
|
The content-hash record that makes re-ingestion a no-op. |
|
What came out: the chunks, and what was skipped as already seen. |
The ledger keys on content, not on filename or path. Renaming a file does not cause re-extraction; editing one does.
4.1.2. Run the chunks through the pipeline
var result = ingestionHandler.handle(artifact);
var context = new SourceAnalysisContext(dataDictionary, resolver, contextId)
.withSourceLocator(sourceLocator) // so provenance can point back at the file
.withMintNewEntities(true);
var propositions = pipeline.process(result.getChunks(), context);
process handles cross-chunk entity identity for you: when minting is enabled it wraps your
resolver so an entity minted in chunk 2 is recognised in chunk 7.
4.1.3. Persist within your own transaction
The pipeline stores nothing. That is deliberate — DICE never decides your transaction boundary.
@Transactional
public void ingest(Path file) {
var result = ingestionHandler.handle(artifactFor(file));
var propositions = pipeline.process(result.getChunks(), context);
propositions.persist(propositionRepository, namedEntityDataRepository);
}
If you forget persist, nothing is saved and nothing complains.
4.1.4. Only process what changed
For a corpus that grows, use incremental analysis rather than reprocessing. ChunkHistoryStore
records which chunks have been analysed; AbstractIncrementalAnalyzer and its subclasses skip
those. dice-storage-autoconfigure provides a ChunkHistoryStore bean for whichever backend is
selected.
4.1.5. Add grounding and provenance
Grounding links a claim to the chunk it came from and is created automatically. Provenance is
richer: a ProvenanceEntry plus a SourceLocator records how to get back to the original material,
which is what RationaleProjector uses to explain a belief.
var context = baseContext.withSourceLocator(sourceLocator);
4.1.6. Filter mentions before resolving
Document text carries far more noise than conversation does — headers, boilerplate, citations. A
MentionFilter drops weak mentions before the expensive resolution step:
pipeline = pipeline.withMentionFilter(
new ObservableMentionFilter(new SchemaValidatedMentionFilter(dataDictionary)));
Wrapping in ObservableMentionFilter while you tune tells you what is being dropped.
4.1.7. Speed it up
Extraction is the slow stage and is stateless per chunk, so it can be dispatched concurrently. Resolution stays serial regardless. See Concurrent extraction — including the thread-safety condition you must verify before turning it on.
4.2. Analyse a conversation incrementally
Prerequisites: dice, a configured LLM, a PropositionRepository and a ChunkHistoryStore.
A conversation is not a document. It arrives a turn at a time, the same claim gets restated and revised, and the user themselves is an entity the extractor should already know about.
4.2.1. The incremental analyser
IncrementalAnalyzer processes an IncrementalSource and remembers what it has already seen.
ConversationSource wraps a conversation; ConversationSegmenter decides where one unit of analysis
ends and the next begins, so a claim spanning three turns is not cut in half.
var source = new ConversationSource(conversation);
var result = analyzer.analyze(source, context);
result.persist(propositionRepository, entityRepository);
Re-analysing the same conversation is cheap: ChunkHistoryStore reports what has already been
processed and the analyser skips it.
4.2.2. Tell the context who is talking
Without this, "I prefer the early recordings" produces a claim about nobody.
var context = SourceAnalysisContext
.withContextId(user.currentContext())
.withEntityResolver(entityResolverForUser(user))
.withSchema(dataDictionary)
.withRelations(relations)
.withKnownEntities(KnownEntity.asCurrentUser(user));
KnownEntity.asCurrentUser(…) is the important line: it gives the resolver a first-class identity
for the speaker, so first-person claims attach to them.
4.2.3. Drive it from events
Conversation analysis costs an LLM call, so it should not sit in the request path.
ConversationAnalysisRequestEvent exists for exactly this:
@Async
@Transactional
@EventListener
public void onConversationExchange(ConversationAnalysisRequestEvent event) {
var context = SourceAnalysisContext
.withContextId(event.user.currentContext())
.withEntityResolver(entityResolverForUser(event.user))
.withSchema(dataDictionary)
.withRelations(relations)
.withKnownEntities(KnownEntity.asCurrentUser(event.user));
var source = new ConversationSource(event.conversation);
var result = analyzer.analyze(source, context);
result.persist(propositionRepository, entityRepository);
}
SourceAnalysisRequestEvent is the general form for non-conversation sources.
4.2.4. Turn on revision
In a conversation the same thing gets said repeatedly, and sometimes retracted. Without revision you accumulate near-duplicates; with it, restatements reinforce and reversals are classified as contradictions.
var pipeline = PropositionPipeline
.withExtractor(extractor)
.withRevision(reviser, propositionRepository);
Revision compares against what is already stored and classifies each new claim as new, merged,
reinforced or contradicted. It still persists nothing — hasRevision tells you which set
propositionsToPersist() will return.
4.3. Tune entity resolution
Prerequisites: Entity resolution read, and a corpus you can re-run.
Resolution has two failure modes, and they pull in opposite directions. Under-merging splits one real thing into several entities, so nothing you know about it is ever retrieved together. Over-merging welds two real things into one, and that is much harder to notice and much harder to undo. Tune towards under-merging: a duplicate can be collected later, a bad merge cannot be cleanly unpicked.
4.3.1. Find out where resolution is happening
LevelResult reports the ResolutionLevel that produced each answer, and how many candidates were
considered. That distribution is the diagnostic:
-
Mostly
EXACT_MATCH— healthy, and free. -
Heavy
LLM_BAKEOFF— expensive. Your cheap searchers are not recalling the right candidates. -
Heavy
NO_MATCHwith minting on — you are creating duplicates. Recall is too narrow. -
Bake-offs with many candidates — recall is too wide, and you are paying an LLM to sort it out.
4.3.2. Fix recall before touching the LLM rungs
The cheap searchers decide what the expensive ones ever see.
| Searcher | When it earns its place |
|---|---|
|
Always. Free, and catches the majority case. |
|
Casing, punctuation and whitespace variation. Almost always worth it. |
|
"Lovelace" for "Ada Lovelace". Necessary for conversation, noisy for catalogues. |
|
Typos and transliteration. Widens recall; watch the bake-off rate after enabling. |
|
When your source text carries real identifiers. |
|
Aliases that share no characters. Needs a vector index. |
|
An LLM goes looking. The most capable and the most expensive — last resort. |
DefaultCandidateSearchers.create(repository) assembles the standard chain;
DefaultCandidateSearchers.withoutVector(repository) drops the vector searcher for stores with no
index.
4.3.3. Decide what happens when nothing matches
// Ambiguity goes to an LLM.
var resolver = EscalatingEntityResolver.create(entityRepository, llmCandidateBakeoff);
// Ambiguity mints a new entity instead.
var resolver = EscalatingEntityResolver.create(entityRepository, null);
And separately, whether minting is allowed at all:
context = context.withMintNewEntities(false); // unresolved mentions are vetoed, not minted
Minting off is the right default for a closed catalogue. Minting on is necessary for open-domain extraction, and is what makes mention filtering matter.
4.3.4. Put known entities first
ChainedEntityResolver tries resolvers in order and takes the first hit, so order is policy:
var resolver = new ChainedEntityResolver(List.of(
new KnownEntityResolver(knownEntities), // the user, the tenant
EscalatingEntityResolver.create(entityRepository, bakeoff) // everything else
));
Do not add InMemoryEntityResolver by hand for a process(…) run — the pipeline adds it when
minting is on, and adding it when minting is off produces mentions resolved to ids that will never
exist.
4.3.5. Reduce what reaches the resolver
Every mention that should never have been a mention is a chance to resolve wrongly. A
MentionFilter is cheaper than the resolution it prevents:
pipeline = pipeline.withMentionFilter(
new ObservableMentionFilter(new SchemaValidatedMentionFilter(dataDictionary)));
Tightening SchemaAdherence to STRICT has the same effect earlier still, at the extractor. See
Context and schema.
4.3.6. Clean up what got through
Some duplication is unavoidable, and reclamation exists to absorb it. The
MultiSignalCollectorStrategy blends vector, lexical, entity-overlap, grounding-overlap and
provenance-overlap signals, with a polarity veto that stops a claim merging with its own negation.
Tune embabel.dice.collector.match-threshold and the per-signal weights rather than loosening
resolution — see Multi-signal collector.
Run it dry first, and read the trace.
4.4. Query propositions
Prerequisites: a populated PropositionRepository.
Every query starts from a ContextId. PropositionQuery has no create() factory, so there is no
short way to write "load everything".
var base = PropositionQuery.againstContext(new ContextId("user:42"));
// or, equivalently
var base = PropositionQuery.forContextId(new ContextId("user:42"));
4.4.1. By belief
Filter on effective confidence, not raw confidence, unless you specifically want the extractor’s original opinion:
// What we currently believe.
repository.query(base.withMinEffectiveConfidence(0.7));
// What we believed at some past moment.
repository.query(base.withMinEffectiveConfidence(0.7)
.withEffectiveConfidenceAsOf(lastMonth));
// What has faded — reclamation candidates.
repository.query(base.withBelowEffectiveConfidence(0.2));
// Repeatedly corroborated claims only.
repository.query(base.withMinReinforceCount(3));
// Weighted by source authority.
repository.query(base.withMinTrustScore(0.6));
4.4.2. By entity
repository.query(base.withEntityId(adaId)); // mentions this entity
repository.query(base.withAnyEntity(adaId, babbageId)); // mentions either
repository.query(base.withAllEntities(adaId, babbageId)); // mentions both
withAllEntities is how you ask about a relationship without traversing a graph: claims that
mention both endpoints are the claims about the pair.
4.4.3. By lifecycle
repository.query(base.withStatus(PropositionStatus.ACTIVE));
repository.query(base.withPinned(true));
repository.query(base.withMinLevel(1)); // abstractions only
repository.query(base.withMaxLevel(0)); // directly extracted claims only
level is the abstraction ladder: 0 is extracted, higher levels were built by the consolidation
AbstractionPass.
4.4.4. By time
Three independent clocks, three independent filters:
base.withCreatedAfter(t) // when the claim first appeared
base.withRevisedAfter(t) // when its content last changed — the decay anchor
base.withAccessedAfter(t) // when it was last read
Filtering on revisedAfter is what you want for "what changed"; createdAfter misses claims that
were revised, and accessedAfter tells you about readers, not writers.
4.4.5. Shape the result
var top = base.withMinEffectiveConfidence(0.5)
.withOrderBy(OrderBy.EFFECTIVE_CONFIDENCE)
.withLimit(20);
count(query) returns the size without materialising the rows.
4.4.6. Beyond the composable query
PropositionStore also offers direct lookups — findById, findByEntity, findByStatus,
findByGrounding, findByMinLevel, findPinned — plus keywordOverlap(base, tokens, limit) for
lexical scoring over a base query, and pin/unpin for exempting a claim from reclamation.
touchAccessed(ids) records that a set of propositions was read, without touching either revision
clock.
4.4.7. Vector, graph and routed retrieval
Capabilities are optional per backend — check before you rely on one:
if (repository instanceof VectorSearchCapable v) { /* similarity search */ }
if (repository instanceof GraphQueryCapable g) { /* neighbourhood, path, lineage */ }
if (repository instanceof TemporalQueryCapable t) { /* validity windows */ }
GraphQueryCapable gives you GraphNeighborhood, GraphPath and PropositionLineage. Backends
that support it may also report honorsAuthorityFilter, letting authority-filtered traversals run
natively rather than being filtered afterwards.
When you would rather not choose, use RetrievalRouter: it picks a RetrievalMode — vector, graph
or lexical — from the shape of the query, and applies authority filtering at query time.
4.5. Project propositions into a graph
Prerequisites: propositions with resolved mentions, and a NamedEntityDataRepository. A graph
backend is not required to project, but is what makes traversal worth doing.
Propositions are sentences. A graph is edges. Projection is the step that turns one into the other, and it is always rebuildable — if the graph and the propositions disagree, the propositions win.
4.5.1. Pick a projector
| Projector | How it decides edges |
|---|---|
|
From the |
|
An LLM infers edges from the claim text. Covers relationships you did not anticipate, at the cost of a call per batch and less predictability. |
Start relation-based. Reach for the LLM projector when you find real relationships your Relations
cannot express.
var projector = new RelationBasedGraphProjector(relations, persister, policy);
var outcome = projector.project(propositions);
GraphProjectionService wraps this for the common case; NamedEntityDataRepositoryGraphRelationshipPersister
is the persister for entity repositories.
4.5.2. Control what gets projected
A ProjectionPolicy decides which propositions are eligible — typically a confidence or trust floor,
so a weakly believed claim does not become a hard edge. ProjectionPolicySupport holds the shipped
building blocks.
The ProjectionEligibilityGate does the same job earlier, at admission time, marking a claim as
stored but not yet worth projecting.
Deciding this deliberately matters: an edge reads as a fact, even when the claim behind it was believed at 0.4.
4.5.3. Describe the edges
RelationshipDescriptionSynthesizer generates human-readable descriptions for projected
relationships; LlmRelationshipDescriptionSynthesizer does it with an LLM. Useful when the graph is
something people will look at rather than only traverse.
4.5.4. Lineage and reconciliation
Every projected edge records which propositions produced it, in a ProjectionRecordStore. Two
things follow.
The stale cascade. When a source proposition changes, the edges derived from it are marked stale rather than left silently wrong. You can find them, and you can see why.
Idempotent reconciliation. Reconciler brings the graph back into agreement with the propositions
without rebuilding it. Re-running projection over unchanged input is a no-op, so projection can be
scheduled rather than orchestrated.
var health = graphQueries.projectionHealth(contextId); // what is stale, what is current
reconciler.reconcile(contextId); // bring the graph back in line
The REST layer exposes the same information at
GET /api/v1/contexts/{contextId}/discovery/projection-health.
4.5.5. Query the result
With a GraphQueryCapable repository:
GraphNeighborhood around = graphQueries.neighborhood(entityId, /* depth = */ 2);
List<GraphPath> paths = graphQueries.paths(fromEntityId, toEntityId);
PropositionLineage why = graphQueries.lineage(propositionId);
lineage is the "why does this edge exist" query: it walks back from a projected edge to the
propositions and, through grounding and provenance, to the source text.
4.6. Give an agent memory
Prerequisites: a populated PropositionRepository scoped by ContextId.
An agent’s context window is small and its conversation is long. Memory projection decides what goes in.
4.6.1. The four kinds of memory
MemoryProjector.project(propositions) returns a MemoryProjection with propositions sorted into
four buckets, following the standard cognitive split:
| Bucket | What belongs there |
|---|---|
|
Durable facts. "The user works in oncology research." |
|
Things that happened. "On Tuesday the user asked about drug interactions." |
|
How to do things in this domain. "This user wants citations with every claim." |
|
The current task’s scratch space. Short-lived by design. |
KnowledgeType is the enum behind the split. MemoryProjection exposes size and all(), so you
can budget the context window across buckets rather than truncating a flat list.
var projection = memoryProjector.project(
repository.query(PropositionQuery.againstContext(contextId)
.withMinEffectiveConfidence(0.5)));
System.out.printf("semantic=%d episodic=%d procedural=%d working=%d%n",
projection.getSemantic().size(), projection.getEpisodic().size(),
projection.getProcedural().size(), projection.getWorking().size());
4.6.2. Recall
MemoryRetriever is the read side, with four entry points shaped by how agents actually ask:
memoryRetriever.recall(contextId, query, limit); // general, relevance-ranked
memoryRetriever.recallAbout(contextId, entityId, limit); // everything about one entity
memoryRetriever.recallByType(contextId, type, limit); // one KnowledgeType only
memoryRetriever.recallRecent(contextId, limit); // most recent first
DefaultMemoryRetriever implements all four. recallAbout is the one to reach for when the agent
has already identified who or what it is talking about — it is more precise than a semantic query
over the same text.
4.6.3. Keep memory from rotting
Memory is where accumulation hurts first: an agent whose recall is 80% stale beliefs is worse than one with no memory at all.
Run the dream loop periodically. DefaultDreamLoopOrchestrator gates each pass on a threshold, so a
quiet context does nothing:
DreamLoopReport report = dreamLoopOrchestrator.run(contextId);
The passes fold sessions into durable claims, build abstractions, resolve contradictions and apply decay. See Knowledge hygiene.
MemoryConsolidator is the lower-level piece if you want to compose your own cycle;
DefaultMemoryMaintenanceOrchestrator is the older four-step pipeline, kept for compatibility.
4.6.4. Pin what must not fade
Some things should never decay out of memory — a user’s name, a hard constraint, a standing instruction:
propositionRepository.pin(propositionId);
Pinned propositions are exempt from decay-driven reclamation. Use it sparingly; every pin is a permanent claim on the context window.
4.6.5. Serve it over HTTP
MemoryController exposes memory at /api/v1/contexts/{contextId}/memory — list, search, recall by
entity, create, fetch and delete. It is opt-in. See Web API.
4.7. Ask questions in natural language
Prerequisites: a populated PropositionRepository and a configured LLM.
Oracle answers a natural-language question from what DICE knows.
Answer answer = oracle.ask("What does the user think about the early recordings?");
// or with the full question type
Answer answer = oracle.ask(new Question(text));
Two implementations ship:
| Implementation | How it answers |
|---|---|
|
Retrieves relevant propositions, then asks an LLM to answer from them. The general case. |
|
Exposes DICE query operations to the LLM as tools, so it can decide what to look up. Better for questions that need several lookups or a traversal. |
PrologTools makes the Prolog projection available to the Oracle as a tool, so a question needing
inference — rather than retrieval — can be answered by running rules. See
Prolog inference.
4.7.1. Why the Oracle is not a chatbot
The Oracle answers from stored propositions, and nothing else. That is the point: every answer is traceable to claims that carry confidence, grounding and provenance. If it does not know, the honest answer is that it does not know — which is only useful because the propositions behind it are auditable.
To show your work, pair the Oracle with RationaleProjector from dice-report:
RationaleArtifact why = rationaleProjector.rationale(proposition);
LlmRationaleProjector produces the narrative form; rationale(PropositionGroup) explains a set of
claims together rather than one at a time.
4.8. Choose a storage backend
Prerequisites: dice-storage-autoconfigure on the classpath.
Backend selection is a property. The code you write against PropositionRepository does not change.
| Backend | embabel.dice.store.type |
Choose it when |
|---|---|---|
In-memory |
|
Development, tests, single-process applications, and anything where losing the store on restart is
acceptable. Needs an |
JSON file |
— (wire |
You want survival across restarts with no infrastructure at all. Not for concurrent writers. |
Graph (Drivine/Neo4j) |
|
Production. Durable, concurrent, with native graph traversal, a vector index, and decay applied in the database. |
4.8.1. Switching to the graph backend
embabel:
dice:
store:
type: graph
Plus Drivine’s own Neo4j connection configuration. Nothing else in your application changes: the
auto-configuration swaps PropositionRepository, ChunkHistoryStore, DecayManager,
ProjectionRecordStore and CollectorRecordStore for their Drivine equivalents, and registers the
schema catalogues that create the constraints and the vector index.
See Graph-backed storage for the full activation detail.
4.8.2. What changes when you switch
| In-memory | Graph | |
|---|---|---|
Survives restart |
No |
Yes |
Concurrent writers |
Single process |
Yes |
Vector search |
Through the |
Native Neo4j vector index |
Graph traversal |
Default implementations over the primitives |
Native Cypher, via |
Decay tick |
In-process, over the whole store |
Executed in the database |
Extra infrastructure |
None |
A Neo4j instance |
4.8.3. Overriding the choice entirely
Every auto-configured bean is @ConditionalOnMissingBean. Define your own PropositionRepository
and the auto-configuration steps aside — which is also how you implement PropositionStore against
a backend DICE does not ship.
Implement only the capabilities you genuinely support. The default implementations on
VectorSearchCapable, GraphTraversalCapable, GraphQueryCapable and TemporalQueryCapable
express each operation over the base primitives, so an unimplemented capability degrades in
performance rather than breaking.
4.8.4. Migrating between backends
There is no shipped migration tool. Propositions are the system of record and everything else is a projection, so a migration is: read every proposition and entity from the old store, write them to the new one, then re-project. Reconciliation is idempotent, so re-projecting is safe to repeat.
4.9. Observe and react to what DICE does
Prerequisites: a Spring application context.
DICE emits domain events, and every expensive or surprising decision has an observable wrapper.
4.9.1. Events
Two decorators publish Spring application events. Neither is wired by default — you opt in by wrapping:
@Bean
PropositionRepository propositionRepository(PropositionRepository delegate,
ApplicationEventPublisher publisher) {
return new EventEmittingPropositionRepository(delegate, publisher);
}
EventEmittingPropositionRepository publishes on save; EventEmittingProjector publishes on
projection. PropositionPersisted is the canonical durable signal — the one to build on when you
need to know something was actually stored.
DiceEvent is the event hierarchy; DiceEventListeners holds the listener support types.
4.9.2. Pipeline events
The pipeline emits its own, pre-persistence events:
pipeline = pipeline.withEventListener(listener);
process emits one aggregate ExtractionBatchCompleted per run. With a reviser configured,
processChunk also emits a per-proposition candidate event for each RevisionResult.
Those candidate events are pre-persistence signals, and the distinction matters: a candidate event
means the pipeline classified something, not that anything was written. If your listener has side
effects, build them on PropositionPersisted instead.
A listener that throws cannot abort a run — the pipeline wraps whatever you register in
SafeDiceEventListener. The default is DiceEventListener.DEV_NULL, so behaviour is unchanged when
no listener is set.
4.9.3. Observable wrappers
Two decorators exist purely so you can see what is being rejected:
new ObservableMentionFilter(delegateFilter); // which mentions were dropped, and why
new ObservableGate(delegateGate); // which propositions a gate rejected
Both are for tuning. A filter or gate you cannot observe is one you will eventually set wrong and not notice — the symptom is silence, and silence looks like success.
4.9.4. Reclamation audit trail
Every live collector run is recorded to a CollectorRecordStore, and detailed decisions to a
CollectorTraceStore. The trace records the edges considered, the connected components found, and
the decision made for each — enough to explain any individual merge after the fact.
embabel:
dice:
collector:
trace:
enabled: true
detail-retention-days: 30 # null keeps detail indefinitely
InMemoryCollectorTraceStore is the default; DrivineCollectorTraceStore persists to Neo4j when the
graph backend is selected.
5. Features
One page per opt-in surface. Each opens with the exact bean or property that switches it on, so you never have to guess whether a feature is active.
| Surface | Activation condition |
|---|---|
|
|
|
|
|
|
|
|
A parallel or batched |
|
|
|
|
|
A |
5.1. Graph-backed storage
Activation: embabel.dice.store.type=graph
Availability: dice-storage and dice-storage-autoconfigure, from DICE 0.2.0. Costs a
Neo4j instance.
5.1.1. When to use it
Use it in production, and whenever you need any of: durability across restarts, more than one process writing, native graph traversal, or decay applied in the database rather than in your heap.
Leave it off for development, tests and single-process applications. The in-memory backend is the default precisely because most of what you do while building DICE into an application does not need Neo4j.
5.1.2. Impact
-
Infrastructure: a Neo4j instance, reached through Drivine (
drivine4j-spring-boot-starter). -
Latency: network round trips replace map lookups. Traversals get much faster; single-id lookups get slower.
-
Memory: propositions no longer live in your heap.
-
LLM calls: unchanged.
-
Startup: schema catalogues create constraints and the vector index on first run.
5.1.3. How to enable
embabel:
dice:
store:
type: graph
Plus Drivine’s Neo4j connection configuration.
5.1.4. What you get
With type=graph, DiceStorageAutoConfiguration provides:
| Bean | Implementation |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
Proposition constraints, lineage-record schema, and the vector index spec |
Every one is @ConditionalOnMissingBean, so your own bean always wins.
5.1.5. The graph model
dice-storage maps propositions onto typed nodes: PropositionNode, SourceNode,
ProcessedChunkNode, Mention and DerivedFrom, with PropositionView and
PropositionWithProvenanceView as the read projections. PropositionGraphMapper handles the
mapping; GraphProjectionCypher holds the traversal queries.
5.1.6. Example
Nothing in your code changes. This is the same call against either backend:
var confident = propositionRepository.query(
PropositionQuery.againstContext(contextId).withMinEffectiveConfidence(0.7));
What changes is that on the graph backend the repository also implements GraphQueryCapable, so
native traversal is available:
if (propositionRepository instanceof GraphQueryCapable g) {
var neighbourhood = g.neighborhood(entityId, 2);
}
|
Kotlin version
|
5.2. Vector index on Neo4j
Activation: embabel.dice.store.vector-index.enabled=true — on by default, and only takes effect
with embabel.dice.store.type=graph and an Ai bean present.
Availability: dice-storage-autoconfigure. No extra infrastructure beyond the Neo4j you already
need for the graph backend.
5.2.1. When to use it
Leave it on. Semantic retrieval over propositions is what makes "what do we know that is relevant to
this?" answerable without an exact match, and it is also what lets VectorCandidateSearcher resolve
aliases that share no characters with the canonical name.
Turn it off when your Neo4j deployment does not support vector indexes, when you are certain all
retrieval is by entity and ContextId, or when the index build on a large existing store is
something you want to schedule rather than have happen at startup.
5.2.2. Impact
-
Startup: the index is created if absent. On a large existing store this takes time.
-
Write: each proposition is embedded on save — an embedding call per proposition.
-
Storage: one embedding vector per proposition.
-
Read: similarity search becomes a native index lookup rather than a scan.
5.2.3. How to enable
embabel:
dice:
store:
type: graph
vector-index:
enabled: true # the default
To disable:
embabel:
dice:
store:
vector-index:
enabled: false
5.2.4. What is configurable
Only enabled. The index’s label, property, name and similarity function are fixed by the
@VectorIndex annotation on PropositionNode.embedding and live as constants on
DrivinePropositionRepository.
This is deliberate: those values must agree between the index definition and every query that uses it, and making them configurable would create a way to have them disagree.
The vector dimension comes from the configured EmbeddingService, so the index matches whatever
embedding model your Ai bean is using.
5.3. Decay and stale pruning
Activation: embabel.dice.store.decay.enabled=true — on by default. Hard deletion is separate:
embabel.dice.store.decay.prune-stale, false by default.
Availability: dice-storage-autoconfigure, both backends. No extra infrastructure.
5.3.1. When to use it
Leave decay on. Without it, a claim made once two years ago competes on equal terms with one made this morning, and retrieval quality degrades as the store grows.
Turn prune-stale on only when you have decided that reclaiming rows matters more than being able
to answer "why did you stop believing that?". For most applications it does not.
5.3.2. Impact
-
Scheduling: one tick per interval, hourly by default.
-
Write: the tick materialises cached effective confidence and applies lifecycle transitions.
-
Latency: on the graph backend the tick executes in the database; on in-memory it walks the store in process. Size the interval accordingly.
-
LLM calls: none. Decay is arithmetic.
5.3.3. How to enable
embabel:
dice:
store:
decay:
enabled: true # default
interval-ms: 3600000 # default: 1 hour
k: 2.0 # decay-rate multiplier
prune-stale: false # default: keep STALE propositions
5.3.4. What the tick does
Two phases:
-
Materialise — recompute and cache effective confidence, so queries filtering on it do not have to compute decay per row.
-
Lifecycle — apply the
StatusTransitionPolicy, moving propositions whose effective confidence has fallen far enough into a stale status. Withprune-stale: true, hard-delete them.
Pinned propositions are exempt.
5.3.5. Tuning k
k is the decay-rate multiplier. Higher k means faster erosion. It appears in three places, and
they are independent:
-
embabel.dice.store.decay.k— used by the scheduled tick. -
PropositionQuery.decayK— used when a query computes effective confidence, defaults to2.0. -
Proposition.effectiveConfidence(k)— the direct call, defaults to2.0.
Setting the property without the query parameter means the tick and your queries disagree about how fast things fade. Set both, or leave both at the default.
5.3.6. Example
var p = repository.findById(id);
System.out.printf("raw=%.2f now=%.2f in 30 days=%.2f%n",
p.getConfidence(),
p.effectiveConfidence(),
p.effectiveConfidenceAt(Instant.now().plus(30, ChronoUnit.DAYS)));
Run this against a claim you care about before choosing k. The right value is the one where a
belief you would want to have forgotten has faded, and one you would want to keep has not.
5.4. Multi-signal collector
Activation: embabel.dice.collector.enabled — on unless explicitly set to false.
Availability: dice-storage-autoconfigure. No extra infrastructure; the trace store follows your
chosen backend.
5.4.1. When to use it
Use it when the same claim reaches you from more than one source and entity resolution alone has not collapsed the duplicates. That is most systems that ingest continuously.
Turn it off when your corpus is small enough to inspect, when duplicates are genuinely rare, or while you are still tuning extraction — merging is much harder to undo than to postpone.
5.4.2. Impact
-
LLM calls: none. Every signal is computed, not generated.
-
Latency: a run is proportional to the candidate pairs the pair sources produce, not to the store.
-
Storage: trace rows per run, retained per
detail-retention-days. -
Risk: this is the component that merges propositions. Dry-run first.
5.4.3. How it decides
MultiSignalCollectorStrategy scores each candidate pair on several signals and blends them. No
single signal can carry a merge:
| Signal | What it measures |
|---|---|
|
Embedding similarity between the two claims. |
|
Surface token overlap. |
|
How much the resolved entity sets agree. |
|
Whether the claims came from the same chunks. |
|
Whether they share evidence links. |
|
A veto, not a score: stops a claim merging with its own negation. |
An edge scoring at or above match-threshold becomes eligible to merge its endpoints. Eligible
edges form connected components, and CollectorSurvivorPolicy picks which proposition in each
component survives, absorbing the others' evidence.
5.4.4. How to enable
embabel:
dice:
collector:
enabled: true # default
match-threshold: 0.6 # aggregate score needed to merge; 0.0-1.0
signals:
vector:
enabled: true
weight: 0.4
similarity-threshold: 0.85 # cosine floor for recall
top-k: 20 # max similar members per cluster seed
lexical:
enabled: true
weight: 0.2
entity-overlap:
enabled: true
weight: 0.2
grounding-overlap:
enabled: true
provenance-overlap:
enabled: false
polarity-veto:
enabled: true
sweep:
delta: false # property binding only; no runner behind it yet
trace:
enabled: true
detail-retention-days: 30
A null weight means "use the scorer’s own default", which is not the same as zero — the nullable
type exists so "unset" and "set to the default number" stay distinguishable.
similarity-threshold and top-k apply to the vector signal only.
5.4.5. Tuning
Raise match-threshold if you see wrong merges. Lower it, or widen the vector signal’s
similarity-threshold and top-k, if duplicates survive.
Prefer tuning here over loosening entity resolution: reclamation is auditable and reversible in review, and a bad resolution is neither.
5.4.6. Example
// Always start here.
var dryRun = collectorRunner.run(contextId, /* dryRun = */ true);
dryRun.getDecisions().forEach(System.out::println);
// Only once the decisions look right.
var live = collectorRunner.run(contextId, false);
Every live run is recorded to CollectorRecordStore; the detailed edges, components and decisions go
to a CollectorTraceStore — InMemoryCollectorTraceStore by default, DrivineCollectorTraceStore
on the graph backend.
5.5. Concurrent extraction
Activation: a parallel or batched ExtractionExecutionStrategy on the pipeline —
pipeline.withExecutionStrategy(…). The default is SerialExtractionStrategy.
Availability: dice. No extra infrastructure; it raises concurrent load on your LLM provider.
5.5.1. When to use it
Use it for bulk ingestion, where extraction dominates wall-clock time and each chunk is an independent LLM call.
Leave it off for conversation analysis — one chunk at a time gains nothing — and until you have verified the condition below.
5.5.2. The condition you must verify first
|
Only the extraction stage is dispatched concurrently. Enabling a parallel or batched strategy in
production is gated on verifying that your A stateful extractor under a parallel strategy produces corrupted output that looks plausible. |
5.5.3. Impact
-
Latency: the extraction stage scales with the strategy’s parallelism. Resolution does not.
-
LLM calls: the same number, issued concurrently. Check your provider’s rate limits.
-
Cost: unchanged.
-
Memory: extraction results for in-flight chunks are held until the resolution stage consumes them.
5.5.4. What stays serial, and why
process(…) has two stages:
-
Extraction — stateless per chunk, touches no resolver. Dispatched by the strategy.
-
Resolution — reads and writes shared entity identity. Always serial.
Keeping resolution serial is what preserves cross-chunk entity identity regardless of how extraction was dispatched. An entity minted while processing chunk 2 is recognised in chunk 7 under every strategy.
Input order is preserved by the strategy, and a chunk whose extraction failed yields a null slot that becomes a typed failed result in the resolution stage — one bad chunk does not fail the batch.
5.5.5. How to enable
// The default.
pipeline.withExecutionStrategy(SerialExtractionStrategy.INSTANCE);
// Parallel.
pipeline.withExecutionStrategy(new ParallelExtractionStrategy(concurrency));
// Batched — batchSize > 1 is concurrent, batchSize == 1 is effectively serial.
pipeline.withExecutionStrategy(new BatchedExtractionStrategy(batchSize));
withExecutionStrategy returns a new pipeline, like every other with* method.
5.5.6. Example
var bulk = PropositionPipeline
.withExtractor(threadSafeExtractor)
.withExecutionStrategy(new ParallelExtractionStrategy(8));
var results = bulk.process(manyChunks, context);
results.persist(propositionRepository, namedEntityDataRepository);
Measure before and after. If extraction was not your bottleneck, this buys nothing and adds a thread-safety obligation you now have to maintain.
5.6. Web API
Activation: @Import(DiceRestConfiguration.class), plus the beans the controllers need
(PropositionPipeline, PropositionStore, and the rest) and spring-webmvc on the classpath.
Availability: dice. No extra module, no extra infrastructure.
5.6.1. When to use it
Use it when something outside the JVM needs to reach DICE — another service, a front end, a tool process.
Leave it off when DICE is a library inside one application. It is off by default because a knowledge store is not something to expose accidentally.
5.6.2. Impact
-
Surface: three controllers under
/api/v1. -
Security: none by default. Read API-key security before exposing this anywhere.
-
Latency and cost: the extraction endpoints make LLM calls, synchronously.
5.6.3. How to enable
@Configuration
@Import(DiceRestConfiguration.class)
public class MyAppConfiguration { }
DiceRestConfiguration imports the controllers. They are ordinary @RestController beans, and each
needs its collaborators present — importing the configuration without the beans behind it will fail
at startup rather than silently serve nothing.
5.6.4. The endpoints
| Method | Path | What it does |
|---|---|---|
|
|
Runs text through the pipeline and returns an |
|
|
Same, from a |
| Method | Path | What it does |
|---|---|---|
|
|
Lists propositions for the context, filterable by status and confidence. |
|
|
Semantic search within the context. |
|
|
Everything the context knows about one entity. |
|
|
Creates a proposition directly, bypassing extraction. |
|
|
Fetches one proposition. |
|
|
Deletes one proposition. |
| Method | Path | What it does |
|---|---|---|
|
|
Runs a discovery query. |
|
|
Finds paths between entities. |
|
|
Explains why a proposition is believed — its evidence and lineage. |
|
|
Reports what is stale in the projection. |
|
|
Reports what reclamation would do, without doing it. |
5.6.5. DTOs, not domain objects
The controllers speak in DTOs — PropositionDto, EntityMentionDto, ProvenanceEntryDto,
ExtractRequest, MemorySearchRequest and the rest — rather than serialising domain types. The
wire format is a contract you can evolve separately from the model, and internal fields do not leak.
ExtractOptions carries per-request extraction settings, including a SchemaAdherenceDto mapping
to Context and schema's three constants.
5.6.6. Example
curl -X POST http://localhost:8080/api/v1/extract \
-H 'Content-Type: application/json' \
-d '{
"text": "Ada Lovelace worked with Charles Babbage.",
"contextId": "demo",
"options": { "schemaAdherence": "DEFAULT" }
}'
|
As shown, that endpoint is unauthenticated and makes LLM calls on demand. Do not expose it without API-key security or your own security configuration. |
5.7. API-key security
Activation: dice.security.api-key.enabled=true
Availability: dice. No extra module, no extra infrastructure.
5.7.1. When to use it
Use it whenever Web API is exposed beyond your own process. The REST endpoints are unauthenticated without it, and the extraction endpoints make LLM calls on demand — an open one is both a data-exposure problem and a billing one.
Leave it off when the endpoints are not exposed, or when you already have Spring Security in front of them.
5.7.2. Impact
-
Latency: one filter, one set lookup per request.
-
Cost: none.
-
Operational: keys in configuration are keys in your configuration. See the warning below.
5.7.3. How to enable
dice:
security:
api-key:
enabled: true # default false
keys:
- ${DICE_API_KEY}
header-name: X-API-Key # default
path-patterns:
- /api/v1/** # default
With enabled: true, ApiKeySecurityAutoConfiguration registers an InMemoryApiKeyAuthenticator
over the configured keys and a servlet filter over the configured path patterns.
If you enable it and configure no keys, the authenticator logs a warning at startup — that message means every request is being rejected.
5.7.4. Use your own authenticator in production
The in-memory list is for development. Provide your own ApiKeyAuthenticator bean and the
auto-configuration steps aside, because it is @ConditionalOnMissingBean:
@Bean
ApiKeyAuthenticator apiKeyAuthenticator(SecretStore secrets) {
return request -> secrets.isValid(request.getHeader("X-API-Key"));
}
That is where key rotation, per-key scoping and revocation belong.
|
Keys listed under |
5.7.5. Scope of protection
The filter protects path-patterns, which defaults to /api/v1/**. Anything outside that pattern —
actuator endpoints, your own controllers, static resources — is untouched. If you mount DICE’s
controllers under a different prefix, update the patterns to match, or you will have enabled
security that guards nothing.
5.8. Prolog inference
Activation: a PrologProjector bean. Nothing is wired by default.
Availability: dice, via tuProlog (2p-kt). No extra infrastructure — the engine is a pure Kotlin
library running in your JVM.
|
Experimental. The API may change without the usual deprecation cycle. |
5.8.1. When to use it
Use it when the questions you need answered are derived rather than stored: transitive closure, mutual exclusion, "who else is affected if this is true", rules a domain expert can read and check.
Leave it off when retrieval answers your questions. A vector search and a graph traversal cover most of what applications ask, and neither requires you to maintain a rule base.
5.8.2. Impact
-
LLM calls: none. Inference is symbolic.
-
Memory: the fact base is materialised in the JVM.
-
Latency: query time depends on your rules, not on the store. An unbounded recursive rule will hang.
-
Maintenance: rules are code, and they need tests like any other code.
5.8.3. How to enable
Provide a projector, project, then query:
@Bean
PrologProjector prologProjector(/* collaborators */) {
return new DefaultPrologProjector(/* ... */);
}
// Propositions -> projected relationships -> Prolog facts.
PrologProjectionResult result = prologProjector.projectAll(relationships);
// Load the facts and your rules into an engine.
PrologEngine engine = PrologEngine.of(result.getFacts(), rules);
5.8.4. Querying
boolean holds = engine.query("collaborated(ada, babbage)");
List<QueryResult> rs = engine.queryAll("collaborated(ada, X)");
QueryResult first = engine.queryFirst("collaborated(ada, X)");
List<String> names = engine.findAll("collaborated(ada, X)", "X");
query is the yes/no form. queryAll returns every solution, queryFirst the first, and findAll
extracts the bindings of one variable as strings — usually what you want when the answer is a list
of entity ids.
5.8.5. Through the Oracle
PrologTools exposes the fact base to ToolOracle as a tool, so a natural-language question that
needs inference rather than retrieval can be answered by running rules. See Ask questions in natural language.
5.8.6. What a projected fact looks like
PrologProjector extends the generic Projector and produces PrologFact values, one per
projected relationship. The projection is derived — if the propositions change, re-project rather
than editing facts.
PrologTypes holds the type mapping between DICE’s model and Prolog terms.
6. Reference
6.1. Configuration properties
Every property DICE binds, with its default and the module that reads it.
6.1.1. embabel.dice.store — DiceStoreProperties
Read by dice-storage-autoconfigure.
| Property | Type | Default | Meaning |
|---|---|---|---|
|
|
|
Backend kind: |
|
|
|
Whether the scheduled decay tick runs. |
|
|
|
Delay between ticks, in milliseconds. One hour. |
|
|
|
Decay-rate multiplier used by the staleness policy. |
|
|
|
Hard-delete |
|
|
|
Whether the proposition embedding vector index is created. Graph backend only. |
The vector index’s label, property, name and similarity are not configurable — they are fixed by
the @VectorIndex annotation on PropositionNode.embedding and live as constants on
DrivinePropositionRepository.
6.1.2. embabel.dice.collector — CollectorProperties
Read by dice-storage-autoconfigure. Validated: match-threshold must be in [0.0, 1.0] and
signal weights must be non-negative.
| Property | Type | Default | Meaning |
|---|---|---|---|
|
|
|
Master switch for the whole collector. |
|
|
|
Minimum aggregate score for an edge to be eligible to merge its endpoints. |
|
|
|
Whether this signal participates. |
|
|
|
Contribution weight. |
|
|
|
Cosine floor for candidate recall. Vector signal only. |
|
|
|
Max similar members considered per cluster seed. Vector signal only. |
|
|
|
Property binding only — there is no runner behind it yet. |
|
|
|
Whether runs are recorded to a |
|
|
|
How long detailed trace rows are kept. |
Valid <name> values are the built-in signals: vector, lexical, entity-overlap,
grounding-overlap, provenance-overlap, polarity-veto.
6.1.3. dice.security.api-key — DiceApiKeyProperties
Read by dice. Note the prefix: dice., not embabel.dice..
| Property | Type | Default | Meaning |
|---|---|---|---|
|
|
|
Enable API-key authentication over the REST endpoints. |
|
|
empty |
Valid API keys. Development only — supply an |
|
|
|
Header carrying the key. |
|
|
|
Path patterns the filter protects. |
6.1.4. embabel.dice.source-analyzer
Read by dice, bound by LlmSourceAnalyzer. Configures the text-to-graph source analyser.
6.1.5. Properties DICE does not own
An LLM provider must be configured, and that configuration belongs to embabel-agent, not to DICE. Neo4j connection settings belong to Drivine. DICE reads neither.
6.1.6. A complete example
embabel:
dice:
store:
type: graph
decay:
enabled: true
interval-ms: 3600000
k: 2.0
prune-stale: false
vector-index:
enabled: true
collector:
enabled: true
match-threshold: 0.65
signals:
vector:
weight: 0.4
similarity-threshold: 0.85
top-k: 20
lexical:
weight: 0.2
trace:
enabled: true
detail-retention-days: 30
dice:
security:
api-key:
enabled: true
keys:
- ${DICE_API_KEY}
6.2. Package structure
The dice module is organised by responsibility. This is the map for finding the type you need.
| Package | What lives there |
|---|---|
|
|
|
|
|
|
|
|
|
Admission gates: |
|
|
|
Policy extension points: |
|
Shared types: |
|
|
|
|
|
|
|
The entity-only pipeline: |
|
|
|
|
|
|
|
|
|
|
|
|
|
Grounding support — the link from claims back to chunks. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Agent-facing integration with embabel-agent. |
|
|
6.2.1. The other modules
| Package | What lives there |
|---|---|
|
|
|
|
|
|
|
|
6.3. Extension points
The interfaces DICE expects you to implement. Everything here has a shipped default, so you extend only what your domain actually needs.
6.3.1. Belief and conflict — com.embabel.dice.spi
| SPI | What you decide |
|---|---|
|
How much a claim is worth, given its confidence and where it came from.
|
|
Which authority tier a source belongs to. Feeds trust scoring and query-time filtering. |
|
Whether two claims conflict, and how — the |
|
When a proposition changes |
|
What happens to a marked proposition. |
|
Where reclamation decision traces are persisted. |
6.3.2. Extraction and resolution
| SPI | What you decide |
|---|---|
|
How text becomes claims. |
|
How a new claim is reconciled against stored ones. |
|
How a mention becomes an entity. Several ship — see Entity resolution. |
|
How candidate entities are recalled for a mention. This is the usual place to add a domain-specific strategy. |
|
How one candidate is chosen from several. |
|
Which mentions are worth resolving at all. |
|
Which extracted propositions are admitted. |
|
How the extraction stage is dispatched. Serial, parallel and batched ship. |
6.3.3. Storage
| SPI | What you decide |
|---|---|
|
The base port: CRUD plus a composable query. Implement this to add a backend. |
|
|
|
How decay is applied. |
|
What has already been processed, for incremental analysis. |
|
Edge lineage, for the stale cascade and reconciliation. |
|
The reclamation audit trail. |
|
The content-hash record that makes re-ingestion a no-op. |
|
How an artifact becomes chunks. |
6.3.4. Projection and output
| SPI | What you decide |
|---|---|
|
Which edges propositions produce. Relation-based and LLM-based ship. |
|
Which propositions are eligible to be projected at all. |
|
Human-readable descriptions for projected edges. |
|
Propositions as Prolog facts. Experimental. |
|
How propositions map to agent memory, and how memory is recalled. |
|
Which propositions get marked for reclamation. |
|
Human-readable output. In |
|
Surprising connections between claims. |
6.3.5. Java interop
Kotlin infix factory methods on PropositionQuery have Java-friendly withXxx and
againstContext equivalents. ContextId is a value class, so Java code uses getContextIdValue().
Proposition.create(…) is the Java-friendly factory. @JvmOverloads is applied where defaults
would otherwise be unreachable from Java.
7. Running in production
What changes when this stops being a quickstart.
7.1. Move off the in-memory store
The in-memory backend loses everything on restart and assumes one writer. Switch to the graph backend before you have data you care about — see Choose a storage backend and Graph-backed storage.
There is no migration tool, and none is needed for a small store: propositions are the system of record, so a migration is read-all, write-all, re-project. Reconciliation is idempotent, so re-projection is safe to repeat and safe to interrupt.
7.2. Choose decay settings deliberately
The defaults — hourly ticks, k=2.0, no pruning — are a reasonable start and a poor finish.
-
Interval: hourly is fine for a store measured in thousands. Lengthen it if the tick’s cost shows up in your metrics.
-
k: set it from how fast your domain’s facts actually go stale. UseeffectiveConfidenceAton real propositions to see what a value means before committing to it. -
Query
decayK: the property configures the tick, not your queries. If you change one, change both, or the two will disagree about how fast beliefs fade. -
prune-stale: leave itfalseunless you have decided that reclaiming rows beats being able to explain why a belief was dropped.
7.3. Concurrency
Extraction is the parallelisable stage and resolution is not — that is a property of the design, not
a limitation to work around. Before enabling a concurrent ExtractionExecutionStrategy, verify your
PropositionExtractor is thread-safe. See Concurrent extraction.
Persistence is yours to bound. The pipeline returns unsaved results and never opens a transaction,
so persist(…) runs inside whatever scope you put it in. On the graph backend, that is the scope
Drivine’s transaction management sees.
7.4. LLM cost
DICE makes LLM calls in five places. Four are optional:
| Call site | Optional | How to reduce it |
|---|---|---|
Extraction |
No |
Chunk larger; use |
Revision |
Yes |
Only enable it where restatement actually happens — conversation, not one-shot documents. |
Entity resolution |
Yes |
The escalating resolver only reaches an LLM at the top two rungs. Fix candidate recall and most resolutions become free. See Tune entity resolution. |
Consolidation passes |
Yes |
Threshold-gated by |
Reports and rationale |
Yes |
Generate on demand rather than eagerly. |
The multi-signal collector makes no LLM calls, which is why tuning duplicates there is cheaper than tuning them at resolution.
7.5. Observability
Instrument these four:
-
Resolution levels. The
ResolutionLeveldistribution tells you whether resolution is cheap and correct. A risingLLM_BAKEOFFrate is a cost problem; a risingNO_MATCHrate with minting on is a duplication problem. -
Gate rejections. Wrap gates in
ObservableGateand filters inObservableMentionFilter. A mis-tuned gate fails silently, and silence looks like success. -
Projection health.
projectionHealth(contextId)reports what is stale — the signal that reconciliation is due. -
Collector traces. Every live run is recorded. Set
detail-retention-daysto something you can actually afford to store, and read a trace before you trust a merge.
Wire EventEmittingPropositionRepository and EventEmittingProjector if you want to push these into
your own telemetry. PropositionPersisted is the canonical durable signal — pipeline candidate
events are pre-persistence and do not mean anything was written.
7.6. Schedule hygiene
Three passes, three cadences. None of them is wired to run automatically except the decay tick:
-
Decay tick — automatic, hourly by default.
-
Reclamation — run it dry, read the trace, then run it live. Weekly is a reasonable starting cadence for a busy store.
-
Consolidation — the dream loop is threshold-gated, so scheduling it often is cheap. Overnight, or per-session for conversational contexts.
Pin anything that must never fade, and pin sparingly: every pin is a permanent claim on retrieval.
7.7. Security
The REST endpoints are opt-in and unauthenticated by default. If you import
DiceRestConfiguration, enable API-key security or put your own security in front of
it — the extraction endpoints make LLM calls on demand, so an open one is a billing exposure as well
as a data one.
Supply your own ApiKeyAuthenticator in production. The in-memory key list is for development, and
keys in configuration files are keys in your configuration files.
Propositions carry what users told you. ContextId is the isolation boundary, and it is only as
good as the discipline of always scoping queries by it — which is why PropositionQuery has no
unscoped factory.
8. Support
8.1. Compatibility
|
DICE is pre-1.0. These values track what the build currently uses and are not yet a support commitment. They get pinned at the 1.0 release. |
8.1.1. Current build
| Axis | Current | Notes |
|---|---|---|
DICE |
0.2.0 |
Pre-1.0. The API may change between minor versions. |
embabel-agent |
1.5.2 |
|
JDK |
21 |
The baseline inherited from |
Spring Boot |
Inherited from embabel-agent’s dependency management |
DICE does not pin Spring Boot itself. |
Neo4j |
Reached through Drivine ( |
Only needed for the graph backend. The in-memory and JSON-file backends need no Neo4j. |
Kotlin |
2.1.10 ( |
A build concern, not a consumer one. The storage modules need the 2.2 compiler for Drivine’s
KSP-generated |
tuProlog (2p-kt) |
1.0.4, pinned |
1.1.x is built with Kotlin 2.2.x and is binary-incompatible with `dice’s Kotlin 2.1.10. Upgrading requires Kotlin 2.2.21+. |
8.1.2. Feature availability by module
| Feature | Needs | Status |
|---|---|---|
Proposition pipeline, revision, gates |
|
Stable |
Entity resolution |
|
Stable |
In-memory and JSON-file stores |
|
Stable |
Graph projection |
|
Stable |
Memory projection and recall |
|
Stable |
Oracle |
|
Stable |
REST endpoints |
|
Opt-in |
API-key security |
|
Opt-in |
Graph-backed storage |
|
Stable |
Vector index |
|
Stable |
Decay tick |
|
Stable |
Multi-signal collector |
|
Stable |
Reports, rationale, link discovery |
|
Stable |
Ingestion SPI and ledger |
|
Stable |
Prolog inference |
|
Experimental |
8.1.3. Building DICE itself
mvn verify # full build with tests
mvn install -DskipTests # faster, no tests
mvn test -pl dice # one module
mvn verify -pl dice-storage # needs Docker
dice-storage integration tests start a Neo4j container through Testcontainers, so Docker must be
running. The root pom pins api.version=1.41 as a Surefire JVM system property to meet the Docker
engine’s minimum API version — without it, container-backed tests fail with HTTP 400.
Drivine’s query DSL is generated by a nested Gradle/KSP project at dice-storage/codegen-gradle,
run automatically during generate-sources. Do not edit the generated files.
Snapshots and releases resolve from repo.embabel.com. An environment without internet access needs
those repositories mirrored locally.
8.2. FAQ
- I called
process(…)and nothing was saved. -
That is correct behaviour. The pipeline returns
PersistablePropositionsand stores nothing until you callpersist(propositionRepository, namedEntityDataRepository)inside your own transaction. If you drop the result, the work is discarded silently. - Why is there no
PropositionQuery.create()? -
Deliberate. Without a required scope, "load everything" would be the shortest thing to write, and in a store that accumulates indefinitely that is a production incident. Start from
againstContext(…)orforContextId(…). - Should I filter on
confidenceoreffectiveConfidence? -
effectiveConfidence, almost always.confidenceis what the extractor thought at extraction time and never changes;effectiveConfidencefolds in decay and is what "do we still believe this?" means. - The same entity appears twice.
-
Resolution did not match the second mention. Look at the
ResolutionLeveldistribution first: heavyNO_MATCHwith minting on means candidate recall is too narrow. Fix the cheap searchers before touching the LLM rungs — see Tune entity resolution. Reclamation can collapse what got through. - Two different entities got merged into one.
-
More serious, and harder to undo. Check the collector’s
match-thresholdand the trace for that merge, and check whether a fuzzy or partial-name searcher is recalling too widely. Raise the threshold and re-run dry. - Why did decay not reset after I updated a proposition?
-
Decay is anchored on
contentRevised, which moves only when the claim’s text changes. Status changes, pinning and grounding touchmetadataRevisedinstead. This is what stops housekeeping from making stale beliefs look fresh. - My queries and the decay tick disagree about how fast things fade.
-
embabel.dice.store.decay.kconfigures the tick.PropositionQuery.decayKandeffectiveConfidence(k)default to2.0independently. Set both, or leave both alone. - Can I run extraction in parallel?
-
Yes, with an
ExtractionExecutionStrategy— but only after verifying yourPropositionExtractoris thread-safe. Resolution stays serial regardless, which is what preserves cross-chunk entity identity. See Concurrent extraction. - Why is resolution serial when extraction is not?
-
Resolution reads and writes shared entity identity. Running it concurrently is how you get two entities for one thing.
- The REST endpoints return 404.
-
They are opt-in. You need
@Import(DiceRestConfiguration.class),spring-webmvcon the classpath, and the beans the controllers depend on. A missing bean fails at startup rather than 404-ing, so a 404 usually means the configuration was never imported. - I enabled API-key security and now everything is 401.
-
Check the startup log. If no keys were configured,
ApiKeySecurityAutoConfigurationlogs a warning and every request is rejected. Also checkpath-patternsmatches where your controllers actually are. - Do I need Neo4j?
-
No. The in-memory backend is the default and needs nothing. Neo4j is for durability, concurrent writers and native traversal — see Choose a storage backend.
- How do I migrate from in-memory to the graph backend?
-
Read every proposition and entity from the old store, write them to the new one, re-project. There is no tool because propositions are the system of record and everything else is derived.
- What should a
ContextIdrepresent? -
Your isolation boundary — usually one per user, tenant, conversation or document collection. Contexts do not nest. If you need a hierarchy, encode it in the value and query by prefix at your own layer.
- Reclamation deleted something I needed.
-
It should not have deleted anything: DICE decays rather than deletes, and hard deletion requires
embabel.dice.store.decay.prune-stale=true, which is off by default. Pin propositions that must never fade, and always dry-run reclamation first. - Where do I find out why the system behaves this way?
-
The design notes under
docs/design/in the repository. This guide covers use; those cover rationale.
9. Resources
9.1. Repositories
-
embabel/dice — this project.
-
embabel/embabel-agent — the agent framework DICE builds on.
-
embabel/impromptu — a classical-music chatbot using DICE in production.
9.2. Design notes
The contributor-facing rationale lives in docs/design/ in the repository, with an index at
docs/design/INDEX.md. Start with architecture.md for a system-level map, then follow to the
extraction pipeline, the proposition lifecycle, knowledge hygiene, graph projection, retrieval and
discovery, durable storage, and the event model.
AGENTS.md at the repository root is the navigation guide for working on DICE rather than with
it.
9.3. Background reading
Johnson, R. (2025). Context Engineering Needs Domain Understanding. Medium. medium.com/@springrod/context-engineering-needs-domain-understanding-b4387e8e4bf8
General User Models (GUM), Stanford and Microsoft. arxiv.org/abs/2505.10831 — the propose/retrieve/revise architecture DICE’s proposition model is derived from.
The Blackboard pattern — the shared-memory architecture the agent framework’s execution model uses.
9.4. Technology
-
tuProlog (2p-kt) — pure Kotlin Prolog engine, used for inference.
-
Drivine — Neo4j access for the graph backend.
-
Spring Framework and Spring Boot — dependency injection, auto-configuration, optional web support.
-
Embabel Agent — LLM integration.
-
Kotlin — the implementation language, with Java-friendly APIs throughout.