DICE API Documentation/com.embabel.dice.incremental/AbstractIncrementalAnalyzer

AbstractIncrementalAnalyzer

abstract class AbstractIncrementalAnalyzer<T, R>(historyStore: ChunkHistoryStore, formatter: IncrementalSourceFormatter<T>, config: WindowConfig = WindowConfig(), contentHasher: ContentHasher = Sha256ContentHasher) : IncrementalAnalyzer<T, R>

Handy base implementation of IncrementalAnalyzer providing:

  • Tracking of what's been processed to avoid re-processing

  • Windowing with overlap to maintain context

  • Hash-based deduplication

Subclasses implement processChunk to produce their specific result type.

Parameters

T

The type of items in the source

R

The type of result produced by analysis

historyStore

Tracks processing history (caller provides implementation)

formatter

Formats source items to text

config

Window and trigger configuration

Inheritors

Constructors

Link copied to clipboard
constructor(historyStore: ChunkHistoryStore, formatter: IncrementalSourceFormatter<T>, config: WindowConfig = WindowConfig(), contentHasher: ContentHasher = Sha256ContentHasher)

Functions

Link copied to clipboard
open override fun analyze(source: IncrementalSource<T>, context: SourceAnalysisContext): R?

Analyze the source if enough new content has accumulated.

Link copied to clipboard
open override fun analyzeNow(source: IncrementalSource<T>, context: SourceAnalysisContext): R?

Force analysis regardless of trigger interval. Implementations may still skip if content was already processed.