Extraction¶
Extractors convert corpus blobs into candidate particles (Client layer). The
Engine-side ingest pipeline (particles.ingest) reconciles them,
applying conflict resolution and writing to the store.
Pipeline¶
particles.ingest.pipeline.extract_snapshot(session, entry_id, snapshot_id, extractor=None, agent_id=_DEFAULT_AGENT, page_stats_out=None, supersede_ids=frozenset(), carry_forward_ids_out=None, suppressed_ids_out=None, completion_pool=None, skip_if_superseded=False, divergences_out=None, outcome_out=None, append_base=AUTO_BASE, ignore_partial=False)
async
¶
Run extraction for a single corpus snapshot.
Returns the list of Particle objects written to the store (ACTIVE or INCONSISTENCY).
Extractor is selected from the plugin registry by source_type.
If page_stats_out is provided, page-level stats from PDF extraction are appended to it.
If carry_forward_ids_out is provided, it is extended with the IDs of
existing ACTIVE particles that the extractor's chunk-hash carry-forward
matched; reindex callers should exclude these from supersession.
If suppressed_ids_out is provided, it is extended with the ID of the
existing ACTIVE particle each suppressed duplicate candidate was folded into,
one entry per suppressed candidate, so len() is the suppression count.
Like carry-forward, these particles must be excluded from supersession.
If divergences_out is provided, it is incremented by the pairs
the observer precondition declined and by how many of them this pass joined
with a CONTRADICTS relation — the count the quality note carries.
If outcome_out is provided, it is filled with how the pass went: LLM calls
that failed (the snapshot is left PENDING and nothing is written) and
replies cut at the output budget (some claims were lost).
completion_pool is the latency-tolerance assertion, threaded
as a parameter and never sniffed: only the consolidation extract pass
passes one, and pool-aware extractors then merge their LLM requests into
the pooled half-price batch. Interactive callers leave it None and get
today's sequential calls unchanged.
skip_if_superseded is passed True by the bulk paths: a
snapshot collapsed since the caller listed its work — by this process or
another — is skipped with no claim and no LLM call. The default False
is the explicit extract <entry> --snapshot-id contract: naming a
generation extracts it, and the claim un-collapses it.
append_base decides what an APPEND_ONLY snapshot read by the
general extractor is read as a delta from: "auto" (the default) finds
the latest earlier COMPLETE snapshot, a snapshot id names it, and None
reads the snapshot whole with no base. Only the reindex replay passes
anything but the default.
A partly failed APPEND_ONLY read keeps what it answered and leaves the
snapshot PENDING only under the default append_base.
ignore_partial is a reindex's: it plans as if the snapshot held no
partial read of its own, skips the whole-read wait, and keeps nothing
from a failed read, since the reindex is about to retire those claims.
Span-wrapped: the work runs under an extract.snapshot span so a
request's time localizes across the embed / LLM / DB child spans, and the
written-particle count feeds the particles.extracted throughput metric.
Snapshot generations¶
particles.ingest.generation.cascade_superseded_generation(session, *, entry_id, current_snapshot_id, exclude_ids=frozenset())
async
¶
Demote ACTIVE particles anchored to a superseded snapshot of entry_id.
A no-op for every entry whose mutability is not MUTABLE, so callers
need not pre-filter: APPEND_ONLY content is additive by definition,
STABLE never changes, and EPHEMERAL is not archived.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
entry_id
|
str
|
The corpus entry whose generations are being reconciled. |
required |
current_snapshot_id
|
str
|
The snapshot that has just been extracted. Every other snapshot of this entry is superseded by it. |
required |
exclude_ids
|
frozenset[str]
|
Particle ids to leave ACTIVE regardless. Callers pass the carry-forward ids. A carried-forward particle's provenance edge keeps naming the snapshot it was originally extracted from (the edge is never re-pointed), so without this it would be misread as a stale generation and demoted. A belt since the re-observation is now also on the particle's provenance, which the attestation check below reads. |
frozenset()
|
A candidate some entry's current snapshot still attests is skipped—
a claim a live source states is not PROVENANCE_STALE
under any reading of that status. The cascade was designed for
single-entry particles; a claim folded from several sources
retires only when none of them states it any more. "Current" is read as
the scope join reads it: this entry's current_snapshot_id, another
MUTABLE entry's latest extracted snapshot, or any snapshot of an entry
of another mutability.
Returns:
| Type | Description |
|---|---|
list[str]
|
The ids demoted, in query order. Does not commit; the caller owns the |
list[str]
|
transaction. |
particles.ingest.generation.backfill_superseded_generations(session, *, dry_run=True)
async
¶
Apply the §2 cascade to entries whose snapshots moved before.
The forward-looking cascade only fires on newly-extracted snapshots, so
stores that predate it carry an accumulated backlog. This walks every
MUTABLE entry with more than one snapshot and demotes what is anchored
to a generation older than that entry's latest COMPLETE snapshot.
"Latest COMPLETE" rather than "latest" is deliberate: if the newest snapshot
is still PENDING, the replacement beliefs do not exist yet, and retiring
the old generation would leave the store with neither.
"COMPLETE" here means an extracted RESPONSE snapshot: a REVISIT is COMPLETE
too, but it records that the content did not change and carries no beliefs
of its own, so it is never the current generation.
dry_run counts without writing (it does not write-then-roll-back), so a
caller may safely share its session with other work. Does not commit; the
caller owns the transaction.
Candidate types¶
particles.extraction.general.CandidateParticle
dataclass
¶
An extractor's proposed particle before conflict resolution and storage.
Attributes:
| Name | Type | Description |
|---|---|---|
content |
str
|
The claim text. |
confidence_value |
float
|
Self-assessed confidence in [0, 1]. |
uncertainty_nature |
UncertaintyNature
|
EPISTEMIC or ALEATORY. |
subjects |
list[str]
|
Subject names/QIDs this claim is about; resolved to UUIDs by the pipeline. |
properties |
dict[str, object] | None
|
Nomisma ontology-keyed structured data; None for free-text.
Also carries the document-scope tag ( |
subject_classes |
dict[str, str]
|
Maps subject name → Nomisma class applied after subject resolution. |
particles.extraction.general.ExtractionResult
dataclass
¶
The complete output of one extractor run.
Attributes:
| Name | Type | Description |
|---|---|---|
candidates |
list[CandidateParticle]
|
Proposed particles ready for conflict resolution. |
quality_notes |
list[str]
|
Human-readable notes about extraction quality or errors. |
page_stats |
list[PageStat]
|
Per-page/chunk statistics (PDF and HTML chunked extractors). |
particles.extraction.general.candidate_to_particle(candidate, corpus_entry_id, snapshot_id, asserted_by=EXTRACTOR_ID, subject_ids=None, extractor_ref=None, calibration=None)
¶
Convert a CandidateParticle to a Particle ready for insertion.
When calibration is None (the default and the historical
behaviour), the constructed particle carries
calibration_source=EXTRACTOR_DIRECT and the raw
candidate.confidence_value. When a calibration record is supplied,
the raw value is passed through a
:class:particles.extraction.calibration.TemperatureScaler and the
particle carries calibration_source=CALIBRATED_BENCHMARK,
calibration_method="temperature_scaling", and a
calibration_ref of the form "<extractor_id>:<fitted_at_iso>"
so the audit trail back to the fit is grep-able.
A supplied record whose transform this SDK will not apply (
today, every fit predating it) is treated exactly as no record: the
particle carries the raw value stamped EXTRACTOR_DIRECT. So is a record
fitted under an extractor version other than extractor_ref.version,
or under an unknown one, when extractor_ref is given.
A candidate that declares its own calibration_source (today,
a migration extractor stamping IMPORTED) overrides both branches: the
raw value is stored as given and no calibration is applied, because the
number is not a model output for a scaler to correct.
Extractors¶
particles.extraction.general.GeneralExtractor
¶
General-purpose LLM extractor. Accepts any source type as a fallback.
PDFs are extracted page-by-page. HTML is converted to Markdown
and processed in 15 K-character chunks. All other content types
are processed in a single LLM call. Requires ANTHROPIC_API_KEY.
extract(snapshot, content, **kwargs)
async
¶
Extract claim-granularity particles from snapshot content.
PDF sources use per-page extraction; all other sources use
a single-pass call after HTML-to-Markdown preprocessing.
LOCAL_MARKDOWN sources have their Obsidian YAML frontmatter
stripped before extraction so the LLM does not extract
metadata-key claims from it.
When the decoded text exceeds the chunked-extraction threshold, the
HTML path routes through extract_with_carry_forward.
That helper consults the particle store for prior chunks whose hash
matches the current chunk's text, so session and
corpus_entry_id are read from kwargs (the pipeline already
passes them).
particles.extraction.wikidata.WikidataExtractor
¶
Extracts particles from a stored Wikibase REST API JSON blob.
particles.extraction.numista.NumistaCoinExtractor
¶
Extracts rich particles from a single Numista coin-type API response.
particles.extraction.numista.NumistaIssuerExtractor
¶
Extracts structured particles from a combined Numista issuer search response.
Produces one structured particle per coin using the summary fields available from the issuer search API (composition, weight, diameter, catalog refs). Fields only available from the individual coin endpoint (currency, demonetization, edge, obverse/reverse) are absent; deposit individual coin pages for full infoboxes.
particles.extraction.numista.NumistaListingExtractor
¶
Parses Numista catalogue listing HTML (NUMISTA_LISTING_HTML source type).
Extracts per-coin structured particles from