Extraction¶
Extractors convert corpus blobs into candidate particles (Client layer). The
Engine-side ingest pipeline (particles.ingest) reconciles them —
applying conflict resolution and writing to the store.
Pipeline¶
particles.ingest.pipeline.extract_snapshot(session, entry_id, snapshot_id, extractor=None, agent_id=_DEFAULT_AGENT, page_stats_out=None, supersede_ids=frozenset(), carry_forward_ids_out=None, suppressed_ids_out=None, completion_pool=None)
async
¶
Run extraction for a single corpus snapshot.
Returns the list of Particle objects written to the store (ACTIVE or INCONSISTENCY).
Extractor is selected from the plugin registry by source_type.
If page_stats_out is provided, page-level stats from PDF extraction are appended to it.
If carry_forward_ids_out is provided, it is extended with the IDs of
existing ACTIVE particles that the extractor's chunk-hash carry-forward
matched — reindex callers should exclude these from supersession.
If suppressed_ids_out is provided, it is extended with the ID of the
existing ACTIVE particle each suppressed duplicate candidate was folded into —
one entry per suppressed candidate, so len() is the suppression count.
Like carry-forward, these particles must be excluded from supersession.
completion_pool is the latency-tolerance assertion, threaded
as a parameter and never sniffed: only the consolidation extract pass
passes one, and pool-aware extractors then merge their LLM requests into
the pooled half-price batch. Interactive callers leave it None and get
today's sequential calls unchanged.
Span-wrapped: the work runs under an extract.snapshot span so a
request's time localizes across the embed / LLM / DB child spans, and the
written-particle count feeds the particles.extracted throughput metric.
Snapshot generations¶
particles.ingest.generation.cascade_superseded_generation(session, *, entry_id, current_snapshot_id, exclude_ids=frozenset())
async
¶
Demote ACTIVE particles anchored to a superseded snapshot of entry_id.
A no-op for every entry whose mutability is not MUTABLE — APPEND_ONLY
content is additive by definition, STABLE never changes, and
EPHEMERAL is not archived — so callers need not pre-filter.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
entry_id
|
str
|
The corpus entry whose generations are being reconciled. |
required |
current_snapshot_id
|
str
|
The snapshot that has just been extracted. Every other snapshot of this entry is superseded by it. |
required |
exclude_ids
|
frozenset[str]
|
Particle ids to leave ACTIVE regardless. Callers pass the carry-forward ids — a carried-forward particle keeps pointing at the snapshot it was originally extracted from (provenance is deliberately not mutated), so without this it would be misread as a stale generation and demoted. |
frozenset()
|
Returns:
| Type | Description |
|---|---|
list[str]
|
The ids demoted, in query order. Does not commit — the caller owns the |
list[str]
|
transaction. |
particles.ingest.generation.backfill_superseded_generations(session, *, dry_run=True)
async
¶
Apply the §2 cascade to entries whose snapshots moved before.
The forward-looking cascade only fires on newly-extracted snapshots, so
stores that predate it carry an accumulated backlog. This walks every
MUTABLE entry with more than one snapshot and demotes what is anchored
to a generation older than that entry's latest COMPLETE snapshot.
"Latest COMPLETE" rather than "latest" is deliberate: if the newest snapshot
is still PENDING, the replacement beliefs do not exist yet, and retiring
the old generation would leave the store with neither.
dry_run counts without writing — it does not write-then-roll-back, so a
caller may safely share its session with other work. Does not commit; the
caller owns the transaction.
Candidate types¶
particles.extraction.general.CandidateParticle
dataclass
¶
An extractor's proposed particle before conflict resolution and storage.
Attributes:
| Name | Type | Description |
|---|---|---|
content |
str
|
The claim text. |
confidence_value |
float
|
Self-assessed confidence in [0, 1]. |
uncertainty_nature |
UncertaintyNature
|
EPISTEMIC or ALEATORY. |
subjects |
list[str]
|
Subject names/QIDs this claim is about; resolved to UUIDs by the pipeline. |
properties |
dict[str, object] | None
|
Nomisma ontology-keyed structured data; None for free-text.
Also carries the document-scope tag ( |
subject_classes |
dict[str, str]
|
Maps subject name → Nomisma class applied after subject resolution. |
particles.extraction.general.ExtractionResult
dataclass
¶
The complete output of one extractor run.
Attributes:
| Name | Type | Description |
|---|---|---|
candidates |
list[CandidateParticle]
|
Proposed particles ready for conflict resolution. |
quality_notes |
list[str]
|
Human-readable notes about extraction quality or errors. |
page_stats |
list[PageStat]
|
Per-page/chunk statistics (PDF and HTML chunked extractors). |
particles.extraction.general.candidate_to_particle(candidate, corpus_entry_id, snapshot_id, asserted_by=EXTRACTOR_ID, subject_ids=None, extractor_ref=None, calibration=None)
¶
Convert a CandidateParticle to a Particle ready for insertion.
When calibration is None (the default and the historical
behaviour), the constructed particle carries
calibration_source=EXTRACTOR_DIRECT and the raw
candidate.confidence_value. When a calibration record is supplied
, the raw value is passed through a
:class:particles.extraction.calibration.TemperatureScaler and the
particle carries calibration_source=CALIBRATED_BENCHMARK,
calibration_method="temperature_scaling", and a
calibration_ref of the form "<extractor_id>:<fitted_at_iso>"
so the audit trail back to the fit is grep-able.
A supplied record whose transform this SDK will not apply (
today, every fit predating it) is treated exactly as no record: the
particle carries the raw value stamped EXTRACTOR_DIRECT.
A candidate that declares its own calibration_source (today,
a migration extractor stamping IMPORTED) overrides both branches: the
raw value is stored as given and no calibration is applied, because the
number is not a model output for a scaler to correct.
Extractors¶
particles.extraction.general.GeneralExtractor
¶
General-purpose LLM extractor. Accepts any source type as a fallback.
PDFs are extracted page-by-page. HTML is converted to Markdown
and processed in 15 K-character chunks. All other content types
are processed in a single LLM call. Requires ANTHROPIC_API_KEY.
extract(snapshot, content, **kwargs)
async
¶
Extract claim-granularity particles from snapshot content.
PDF sources use per-page extraction; all other sources use
a single-pass call after HTML-to-Markdown preprocessing.
LOCAL_MARKDOWN sources have their Obsidian YAML frontmatter
stripped before extraction so the LLM does not extract
metadata-key claims from it.
When the decoded text exceeds the chunked-extraction threshold, the
HTML path routes through extract_with_carry_forward.
That helper consults the particle store for prior chunks whose hash
matches the current chunk's text, so session and
corpus_entry_id are read from kwargs (the pipeline already
passes them).
particles.extraction.wikidata.WikidataExtractor
¶
Extracts particles from a stored Wikibase REST API JSON blob.
particles.extraction.numista.NumistaCoinExtractor
¶
Extracts rich particles from a single Numista coin-type API response.
particles.extraction.numista.NumistaIssuerExtractor
¶
Extracts structured particles from a combined Numista issuer search response.
Produces one structured particle per coin using the summary fields available from the issuer search API (composition, weight, diameter, catalog refs). Fields only available from the individual coin endpoint (currency, demonetization, edge, obverse/reverse) are absent — deposit individual coin pages for full infoboxes.
particles.extraction.numista.NumistaListingExtractor
¶
Parses Numista catalogue listing HTML (NUMISTA_LISTING_HTML source type).
Extracts per-coin structured particles from