fix(vector): dead-letter terminally-failed documents to stop multi-user re-queue loop

A pathological PDF (a 206-page ChronoScan scan with ~3400 JBIG2/JPX images)
jammed a tenant's structured ingest worker in an infinite reprocess loop,
re-burning a 120s pymupdf4llm parse (and occasionally OOM-racing the 2Gi pod)
every few minutes.

Root cause: the per-user placeholder "failed" mark could not stop the loop. The
placeholder point ID is user-agnostic (uuid5("file:<doc_id>:placeholder")) but
the scanner's freshness gate, query, and status update all filter by user_id.
For a file visible to several users the single shared placeholder's user_id is
overwritten by whoever scanned last, so every other user's scan sees "no record"
and re-queues -- an N-user ping-pong that never honours the failed status.

Fix: when a parse fails terminally (no higher escalation tier available, e.g.
structured with OCR off) record a durable, content-addressed, user-agnostic
dead-letter marker (mirrors vector/sharing_state.py). The scanner consults it
tenant-wide for every user and skips re-queuing until the content (etag) OR the
escalation-tier set (tiers_sig -- e.g. OCR enabled) changes, so the document is
attempted once per content-version instead of forever.

- new vector/dead_letter.py: mark/is/clear, content-addressed marker carrying
  is_placeholder=True (inherits search exclusion) + dead_letter=True
- escalation.escalation_tiers_signature(settings): retry-on-tier-change key
- processor: dead-letter terminal failures, clear on successful (re-)index
- scanner: user-agnostic is_dead_lettered skip beside claim_existing_index
- placeholder: exempt dead_letter markers from the orphan sweep (durability)
- metrics: astrolabe_document_dead_lettered_total{reason}

Deck #349.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-17 19:09:49 +02:00
co-authored by Claude Opus 4.8
parent d969526613
commit 8c9339501e
10 changed files with 739 additions and 23 deletions
@@ -397,6 +397,13 @@ async def sweep_orphan_placeholders(
orphan_ids = []
for point in points:
payload = point.payload or {}
# Dead-letter markers (vector/dead_letter.py) reuse is_placeholder=True
# for the search exclusion but are DURABLE terminal-state records, not
# in-flight placeholders -- they carry no/foreign instance_id and must
# survive a Pod restart, so never sweep them as orphans.
if payload.get("dead_letter") is True:
kept += 1
continue
point_instance = payload.get("instance_id")
if point_instance == _INSTANCE_ID:
kept += 1