fix(vector): dead-letter terminally-failed documents to stop multi-user re-queue loop

A pathological PDF (a 206-page ChronoScan scan with ~3400 JBIG2/JPX images)
jammed a tenant's structured ingest worker in an infinite reprocess loop,
re-burning a 120s pymupdf4llm parse (and occasionally OOM-racing the 2Gi pod)
every few minutes.

Root cause: the per-user placeholder "failed" mark could not stop the loop. The
placeholder point ID is user-agnostic (uuid5("file:<doc_id>:placeholder")) but
the scanner's freshness gate, query, and status update all filter by user_id.
For a file visible to several users the single shared placeholder's user_id is
overwritten by whoever scanned last, so every other user's scan sees "no record"
and re-queues -- an N-user ping-pong that never honours the failed status.

Fix: when a parse fails terminally (no higher escalation tier available, e.g.
structured with OCR off) record a durable, content-addressed, user-agnostic
dead-letter marker (mirrors vector/sharing_state.py). The scanner consults it
tenant-wide for every user and skips re-queuing until the content (etag) OR the
escalation-tier set (tiers_sig -- e.g. OCR enabled) changes, so the document is
attempted once per content-version instead of forever.

- new vector/dead_letter.py: mark/is/clear, content-addressed marker carrying
  is_placeholder=True (inherits search exclusion) + dead_letter=True
- escalation.escalation_tiers_signature(settings): retry-on-tier-change key
- processor: dead-letter terminal failures, clear on successful (re-)index
- scanner: user-agnostic is_dead_lettered skip beside claim_existing_index
- placeholder: exempt dead_letter markers from the orphan sweep (durability)
- metrics: astrolabe_document_dead_lettered_total{reason}

Deck #349.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-17 19:09:49 +02:00
co-authored by Claude Opus 4.8
parent d969526613
commit 8c9339501e
10 changed files with 739 additions and 23 deletions
@@ -305,6 +305,20 @@ document_parse_failed_total = Counter(
["reason"], # reason: timeout | oom | error
)
# Documents dead-lettered after a terminal parse failure: the failing tier had
# no higher escalation tier available (e.g. structured timed out with OCR off),
# so the document is recorded as permanently failed for this content-version and
# stops being re-queued (vector/dead_letter.py). Distinct from
# ``document_parse_failed_total`` (which counts every failed parse attempt,
# including the ones that will be retried) -- this fires once when a document
# is given up on, and clears implicitly when its etag or the escalation-tier set
# changes and it is re-attempted.
document_dead_lettered_total = Counter(
"astrolabe_document_dead_lettered_total",
"Documents dead-lettered after a terminal parse failure (no escalation tier)",
["reason"], # reason: timeout | oom | error
)
# Documents dropped after exhausting in-process indexing retries (the scanner
# re-picks them on a later full scan, so this is "dropped for this cycle", not
# "lost forever"). Labelled by classified cause so the embed-drop rate from a
@@ -787,6 +801,16 @@ def record_document_parse_failed(reason: str) -> None:
document_parse_failed_total.labels(reason=reason).inc()
def record_document_dead_lettered(reason: str) -> None:
"""Record a document dead-lettered after a terminal parse failure.
Args:
reason: ``timeout`` | ``oom`` | ``error`` (the terminal parse failure
reason carried from the isolated worker).
"""
document_dead_lettered_total.labels(reason=reason).inc()
def record_ingest_dropped(reason: str) -> None:
"""Record a document dropped after exhausting in-process indexing retries.