fix(vector): dead-letter terminally-failed documents to stop multi-user re-queue loop
A pathological PDF (a 206-page ChronoScan scan with ~3400 JBIG2/JPX images)
jammed a tenant's structured ingest worker in an infinite reprocess loop,
re-burning a 120s pymupdf4llm parse (and occasionally OOM-racing the 2Gi pod)
every few minutes.
Root cause: the per-user placeholder "failed" mark could not stop the loop. The
placeholder point ID is user-agnostic (uuid5("file:<doc_id>:placeholder")) but
the scanner's freshness gate, query, and status update all filter by user_id.
For a file visible to several users the single shared placeholder's user_id is
overwritten by whoever scanned last, so every other user's scan sees "no record"
and re-queues -- an N-user ping-pong that never honours the failed status.
Fix: when a parse fails terminally (no higher escalation tier available, e.g.
structured with OCR off) record a durable, content-addressed, user-agnostic
dead-letter marker (mirrors vector/sharing_state.py). The scanner consults it
tenant-wide for every user and skips re-queuing until the content (etag) OR the
escalation-tier set (tiers_sig -- e.g. OCR enabled) changes, so the document is
attempted once per content-version instead of forever.
- new vector/dead_letter.py: mark/is/clear, content-addressed marker carrying
is_placeholder=True (inherits search exclusion) + dead_letter=True
- escalation.escalation_tiers_signature(settings): retry-on-tier-change key
- processor: dead-letter terminal failures, clear on successful (re-)index
- scanner: user-agnostic is_dead_lettered skip beside claim_existing_index
- placeholder: exempt dead_letter markers from the orphan sweep (durability)
- metrics: astrolabe_document_dead_lettered_total{reason}
Deck #349.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
d969526613
commit
8c9339501e
@@ -305,6 +305,20 @@ document_parse_failed_total = Counter(
|
||||
["reason"], # reason: timeout | oom | error
|
||||
)
|
||||
|
||||
# Documents dead-lettered after a terminal parse failure: the failing tier had
|
||||
# no higher escalation tier available (e.g. structured timed out with OCR off),
|
||||
# so the document is recorded as permanently failed for this content-version and
|
||||
# stops being re-queued (vector/dead_letter.py). Distinct from
|
||||
# ``document_parse_failed_total`` (which counts every failed parse attempt,
|
||||
# including the ones that will be retried) -- this fires once when a document
|
||||
# is given up on, and clears implicitly when its etag or the escalation-tier set
|
||||
# changes and it is re-attempted.
|
||||
document_dead_lettered_total = Counter(
|
||||
"astrolabe_document_dead_lettered_total",
|
||||
"Documents dead-lettered after a terminal parse failure (no escalation tier)",
|
||||
["reason"], # reason: timeout | oom | error
|
||||
)
|
||||
|
||||
# Documents dropped after exhausting in-process indexing retries (the scanner
|
||||
# re-picks them on a later full scan, so this is "dropped for this cycle", not
|
||||
# "lost forever"). Labelled by classified cause so the embed-drop rate from a
|
||||
@@ -787,6 +801,16 @@ def record_document_parse_failed(reason: str) -> None:
|
||||
document_parse_failed_total.labels(reason=reason).inc()
|
||||
|
||||
|
||||
def record_document_dead_lettered(reason: str) -> None:
|
||||
"""Record a document dead-lettered after a terminal parse failure.
|
||||
|
||||
Args:
|
||||
reason: ``timeout`` | ``oom`` | ``error`` (the terminal parse failure
|
||||
reason carried from the isolated worker).
|
||||
"""
|
||||
document_dead_lettered_total.labels(reason=reason).inc()
|
||||
|
||||
|
||||
def record_ingest_dropped(reason: str) -> None:
|
||||
"""Record a document dropped after exhausting in-process indexing retries.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user