fix(vector): dead-letter terminally-failed documents to stop multi-user re-queue loop
A pathological PDF (a 206-page ChronoScan scan with ~3400 JBIG2/JPX images)
jammed a tenant's structured ingest worker in an infinite reprocess loop,
re-burning a 120s pymupdf4llm parse (and occasionally OOM-racing the 2Gi pod)
every few minutes.
Root cause: the per-user placeholder "failed" mark could not stop the loop. The
placeholder point ID is user-agnostic (uuid5("file:<doc_id>:placeholder")) but
the scanner's freshness gate, query, and status update all filter by user_id.
For a file visible to several users the single shared placeholder's user_id is
overwritten by whoever scanned last, so every other user's scan sees "no record"
and re-queues -- an N-user ping-pong that never honours the failed status.
Fix: when a parse fails terminally (no higher escalation tier available, e.g.
structured with OCR off) record a durable, content-addressed, user-agnostic
dead-letter marker (mirrors vector/sharing_state.py). The scanner consults it
tenant-wide for every user and skips re-queuing until the content (etag) OR the
escalation-tier set (tiers_sig -- e.g. OCR enabled) changes, so the document is
attempted once per content-version instead of forever.
- new vector/dead_letter.py: mark/is/clear, content-addressed marker carrying
is_placeholder=True (inherits search exclusion) + dead_letter=True
- escalation.escalation_tiers_signature(settings): retry-on-tier-change key
- processor: dead-letter terminal failures, clear on successful (re-)index
- scanner: user-agnostic is_dead_lettered skip beside claim_existing_index
- placeholder: exempt dead_letter markers from the orphan sweep (durability)
- metrics: astrolabe_document_dead_lettered_total{reason}
Deck #349.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
d969526613
commit
8c9339501e
@@ -22,13 +22,36 @@ mapping lives in the queue layer, which imports :class:`EscalateError` from here
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal
|
||||
from typing import Any, Literal
|
||||
|
||||
# Cheapest-first. ``llm`` is reserved (see base.DocumentProcessor.tier) and not
|
||||
# wired yet, so it is intentionally absent from the live ladder.
|
||||
TIER_LADDER: tuple[str, ...] = ("fast", "structured", "ocr")
|
||||
|
||||
|
||||
def escalation_tiers_signature(settings: Any) -> str:
|
||||
"""A stable string fingerprint of the runtime escalation-tier configuration.
|
||||
|
||||
Used by the document dead-letter marker (``vector/dead_letter.py``) as part of
|
||||
its content key: a document that fails its terminal tier is dead-lettered until
|
||||
either its content (etag) OR this signature changes. The signature therefore
|
||||
captures every setting that can make a *new* escalation tier become available
|
||||
at runtime — flip it and previously dead-lettered documents become retryable.
|
||||
|
||||
Derived purely from settings (not the live ``ProcessorRegistry``) so it is
|
||||
identical across the API/scanner and worker roles: the *registered* processor
|
||||
set is build-constant, so the only runtime variables are the OCR-enabled gate
|
||||
(``document_ocr_enabled`` — the single tier toggle today) and the tier-1 engine
|
||||
pin (``document_tier1_engine``). Enabling OCR changes the signature, so the
|
||||
pathological-but-OCR-recoverable documents dead-lettered while OCR was off are
|
||||
re-attempted automatically.
|
||||
"""
|
||||
return (
|
||||
f"ocr={int(bool(settings.document_ocr_enabled))};"
|
||||
f"t1={settings.document_tier1_engine}"
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EscalationDecision:
|
||||
"""Outcome of the post-parse quality gate (``ProcessorRegistry.evaluate_escalation``).
|
||||
|
||||
Reference in New Issue
Block a user