Addresses round-2 review on PR #920:
- Only dead-letter a terminal failure when the file has an etag to
content-address the marker; without one, fall back to the legacy per-user
placeholder mark (an etagless marker is unmatchable). + test.
- _dead_letter_filter now also matches is_placeholder=True (redundant with
dead_letter=True but lets Qdrant use the is_placeholder payload index).
- TODO(deck-349) documenting the dead-lettered-then-deleted orphan-marker leak
(out of scope; needs a marker sweep or TTL field) per reviewer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses round-1 review on PR #920:
- Delete path now clears the file's dead-letter marker after
release_document_for_user (whose principal-based filter misses the
user-agnostic, principal-less marker), preventing orphan-marker accumulation
for dead-lettered-then-deleted files.
- Oversize PDFs (rejected by the pre-parse size guard, no pipeline_tier stamped)
are now treated as terminal regardless of failing_tier -- no tier can parse an
oversize file -- so they dead-letter instead of falling to the legacy per-user
mark on the inline path.
- Gate the success-path clear on a non-empty etag (an etag-less file can never
have a marker, mirroring is_dead_lettered's early return).
- dead_letter.py: payload typed dict[str, Any] (CLAUDE.md).
Tests: oversize-terminal dead-letter and delete-path marker clear.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A pathological PDF (a 206-page ChronoScan scan with ~3400 JBIG2/JPX images)
jammed a tenant's structured ingest worker in an infinite reprocess loop,
re-burning a 120s pymupdf4llm parse (and occasionally OOM-racing the 2Gi pod)
every few minutes.
Root cause: the per-user placeholder "failed" mark could not stop the loop. The
placeholder point ID is user-agnostic (uuid5("file:<doc_id>:placeholder")) but
the scanner's freshness gate, query, and status update all filter by user_id.
For a file visible to several users the single shared placeholder's user_id is
overwritten by whoever scanned last, so every other user's scan sees "no record"
and re-queues -- an N-user ping-pong that never honours the failed status.
Fix: when a parse fails terminally (no higher escalation tier available, e.g.
structured with OCR off) record a durable, content-addressed, user-agnostic
dead-letter marker (mirrors vector/sharing_state.py). The scanner consults it
tenant-wide for every user and skips re-queuing until the content (etag) OR the
escalation-tier set (tiers_sig -- e.g. OCR enabled) changes, so the document is
attempted once per content-version instead of forever.
- new vector/dead_letter.py: mark/is/clear, content-addressed marker carrying
is_placeholder=True (inherits search exclusion) + dead_letter=True
- escalation.escalation_tiers_signature(settings): retry-on-tier-change key
- processor: dead-letter terminal failures, clear on successful (re-)index
- scanner: user-agnostic is_dead_lettered skip beside claim_existing_index
- placeholder: exempt dead_letter markers from the orphan sweep (durability)
- metrics: astrolabe_document_dead_lettered_total{reason}
Deck #349.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
