fix(vector): dead-letter terminally-failed documents to stop multi-user re-queue loop

A pathological PDF (a 206-page ChronoScan scan with ~3400 JBIG2/JPX images)
jammed a tenant's structured ingest worker in an infinite reprocess loop,
re-burning a 120s pymupdf4llm parse (and occasionally OOM-racing the 2Gi pod)
every few minutes.

Root cause: the per-user placeholder "failed" mark could not stop the loop. The
placeholder point ID is user-agnostic (uuid5("file:<doc_id>:placeholder")) but
the scanner's freshness gate, query, and status update all filter by user_id.
For a file visible to several users the single shared placeholder's user_id is
overwritten by whoever scanned last, so every other user's scan sees "no record"
and re-queues -- an N-user ping-pong that never honours the failed status.

Fix: when a parse fails terminally (no higher escalation tier available, e.g.
structured with OCR off) record a durable, content-addressed, user-agnostic
dead-letter marker (mirrors vector/sharing_state.py). The scanner consults it
tenant-wide for every user and skips re-queuing until the content (etag) OR the
escalation-tier set (tiers_sig -- e.g. OCR enabled) changes, so the document is
attempted once per content-version instead of forever.

- new vector/dead_letter.py: mark/is/clear, content-addressed marker carrying
  is_placeholder=True (inherits search exclusion) + dead_letter=True
- escalation.escalation_tiers_signature(settings): retry-on-tier-change key
- processor: dead-letter terminal failures, clear on successful (re-)index
- scanner: user-agnostic is_dead_lettered skip beside claim_existing_index
- placeholder: exempt dead_letter markers from the orphan sweep (durability)
- metrics: astrolabe_document_dead_lettered_total{reason}

Deck #349.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-17 19:09:49 +02:00
co-authored by Claude Opus 4.8
parent d969526613
commit 8c9339501e
10 changed files with 739 additions and 23 deletions
+29
View File
@@ -27,6 +27,7 @@ from nextcloud_mcp_server.server.tag_exclusion import (
get_excluded_file_paths,
is_path_excluded,
)
from nextcloud_mcp_server.vector.dead_letter import is_dead_lettered
from nextcloud_mcp_server.vector.placeholder import (
query_document_metadata,
write_placeholder_point,
@@ -664,6 +665,16 @@ async def scan_user_documents(
skipped,
)
# Escalation-tier fingerprint for the dead-letter skip below, computed
# once per scan. Lazy import: document_processors.__init__ pulls the
# heavy parse stack (pymupdf/_isolation, Unix-only ``resource``; #877),
# which the scanner (API role) must not load at module import.
from nextcloud_mcp_server.document_processors.escalation import ( # noqa: PLC0415
escalation_tiers_signature,
)
tiers_sig = escalation_tiers_signature(get_settings())
for file_info in tagged_files:
# Files are already filtered by MIME type in find_files_by_tag()
file_count += 1
@@ -703,6 +714,24 @@ async def scan_user_documents(
)
continue
# Tenant-wide dead-letter skip: a document that terminally failed
# parsing (no escalation tier) is recorded user-agnostically, so
# EVERY user's scan skips re-queuing it until its content (etag) or
# the escalation-tier set (tiers_sig, e.g. OCR enabled) changes.
# Unlike the per-user placeholder "failed" mark this is not
# defeated by a file shared across users -- whose single
# user-agnostic placeholder's user_id is overwritten by the last
# scanner, so every other user re-queued it on a loop.
if etag and await is_dead_lettered(file_id, "file", etag, tiers_sig):
_potentially_deleted.pop((user_id, file_id), None)
logger.debug(
"Skipping dead-lettered file %s (ID: %s) until content/"
"tier change",
file_path,
file_id,
)
continue
if initial_sync:
# Send everything on first sync - write placeholder first
await write_placeholder_point(