fix(vector): dead-letter terminally-failed documents to stop multi-user re-queue loop
A pathological PDF (a 206-page ChronoScan scan with ~3400 JBIG2/JPX images)
jammed a tenant's structured ingest worker in an infinite reprocess loop,
re-burning a 120s pymupdf4llm parse (and occasionally OOM-racing the 2Gi pod)
every few minutes.
Root cause: the per-user placeholder "failed" mark could not stop the loop. The
placeholder point ID is user-agnostic (uuid5("file:<doc_id>:placeholder")) but
the scanner's freshness gate, query, and status update all filter by user_id.
For a file visible to several users the single shared placeholder's user_id is
overwritten by whoever scanned last, so every other user's scan sees "no record"
and re-queues -- an N-user ping-pong that never honours the failed status.
Fix: when a parse fails terminally (no higher escalation tier available, e.g.
structured with OCR off) record a durable, content-addressed, user-agnostic
dead-letter marker (mirrors vector/sharing_state.py). The scanner consults it
tenant-wide for every user and skips re-queuing until the content (etag) OR the
escalation-tier set (tiers_sig -- e.g. OCR enabled) changes, so the document is
attempted once per content-version instead of forever.
- new vector/dead_letter.py: mark/is/clear, content-addressed marker carrying
is_placeholder=True (inherits search exclusion) + dead_letter=True
- escalation.escalation_tiers_signature(settings): retry-on-tier-change key
- processor: dead-letter terminal failures, clear on successful (re-)index
- scanner: user-agnostic is_dead_lettered skip beside claim_existing_index
- placeholder: exempt dead_letter markers from the orphan sweep (durability)
- metrics: astrolabe_document_dead_lettered_total{reason}
Deck #349.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
d969526613
commit
8c9339501e
@@ -27,6 +27,7 @@ from nextcloud_mcp_server.server.tag_exclusion import (
|
||||
get_excluded_file_paths,
|
||||
is_path_excluded,
|
||||
)
|
||||
from nextcloud_mcp_server.vector.dead_letter import is_dead_lettered
|
||||
from nextcloud_mcp_server.vector.placeholder import (
|
||||
query_document_metadata,
|
||||
write_placeholder_point,
|
||||
@@ -664,6 +665,16 @@ async def scan_user_documents(
|
||||
skipped,
|
||||
)
|
||||
|
||||
# Escalation-tier fingerprint for the dead-letter skip below, computed
|
||||
# once per scan. Lazy import: document_processors.__init__ pulls the
|
||||
# heavy parse stack (pymupdf/_isolation, Unix-only ``resource``; #877),
|
||||
# which the scanner (API role) must not load at module import.
|
||||
from nextcloud_mcp_server.document_processors.escalation import ( # noqa: PLC0415
|
||||
escalation_tiers_signature,
|
||||
)
|
||||
|
||||
tiers_sig = escalation_tiers_signature(get_settings())
|
||||
|
||||
for file_info in tagged_files:
|
||||
# Files are already filtered by MIME type in find_files_by_tag()
|
||||
file_count += 1
|
||||
@@ -703,6 +714,24 @@ async def scan_user_documents(
|
||||
)
|
||||
continue
|
||||
|
||||
# Tenant-wide dead-letter skip: a document that terminally failed
|
||||
# parsing (no escalation tier) is recorded user-agnostically, so
|
||||
# EVERY user's scan skips re-queuing it until its content (etag) or
|
||||
# the escalation-tier set (tiers_sig, e.g. OCR enabled) changes.
|
||||
# Unlike the per-user placeholder "failed" mark this is not
|
||||
# defeated by a file shared across users -- whose single
|
||||
# user-agnostic placeholder's user_id is overwritten by the last
|
||||
# scanner, so every other user re-queued it on a loop.
|
||||
if etag and await is_dead_lettered(file_id, "file", etag, tiers_sig):
|
||||
_potentially_deleted.pop((user_id, file_id), None)
|
||||
logger.debug(
|
||||
"Skipping dead-lettered file %s (ID: %s) until content/"
|
||||
"tier change",
|
||||
file_path,
|
||||
file_id,
|
||||
)
|
||||
continue
|
||||
|
||||
if initial_sync:
|
||||
# Send everything on first sync - write placeholder first
|
||||
await write_placeholder_point(
|
||||
|
||||
Reference in New Issue
Block a user