admin/mcp-nextcloud
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(vector): dead-letter terminally-failed documents to stop multi-user re-queue loop

Browse Source 
A pathological PDF (a 206-page ChronoScan scan with ~3400 JBIG2/JPX images)
jammed a tenant's structured ingest worker in an infinite reprocess loop,
re-burning a 120s pymupdf4llm parse (and occasionally OOM-racing the 2Gi pod)
every few minutes.

Root cause: the per-user placeholder "failed" mark could not stop the loop. The
placeholder point ID is user-agnostic (uuid5("file:<doc_id>:placeholder")) but
the scanner's freshness gate, query, and status update all filter by user_id.
For a file visible to several users the single shared placeholder's user_id is
overwritten by whoever scanned last, so every other user's scan sees "no record"
and re-queues -- an N-user ping-pong that never honours the failed status.

Fix: when a parse fails terminally (no higher escalation tier available, e.g.
structured with OCR off) record a durable, content-addressed, user-agnostic
dead-letter marker (mirrors vector/sharing_state.py). The scanner consults it
tenant-wide for every user and skips re-queuing until the content (etag) OR the
escalation-tier set (tiers_sig -- e.g. OCR enabled) changes, so the document is
attempted once per content-version instead of forever.

- new vector/dead_letter.py: mark/is/clear, content-addressed marker carrying
  is_placeholder=True (inherits search exclusion) + dead_letter=True
- escalation.escalation_tiers_signature(settings): retry-on-tier-change key
- processor: dead-letter terminal failures, clear on successful (re-)index
- scanner: user-agnostic is_dead_lettered skip beside claim_existing_index
- placeholder: exempt dead_letter markers from the orphan sweep (durability)
- metrics: astrolabe_document_dead_lettered_total{reason}

Deck #349.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-17 19:09:49 +02:00
co-authored by Claude Opus 4.8
parent d969526613
commit 8c9339501e
10 changed files with 739 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
nextcloud_mcp_server/vector/processor.py
+92 -22
View File
@@ -28,6 +28,7 @@ from nextcloud_mcp_server.embedding import get_bm25_service, get_embedding_servi
from nextcloud_mcp_server.models.deck import DeckCard
from nextcloud_mcp_server.observability.metrics import (
record_document_chunks,
record_document_dead_lettered,
record_document_escalation,
record_document_escalation_suppressed,
record_document_parse_failed,
@@ -43,6 +44,10 @@ from nextcloud_mcp_server.search.pdf_highlighter import PDFHighlighter
from nextcloud_mcp_server.usage import UsageEventStore
from nextcloud_mcp_server.vector import payload_keys
from nextcloud_mcp_server.vector._errors import format_exception_group
from nextcloud_mcp_server.vector.dead_letter import (
clear_dead_letter,
mark_dead_letter,
)
from nextcloud_mcp_server.vector.document_chunker import (
DocumentChunker,
PageAwareChunker,
@@ -993,8 +998,10 @@ async def _index_document(
get_registry,
)
from nextcloud_mcp_server.document_processors.escalation import ( # noqa: PLC0415
TIER_LADDER,
BatchPending,
EscalateError,
escalation_tiers_signature,
)
registry = get_registry()
@@ -1035,34 +1042,89 @@ async def _index_document(
# A permanent parse failure (e.g. an isolated-worker OOM/timeout
# on a pathological PDF) returns success=False rather than
# raising -- there is nothing to index and retrying would just
# fail again. Mark the placeholder "failed" so the scanner stops
# re-queuing it (until the file changes) and return False so the
# caller skips the success metrics (it was not indexed).
# fail again.
if not result.success:
reason = result.metadata.get("parse_failed_reason", "error")
record_document_parse_failed(reason)
logger.warning(
"Permanent parse failure for %s (reason=%s); marking "
"failed and skipping index",
file_path,
reason,
# The tier that produced this failed result: the worker's own
# tier on the per-tier path, else the deepest tier the inline
# pipeline reached (recorded as ``pipeline_tier``).
failing_tier = tier or result.metadata.get(
"pipeline_tier", TIER_LADDER[0]
)
try:
await update_placeholder_status(
doc_id=doc_task.doc_id,
doc_type=doc_task.doc_type,
user_id=doc_task.user_id,
status="failed",
)
except Exception:
# Best-effort: a transient Qdrant error here only means
# the placeholder isn't marked, so the scanner retries
# the (still un-indexable) file later -- not fatal.
logger.debug(
"Could not mark placeholder failed for %s",
terminal = (
registry.next_available_tier(failing_tier, settings) is None
)
if terminal:
# No higher tier can run (e.g. structured timed out with
# OCR off), so retrying just re-burns the same failing
# parse. Dead-letter the document tenant-wide
# (content-addressed, user-agnostic) so EVERY user's scan
# stops re-queuing it until its content (etag) or the
# escalation-tier set (e.g. OCR enabled -> new tiers_sig)
# changes. This fixes the multi-user placeholder
# ping-pong the per-user "failed" mark could not: a file
# shared by N users has ONE user-agnostic placeholder
# whose user_id is overwritten by the last scanner, so
# every other user re-queued it forever.
await mark_dead_letter(
doc_task.doc_id,
exc_info=True,
doc_task.doc_type,
doc_task.etag or "",
escalation_tiers_signature(settings),
reason,
file_path=file_path,
)
record_document_dead_lettered(reason)
logger.warning(
"Permanent parse failure for %s (reason=%s); "
"dead-lettered (terminal tier=%s, no escalation) and "
"skipping index",
file_path,
reason,
failing_tier,
)
# Drop the volatile in-flight placeholder; the durable
# marker is now the document's terminal-state record.
try:
await delete_placeholder_point(
doc_id=doc_task.doc_id,
doc_type=doc_task.doc_type,
user_id=doc_task.user_id,
)
except Exception:
logger.debug(
"Could not delete placeholder for dead-lettered %s",
doc_task.doc_id,
exc_info=True,
)
else:
# A higher tier exists; parse failures don't escalate to
# it today, so keep the legacy per-user "failed"
# placeholder mark (not dead-lettered -- a future change
# may route the failure to that tier).
logger.warning(
"Permanent parse failure for %s (reason=%s); marking "
"failed and skipping index",
file_path,
reason,
)
try:
await update_placeholder_status(
doc_id=doc_task.doc_id,
doc_type=doc_task.doc_type,
user_id=doc_task.user_id,
status="failed",
)
except Exception:
# Best-effort: a transient Qdrant error here only
# means the placeholder isn't marked, so the scanner
# retries the (still un-indexable) file later.
logger.debug(
"Could not mark placeholder failed for %s",
doc_task.doc_id,
exc_info=True,
)
return False
content = result.text
@@ -1529,6 +1591,14 @@ async def _index_document(
)
)
# A successful (re-)index supersedes any prior terminal failure: clear a
# stale dead-letter marker (e.g. the file was fixed/replaced, or a new
# escalation tier finally parsed it) so it isn't left behind. Only files are
# ever dead-lettered (the mark lives in the file branch), so skip the extra
# Qdrant round-trip for the other doc types on the hot indexing path.
if doc_task.doc_type == "file":
await clear_dead_letter(doc_task.doc_id, doc_task.doc_type)
# Delete placeholder before writing real vectors
# This prevents duplicates and cleans up the placeholder state
try: