admin/mcp-nextcloud
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(vector): dead-letter terminally-failed documents to stop multi-user re-queue loop

Browse Source 
A pathological PDF (a 206-page ChronoScan scan with ~3400 JBIG2/JPX images)
jammed a tenant's structured ingest worker in an infinite reprocess loop,
re-burning a 120s pymupdf4llm parse (and occasionally OOM-racing the 2Gi pod)
every few minutes.

Root cause: the per-user placeholder "failed" mark could not stop the loop. The
placeholder point ID is user-agnostic (uuid5("file:<doc_id>:placeholder")) but
the scanner's freshness gate, query, and status update all filter by user_id.
For a file visible to several users the single shared placeholder's user_id is
overwritten by whoever scanned last, so every other user's scan sees "no record"
and re-queues -- an N-user ping-pong that never honours the failed status.

Fix: when a parse fails terminally (no higher escalation tier available, e.g.
structured with OCR off) record a durable, content-addressed, user-agnostic
dead-letter marker (mirrors vector/sharing_state.py). The scanner consults it
tenant-wide for every user and skips re-queuing until the content (etag) OR the
escalation-tier set (tiers_sig -- e.g. OCR enabled) changes, so the document is
attempted once per content-version instead of forever.

- new vector/dead_letter.py: mark/is/clear, content-addressed marker carrying
  is_placeholder=True (inherits search exclusion) + dead_letter=True
- escalation.escalation_tiers_signature(settings): retry-on-tier-change key
- processor: dead-letter terminal failures, clear on successful (re-)index
- scanner: user-agnostic is_dead_lettered skip beside claim_existing_index
- placeholder: exempt dead_letter markers from the orphan sweep (durability)
- metrics: astrolabe_document_dead_lettered_total{reason}

Deck #349.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-17 19:09:49 +02:00
co-authored by Claude Opus 4.8
parent d969526613
commit 8c9339501e
10 changed files with 739 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
nextcloud_mcp_server/vector/dead_letter.py
+190
View File
@@ -0,0 +1,190 @@
"""Content-addressed dead-letter markers for terminally-failed documents.
A document that fails its terminal extraction tier (a hard parse failure with no
higher tier to escalate to — e.g. the ``structured`` tier timing out while OCR is
disabled) must not be retried forever. The per-user placeholder ``status="failed"``
mark cannot stop the loop on its own: placeholder point IDs are user-agnostic
(``uuid5("file:<doc_id>:placeholder")``) but the scanner's freshness gate filters
by ``user_id``, so for a file visible to *several* users the single shared
placeholder's ``user_id`` is overwritten by whoever scanned last and every other
user's scan sees "no record → re-queue", re-burning the (failing) parse on a loop.
This module records a **durable, content-addressed, user-agnostic** dead-letter
marker instead. One marker point per document (a distinct deterministic ID, kept
separate from the in-flight placeholder), carrying the ``etag`` and an escalation
``tiers_sig`` (see ``document_processors.escalation.escalation_tiers_signature``).
The scanner consults it tenant-wide — for every user — and skips re-queuing while
BOTH still match, so the document is attempted once per content-version and never
loops. A content change (new ``etag``) or a config change that adds an escalation
tier (e.g. enabling OCR — new ``tiers_sig``) makes the marker stale and the
document retryable again.
The marker carries ``is_placeholder=True`` so the existing search exclusion
(``get_placeholder_filter``) keeps it out of user-facing results with no extra
filter, plus ``dead_letter=True`` so the orphan-placeholder sweep and the scanner
can tell it apart from a volatile in-flight placeholder. Mirrors the fail-safe
philosophy of ``sharing_state``: a Qdrant error never aborts ingest — a failed
lookup degrades to "process normally", a failed write is logged, not raised.
"""
import logging
import time
import uuid
from qdrant_client import models
from qdrant_client.models import FieldCondition, Filter, MatchValue, PointStruct
from nextcloud_mcp_server.config import get_settings
from nextcloud_mcp_server.embedding import get_embedding_service
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
logger = logging.getLogger(__name__)
# Payload flag distinguishing a durable dead-letter marker from an in-flight
# placeholder (both carry is_placeholder=True to inherit the search exclusion).
DEAD_LETTER_KEY = "dead_letter"
def _generate_dead_letter_id(doc_type: str, doc_id: str) -> str:
"""Deterministic, user-agnostic point ID for a document's dead-letter marker.
Distinct from the in-flight placeholder ID (``…:placeholder``) so the two can
coexist briefly and never collide; one marker per ``(doc_type, doc_id)``, so
a re-failure upserts in place rather than accumulating.
"""
return str(uuid.uuid5(uuid.NAMESPACE_DNS, f"{doc_type}:{doc_id}:deadletter"))
def _dead_letter_filter(doc_id: str, doc_type: str) -> Filter:
"""Match the dead-letter marker for one document (tenant-wide, no user_id)."""
return Filter(
must=[
FieldCondition(key="doc_id", match=MatchValue(value=doc_id)),
FieldCondition(key="doc_type", match=MatchValue(value=doc_type)),
FieldCondition(key=DEAD_LETTER_KEY, match=MatchValue(value=True)),
]
)
async def mark_dead_letter(
doc_id: str,
doc_type: str,
etag: str,
tiers_sig: str,
reason: str,
*,
file_path: str | None = None,
) -> None:
"""Upsert a durable dead-letter marker for a terminally-failed document.
Keyed by content (``etag``) + escalation config (``tiers_sig``); the scanner
skips re-queuing while both match. ``reason`` is the parse failure reason
(``timeout`` | ``oom`` | ``error``). Fail-safe: a Qdrant error is logged, not
raised — a missed mark just means the document is retried (the bounded prior
behaviour), never a crash.
"""
try:
qdrant_client = await get_qdrant_client()
settings = get_settings()
dimension = get_embedding_service().get_dimension()
payload: dict = {
"doc_id": doc_id,
"doc_type": doc_type,
"is_placeholder": True,
DEAD_LETTER_KEY: True,
"etag": etag,
"tiers_sig": tiers_sig,
"reason": reason,
"failed_at": int(time.time()),
}
if doc_type == "file" and file_path:
payload["file_path"] = file_path
point = PointStruct(
id=_generate_dead_letter_id(doc_type, doc_id),
vector={
"dense": [0.0] * dimension,
"sparse": models.SparseVector(indices=[], values=[]),
},
payload=payload,
)
await qdrant_client.upsert(
collection_name=settings.get_collection_name(),
points=[point],
wait=True,
)
logger.info(
"Dead-lettered %s_%s (reason=%s, etag=%s)",
doc_type,
doc_id,
reason,
etag,
)
except Exception as e:
logger.warning(
"Failed to write dead-letter marker for %s_%s: %s", doc_type, doc_id, e
)
# Don't raise — dead-lettering is best-effort; a miss just retries.
async def is_dead_lettered(
doc_id: str,
doc_type: str,
etag: str,
tiers_sig: str,
) -> bool:
"""Whether this exact content is currently dead-lettered (skip re-queuing).
Returns True only when a marker exists for ``(doc_id, doc_type)`` whose stored
``etag`` AND ``tiers_sig`` both match the current values — so a content change
or a new escalation tier (e.g. OCR enabled) makes the document retryable
again. An empty ``etag`` is never dead-lettered (we cannot content-address it).
Fail-safe: a Qdrant error degrades to False (process normally), mirroring
``sharing_state.claim_existing_index``.
"""
if not etag:
return False
try:
qdrant_client = await get_qdrant_client()
settings = get_settings()
points, _ = await qdrant_client.scroll(
collection_name=settings.get_collection_name(),
scroll_filter=_dead_letter_filter(doc_id, doc_type),
limit=1,
with_payload=True,
with_vectors=False,
)
except Exception as e:
logger.warning(
"Dead-letter lookup failed for %s_%s (%s); processing normally",
doc_type,
doc_id,
e,
)
return False
if not points:
return False
payload = dict(points[0].payload or {})
return payload.get("etag") == etag and payload.get("tiers_sig") == tiers_sig
async def clear_dead_letter(doc_id: str, doc_type: str) -> None:
"""Delete a document's dead-letter marker (on successful index / release).
Idempotent and fail-safe: deleting a non-existent marker is a Qdrant no-op,
and an error is logged rather than raised so it never breaks the indexing
path that calls it.
"""
try:
qdrant_client = await get_qdrant_client()
settings = get_settings()
await qdrant_client.delete(
collection_name=settings.get_collection_name(),
points_selector=_dead_letter_filter(doc_id, doc_type),
)
logger.debug("Cleared dead-letter marker for %s_%s", doc_type, doc_id)
except Exception as e:
logger.warning(
"Failed to clear dead-letter marker for %s_%s: %s", doc_type, doc_id, e
)
nextcloud_mcp_server/vector/placeholder.py
+7
View File
@@ -397,6 +397,13 @@ async def sweep_orphan_placeholders(
orphan_ids = []
for point in points:
payload = point.payload or {}
# Dead-letter markers (vector/dead_letter.py) reuse is_placeholder=True
# for the search exclusion but are DURABLE terminal-state records, not
# in-flight placeholders -- they carry no/foreign instance_id and must
# survive a Pod restart, so never sweep them as orphans.
if payload.get("dead_letter") is True:
kept += 1
continue
point_instance = payload.get("instance_id")
if point_instance == _INSTANCE_ID:
kept += 1
nextcloud_mcp_server/vector/processor.py
+92 -22
View File
@@ -28,6 +28,7 @@ from nextcloud_mcp_server.embedding import get_bm25_service, get_embedding_servi
from nextcloud_mcp_server.models.deck import DeckCard
from nextcloud_mcp_server.observability.metrics import (
record_document_chunks,
record_document_dead_lettered,
record_document_escalation,
record_document_escalation_suppressed,
record_document_parse_failed,
@@ -43,6 +44,10 @@ from nextcloud_mcp_server.search.pdf_highlighter import PDFHighlighter
from nextcloud_mcp_server.usage import UsageEventStore
from nextcloud_mcp_server.vector import payload_keys
from nextcloud_mcp_server.vector._errors import format_exception_group
from nextcloud_mcp_server.vector.dead_letter import (
clear_dead_letter,
mark_dead_letter,
)
from nextcloud_mcp_server.vector.document_chunker import (
DocumentChunker,
PageAwareChunker,
@@ -993,8 +998,10 @@ async def _index_document(
get_registry,
)
from nextcloud_mcp_server.document_processors.escalation import ( # noqa: PLC0415
TIER_LADDER,
BatchPending,
EscalateError,
escalation_tiers_signature,
)
registry = get_registry()
@@ -1035,34 +1042,89 @@ async def _index_document(
# A permanent parse failure (e.g. an isolated-worker OOM/timeout
# on a pathological PDF) returns success=False rather than
# raising -- there is nothing to index and retrying would just
# fail again. Mark the placeholder "failed" so the scanner stops
# re-queuing it (until the file changes) and return False so the
# caller skips the success metrics (it was not indexed).
# fail again.
if not result.success:
reason = result.metadata.get("parse_failed_reason", "error")
record_document_parse_failed(reason)
logger.warning(
"Permanent parse failure for %s (reason=%s); marking "
"failed and skipping index",
file_path,
reason,
# The tier that produced this failed result: the worker's own
# tier on the per-tier path, else the deepest tier the inline
# pipeline reached (recorded as ``pipeline_tier``).
failing_tier = tier or result.metadata.get(
"pipeline_tier", TIER_LADDER[0]
)
try:
await update_placeholder_status(
doc_id=doc_task.doc_id,
doc_type=doc_task.doc_type,
user_id=doc_task.user_id,
status="failed",
)
except Exception:
# Best-effort: a transient Qdrant error here only means
# the placeholder isn't marked, so the scanner retries
# the (still un-indexable) file later -- not fatal.
logger.debug(
"Could not mark placeholder failed for %s",
terminal = (
registry.next_available_tier(failing_tier, settings) is None
)
if terminal:
# No higher tier can run (e.g. structured timed out with
# OCR off), so retrying just re-burns the same failing
# parse. Dead-letter the document tenant-wide
# (content-addressed, user-agnostic) so EVERY user's scan
# stops re-queuing it until its content (etag) or the
# escalation-tier set (e.g. OCR enabled -> new tiers_sig)
# changes. This fixes the multi-user placeholder
# ping-pong the per-user "failed" mark could not: a file
# shared by N users has ONE user-agnostic placeholder
# whose user_id is overwritten by the last scanner, so
# every other user re-queued it forever.
await mark_dead_letter(
doc_task.doc_id,
exc_info=True,
doc_task.doc_type,
doc_task.etag or "",
escalation_tiers_signature(settings),
reason,
file_path=file_path,
)
record_document_dead_lettered(reason)
logger.warning(
"Permanent parse failure for %s (reason=%s); "
"dead-lettered (terminal tier=%s, no escalation) and "
"skipping index",
file_path,
reason,
failing_tier,
)
# Drop the volatile in-flight placeholder; the durable
# marker is now the document's terminal-state record.
try:
await delete_placeholder_point(
doc_id=doc_task.doc_id,
doc_type=doc_task.doc_type,
user_id=doc_task.user_id,
)
except Exception:
logger.debug(
"Could not delete placeholder for dead-lettered %s",
doc_task.doc_id,
exc_info=True,
)
else:
# A higher tier exists; parse failures don't escalate to
# it today, so keep the legacy per-user "failed"
# placeholder mark (not dead-lettered -- a future change
# may route the failure to that tier).
logger.warning(
"Permanent parse failure for %s (reason=%s); marking "
"failed and skipping index",
file_path,
reason,
)
try:
await update_placeholder_status(
doc_id=doc_task.doc_id,
doc_type=doc_task.doc_type,
user_id=doc_task.user_id,
status="failed",
)
except Exception:
# Best-effort: a transient Qdrant error here only
# means the placeholder isn't marked, so the scanner
# retries the (still un-indexable) file later.
logger.debug(
"Could not mark placeholder failed for %s",
doc_task.doc_id,
exc_info=True,
)
return False
content = result.text
@@ -1529,6 +1591,14 @@ async def _index_document(
)
)
# A successful (re-)index supersedes any prior terminal failure: clear a
# stale dead-letter marker (e.g. the file was fixed/replaced, or a new
# escalation tier finally parsed it) so it isn't left behind. Only files are
# ever dead-lettered (the mark lives in the file branch), so skip the extra
# Qdrant round-trip for the other doc types on the hot indexing path.
if doc_task.doc_type == "file":
await clear_dead_letter(doc_task.doc_id, doc_task.doc_type)
# Delete placeholder before writing real vectors
# This prevents duplicates and cleans up the placeholder state
try:
nextcloud_mcp_server/vector/scanner.py
+29
View File
@@ -27,6 +27,7 @@ from nextcloud_mcp_server.server.tag_exclusion import (
get_excluded_file_paths,
is_path_excluded,
)
from nextcloud_mcp_server.vector.dead_letter import is_dead_lettered
from nextcloud_mcp_server.vector.placeholder import (
query_document_metadata,
write_placeholder_point,
@@ -664,6 +665,16 @@ async def scan_user_documents(
skipped,
)
# Escalation-tier fingerprint for the dead-letter skip below, computed
# once per scan. Lazy import: document_processors.__init__ pulls the
# heavy parse stack (pymupdf/_isolation, Unix-only ``resource``; #877),
# which the scanner (API role) must not load at module import.
from nextcloud_mcp_server.document_processors.escalation import ( # noqa: PLC0415
escalation_tiers_signature,
)
tiers_sig = escalation_tiers_signature(get_settings())
for file_info in tagged_files:
# Files are already filtered by MIME type in find_files_by_tag()
file_count += 1
@@ -703,6 +714,24 @@ async def scan_user_documents(
)
continue
# Tenant-wide dead-letter skip: a document that terminally failed
# parsing (no escalation tier) is recorded user-agnostically, so
# EVERY user's scan skips re-queuing it until its content (etag) or
# the escalation-tier set (tiers_sig, e.g. OCR enabled) changes.
# Unlike the per-user placeholder "failed" mark this is not
# defeated by a file shared across users -- whose single
# user-agnostic placeholder's user_id is overwritten by the last
# scanner, so every other user re-queued it on a loop.
if etag and await is_dead_lettered(file_id, "file", etag, tiers_sig):
_potentially_deleted.pop((user_id, file_id), None)
logger.debug(
"Skipping dead-lettered file %s (ID: %s) until content/"
"tier change",
file_path,
file_id,
)
continue
if initial_sync:
# Send everything on first sync - write placeholder first
await write_placeholder_point(