fix: isolate PDF parse in a subprocess so a bad file can't OOM the pod

The document processor crash-looped on one pathological PDF: pymupdf4llm's
table/graphics detection over a page with ~1M vector path items ballooned past
the 2 GiB pod limit. The parse ran in a thread, so nothing could interrupt or
memory-bound it -- a single bad file OOM-killed the whole pod.

Run the parse in an isolated worker subprocess (anyio.to_process, cancellable)
with an RLIMIT_AS memory cap and a wall-clock timeout, so a pathological file
fails THAT document instead of the pod (new document_processors/_isolation.py).
Also pass graphics_limit (default 5000) to to_markdown -- validated to cut the
known trigger page from 112 s to 23 s with bounded memory.

On a permanent parse failure the processor returns success=False (instead of
raising, which would retry 3x); vector/processor.py marks the placeholder
"failed" and skips indexing, and the scanner stops re-queuing failed placeholders
until the file changes -- so a doomed file no longer churns.

New per-tenant (per-pod env) settings: DOCUMENT_PDF_GRAPHICS_LIMIT,
DOCUMENT_PARSE_TIMEOUT_SECONDS, DOCUMENT_PARSE_MEM_LIMIT_MB. New metric
astrolabe_document_parse_failed_total{reason=timeout|oom|error} surfaces hard
failures that previously killed the process before any except ran.

First PR of the tiered document-processor effort (Deck #199); tier 0/1/3
pipeline tracked separately.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-04 22:32:34 +02:00
co-authored by Claude Opus 4.8
parent 09e84783e5
commit 7db8d3e301
7 changed files with 450 additions and 19 deletions
+39 -1
View File
@@ -21,6 +21,7 @@ from nextcloud_mcp_server.embedding import get_bm25_service, get_embedding_servi
from nextcloud_mcp_server.models.deck import DeckCard
from nextcloud_mcp_server.observability.metrics import (
record_document_chunks,
record_document_parse_failed,
record_embedding,
record_qdrant_operation,
record_vector_sync_processing,
@@ -31,7 +32,10 @@ from nextcloud_mcp_server.search.pdf_highlighter import PDFHighlighter
from nextcloud_mcp_server.vector import payload_keys
from nextcloud_mcp_server.vector.document_chunker import DocumentChunker
from nextcloud_mcp_server.vector.html_processor import html_to_markdown
from nextcloud_mcp_server.vector.placeholder import delete_placeholder_point
from nextcloud_mcp_server.vector.placeholder import (
delete_placeholder_point,
update_placeholder_status,
)
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
from nextcloud_mcp_server.vector.scanner import DocumentTask
from nextcloud_mcp_server.vector.sharing_state import (
@@ -525,6 +529,40 @@ async def _index_document(
content_type=content_type,
filename=file_path,
)
# A permanent parse failure (e.g. an isolated-worker OOM/timeout
# on a pathological PDF) returns success=False rather than
# raising -- there is nothing to index and retrying would just
# fail again. Mark the placeholder "failed" so the scanner stops
# re-queuing it (until the file changes) and return without
# indexing empty content.
if not result.success:
reason = result.metadata.get("parse_failed_reason", "error")
record_document_parse_failed(reason)
logger.warning(
"Permanent parse failure for %s (reason=%s); marking "
"failed and skipping index",
file_path,
reason,
)
try:
await update_placeholder_status(
doc_id=doc_task.doc_id,
doc_type=doc_task.doc_type,
user_id=doc_task.user_id,
status="failed",
)
except Exception:
# Best-effort: a transient Qdrant error here only means
# the placeholder isn't marked, so the scanner retries
# the (still un-indexable) file later -- not fatal.
logger.debug(
"Could not mark placeholder failed for %s",
doc_task.doc_id,
exc_info=True,
)
return
content = result.text
file_metadata = result.metadata
title = file_metadata.get("title") or file_path.split("/")[-1]