fix: isolate PDF parse in a subprocess so a bad file can't OOM the pod

The document processor crash-looped on one pathological PDF: pymupdf4llm's
table/graphics detection over a page with ~1M vector path items ballooned past
the 2 GiB pod limit. The parse ran in a thread, so nothing could interrupt or
memory-bound it -- a single bad file OOM-killed the whole pod.

Run the parse in an isolated worker subprocess (anyio.to_process, cancellable)
with an RLIMIT_AS memory cap and a wall-clock timeout, so a pathological file
fails THAT document instead of the pod (new document_processors/_isolation.py).
Also pass graphics_limit (default 5000) to to_markdown -- validated to cut the
known trigger page from 112 s to 23 s with bounded memory.

On a permanent parse failure the processor returns success=False (instead of
raising, which would retry 3x); vector/processor.py marks the placeholder
"failed" and skips indexing, and the scanner stops re-queuing failed placeholders
until the file changes -- so a doomed file no longer churns.

New per-tenant (per-pod env) settings: DOCUMENT_PDF_GRAPHICS_LIMIT,
DOCUMENT_PARSE_TIMEOUT_SECONDS, DOCUMENT_PARSE_MEM_LIMIT_MB. New metric
astrolabe_document_parse_failed_total{reason=timeout|oom|error} surfaces hard
failures that previously killed the process before any except ran.

First PR of the tiered document-processor effort (Deck #199); tier 0/1/3
pipeline tracked separately.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-04 22:32:34 +02:00
co-authored by Claude Opus 4.8
parent 09e84783e5
commit 7db8d3e301
7 changed files with 450 additions and 19 deletions
+15 -4
View File
@@ -526,13 +526,24 @@ async def scan_user_documents(
# File modified since last indexing
needs_indexing = True
elif existing_metadata.get("is_placeholder", False):
# Placeholder exists - check if it's stale (processing may have failed)
# Only requeue if placeholder is older than 5x scan interval
# (Large PDFs can take 3-4 minutes to process)
# Placeholder exists - check its status / staleness.
queued_at = existing_metadata.get("queued_at", 0)
placeholder_age = time.time() - queued_at
stale_threshold = get_settings().vector_sync_scan_interval * 5
if placeholder_age > stale_threshold:
if existing_metadata.get("status") == "failed":
# A permanent parse failure (e.g. an isolated-worker
# OOM/timeout on a pathological PDF). Don't keep
# re-queuing an unchanged file that will just fail
# again -- the modified_at branch above still retries
# it once the file actually changes.
logger.debug(
"Skipping file %s (ID: %s): previous parse failed permanently",
file_path,
file_id,
)
elif placeholder_age > stale_threshold:
# Only requeue if placeholder is older than 5x scan
# interval (large PDFs can take minutes to process).
logger.debug(
"Found stale placeholder for file %s (ID: %s) (age=%ss), requeuing",
file_path,