fix: isolate PDF parse in a subprocess so a bad file can't OOM the pod

The document processor crash-looped on one pathological PDF: pymupdf4llm's
table/graphics detection over a page with ~1M vector path items ballooned past
the 2 GiB pod limit. The parse ran in a thread, so nothing could interrupt or
memory-bound it -- a single bad file OOM-killed the whole pod.

Run the parse in an isolated worker subprocess (anyio.to_process, cancellable)
with an RLIMIT_AS memory cap and a wall-clock timeout, so a pathological file
fails THAT document instead of the pod (new document_processors/_isolation.py).
Also pass graphics_limit (default 5000) to to_markdown -- validated to cut the
known trigger page from 112 s to 23 s with bounded memory.

On a permanent parse failure the processor returns success=False (instead of
raising, which would retry 3x); vector/processor.py marks the placeholder
"failed" and skips indexing, and the scanner stops re-queuing failed placeholders
until the file changes -- so a doomed file no longer churns.

New per-tenant (per-pod env) settings: DOCUMENT_PDF_GRAPHICS_LIMIT,
DOCUMENT_PARSE_TIMEOUT_SECONDS, DOCUMENT_PARSE_MEM_LIMIT_MB. New metric
astrolabe_document_parse_failed_total{reason=timeout|oom|error} surfaces hard
failures that previously killed the process before any except ran.

First PR of the tiered document-processor effort (Deck #199); tier 0/1/3
pipeline tracked separately.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-04 22:32:34 +02:00
co-authored by Claude Opus 4.8
parent 09e84783e5
commit 7db8d3e301
7 changed files with 450 additions and 19 deletions
@@ -261,6 +261,17 @@ document_escalation_total = Counter(
["from_tier", "to_tier", "reason"],
)
# Hard parse failures: the parse now runs in an isolated subprocess, so a
# timeout/OOM that kills the worker is caught here. This is distinct from
# ``document_parse_total{status="error"}`` (an in-process exception): a hard
# OOM previously killed the pod before any except ran, so it incremented
# nothing -- this counter makes those failures visible.
document_parse_failed_total = Counter(
"astrolabe_document_parse_failed_total",
"Document parses that failed in the isolated worker (process killed)",
["reason"], # reason: timeout | oom | error
)
# --- Embedding stages ---------------------------------------------------------
embedding_duration_seconds = Histogram(
@@ -617,6 +628,15 @@ def record_document_escalation(from_tier: str, to_tier: str, reason: str) -> Non
).inc()
def record_document_parse_failed(reason: str) -> None:
"""Record a hard parse failure from the isolated worker.
Args:
reason: ``timeout`` | ``oom`` | ``error``
"""
document_parse_failed_total.labels(reason=reason).inc()
def record_embedding(
kind: str,
provider: str,