fix: isolate PDF parse in a subprocess so a bad file can't OOM the pod
The document processor crash-looped on one pathological PDF: pymupdf4llm's
table/graphics detection over a page with ~1M vector path items ballooned past
the 2 GiB pod limit. The parse ran in a thread, so nothing could interrupt or
memory-bound it -- a single bad file OOM-killed the whole pod.
Run the parse in an isolated worker subprocess (anyio.to_process, cancellable)
with an RLIMIT_AS memory cap and a wall-clock timeout, so a pathological file
fails THAT document instead of the pod (new document_processors/_isolation.py).
Also pass graphics_limit (default 5000) to to_markdown -- validated to cut the
known trigger page from 112 s to 23 s with bounded memory.
On a permanent parse failure the processor returns success=False (instead of
raising, which would retry 3x); vector/processor.py marks the placeholder
"failed" and skips indexing, and the scanner stops re-queuing failed placeholders
until the file changes -- so a doomed file no longer churns.
New per-tenant (per-pod env) settings: DOCUMENT_PDF_GRAPHICS_LIMIT,
DOCUMENT_PARSE_TIMEOUT_SECONDS, DOCUMENT_PARSE_MEM_LIMIT_MB. New metric
astrolabe_document_parse_failed_total{reason=timeout|oom|error} surfaces hard
failures that previously killed the process before any except ran.
First PR of the tiered document-processor effort (Deck #199); tier 0/1/3
pipeline tracked separately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
09e84783e5
commit
7db8d3e301
@@ -261,6 +261,17 @@ document_escalation_total = Counter(
|
||||
["from_tier", "to_tier", "reason"],
|
||||
)
|
||||
|
||||
# Hard parse failures: the parse now runs in an isolated subprocess, so a
|
||||
# timeout/OOM that kills the worker is caught here. This is distinct from
|
||||
# ``document_parse_total{status="error"}`` (an in-process exception): a hard
|
||||
# OOM previously killed the pod before any except ran, so it incremented
|
||||
# nothing -- this counter makes those failures visible.
|
||||
document_parse_failed_total = Counter(
|
||||
"astrolabe_document_parse_failed_total",
|
||||
"Document parses that failed in the isolated worker (process killed)",
|
||||
["reason"], # reason: timeout | oom | error
|
||||
)
|
||||
|
||||
# --- Embedding stages ---------------------------------------------------------
|
||||
|
||||
embedding_duration_seconds = Histogram(
|
||||
@@ -617,6 +628,15 @@ def record_document_escalation(from_tier: str, to_tier: str, reason: str) -> Non
|
||||
).inc()
|
||||
|
||||
|
||||
def record_document_parse_failed(reason: str) -> None:
|
||||
"""Record a hard parse failure from the isolated worker.
|
||||
|
||||
Args:
|
||||
reason: ``timeout`` | ``oom`` | ``error``
|
||||
"""
|
||||
document_parse_failed_total.labels(reason=reason).inc()
|
||||
|
||||
|
||||
def record_embedding(
|
||||
kind: str,
|
||||
provider: str,
|
||||
|
||||
Reference in New Issue
Block a user