fix: isolate PDF parse in a subprocess so a bad file can't OOM the pod

The document processor crash-looped on one pathological PDF: pymupdf4llm's
table/graphics detection over a page with ~1M vector path items ballooned past
the 2 GiB pod limit. The parse ran in a thread, so nothing could interrupt or
memory-bound it -- a single bad file OOM-killed the whole pod.

Run the parse in an isolated worker subprocess (anyio.to_process, cancellable)
with an RLIMIT_AS memory cap and a wall-clock timeout, so a pathological file
fails THAT document instead of the pod (new document_processors/_isolation.py).
Also pass graphics_limit (default 5000) to to_markdown -- validated to cut the
known trigger page from 112 s to 23 s with bounded memory.

On a permanent parse failure the processor returns success=False (instead of
raising, which would retry 3x); vector/processor.py marks the placeholder
"failed" and skips indexing, and the scanner stops re-queuing failed placeholders
until the file changes -- so a doomed file no longer churns.

New per-tenant (per-pod env) settings: DOCUMENT_PDF_GRAPHICS_LIMIT,
DOCUMENT_PARSE_TIMEOUT_SECONDS, DOCUMENT_PARSE_MEM_LIMIT_MB. New metric
astrolabe_document_parse_failed_total{reason=timeout|oom|error} surfaces hard
failures that previously killed the process before any except ran.

First PR of the tiered document-processor effort (Deck #199); tier 0/1/3
pipeline tracked separately.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-04 22:32:34 +02:00
co-authored by Claude Opus 4.8
parent 09e84783e5
commit 7db8d3e301
7 changed files with 450 additions and 19 deletions
+22
View File
@@ -132,6 +132,10 @@ _DEFAULTS: dict[str, Any] = {
# Document chunking
"document_chunk_size": 2048,
"document_chunk_overlap": 200,
# PDF parse isolation (OOM guard)
"document_pdf_graphics_limit": 5000,
"document_parse_timeout_seconds": 120,
"document_parse_mem_limit_mb": 1536,
# Observability
"metrics_enabled": True,
"metrics_port": 9090,
@@ -275,8 +279,11 @@ _dynaconf = Dynaconf(
Validator("VECTOR_SYNC_USER_POLL_INTERVAL", gte=1),
Validator("VERIFICATION_CONCURRENCY", gte=1),
Validator("DOCUMENT_CHUNK_SIZE", gte=1),
Validator("DOCUMENT_PARSE_TIMEOUT_SECONDS", gte=1),
Validator("DOCUMENT_PARSE_MEM_LIMIT_MB", gte=128),
# Non-negative
Validator("DOCUMENT_CHUNK_OVERLAP", gte=0),
Validator("DOCUMENT_PDF_GRAPHICS_LIMIT", gte=0),
# Non-empty strings
Validator("VECTOR_SYNC_PDF_TAG", len_min=1),
# Enum constraints
@@ -701,6 +708,18 @@ class Settings:
document_chunk_size: int = 2048 # Characters per chunk
document_chunk_overlap: int = 200 # Overlapping characters between chunks
# PDF parse isolation (OOM guard). The parse runs in a subprocess so one
# pathological file fails that doc, not the pod.
document_pdf_graphics_limit: int = (
5000 # to_markdown graphics cap; pages above skip graphics analysis
)
document_parse_timeout_seconds: int = (
120 # wall-clock cap per parse; the worker subprocess is killed on timeout
)
document_parse_mem_limit_mb: int = (
1536 # RLIMIT_AS in the parse subprocess (kept below the pod memory limit)
)
# Observability settings
metrics_enabled: bool = True
metrics_port: int = 9090
@@ -1309,6 +1328,9 @@ def get_settings() -> Settings:
# Document chunking settings
"document_chunk_size": "DOCUMENT_CHUNK_SIZE",
"document_chunk_overlap": "DOCUMENT_CHUNK_OVERLAP",
"document_pdf_graphics_limit": "DOCUMENT_PDF_GRAPHICS_LIMIT",
"document_parse_timeout_seconds": "DOCUMENT_PARSE_TIMEOUT_SECONDS",
"document_parse_mem_limit_mb": "DOCUMENT_PARSE_MEM_LIMIT_MB",
# Observability settings
"metrics_enabled": "METRICS_ENABLED",
"metrics_port": "METRICS_PORT",