fix: isolate PDF parse in a subprocess so a bad file can't OOM the pod
The document processor crash-looped on one pathological PDF: pymupdf4llm's
table/graphics detection over a page with ~1M vector path items ballooned past
the 2 GiB pod limit. The parse ran in a thread, so nothing could interrupt or
memory-bound it -- a single bad file OOM-killed the whole pod.
Run the parse in an isolated worker subprocess (anyio.to_process, cancellable)
with an RLIMIT_AS memory cap and a wall-clock timeout, so a pathological file
fails THAT document instead of the pod (new document_processors/_isolation.py).
Also pass graphics_limit (default 5000) to to_markdown -- validated to cut the
known trigger page from 112 s to 23 s with bounded memory.
On a permanent parse failure the processor returns success=False (instead of
raising, which would retry 3x); vector/processor.py marks the placeholder
"failed" and skips indexing, and the scanner stops re-queuing failed placeholders
until the file changes -- so a doomed file no longer churns.
New per-tenant (per-pod env) settings: DOCUMENT_PDF_GRAPHICS_LIMIT,
DOCUMENT_PARSE_TIMEOUT_SECONDS, DOCUMENT_PARSE_MEM_LIMIT_MB. New metric
astrolabe_document_parse_failed_total{reason=timeout|oom|error} surfaces hard
failures that previously killed the process before any except ran.
First PR of the tiered document-processor effort (Deck #199); tier 0/1/3
pipeline tracked separately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
09e84783e5
commit
7db8d3e301
@@ -132,6 +132,10 @@ _DEFAULTS: dict[str, Any] = {
|
||||
# Document chunking
|
||||
"document_chunk_size": 2048,
|
||||
"document_chunk_overlap": 200,
|
||||
# PDF parse isolation (OOM guard)
|
||||
"document_pdf_graphics_limit": 5000,
|
||||
"document_parse_timeout_seconds": 120,
|
||||
"document_parse_mem_limit_mb": 1536,
|
||||
# Observability
|
||||
"metrics_enabled": True,
|
||||
"metrics_port": 9090,
|
||||
@@ -275,8 +279,11 @@ _dynaconf = Dynaconf(
|
||||
Validator("VECTOR_SYNC_USER_POLL_INTERVAL", gte=1),
|
||||
Validator("VERIFICATION_CONCURRENCY", gte=1),
|
||||
Validator("DOCUMENT_CHUNK_SIZE", gte=1),
|
||||
Validator("DOCUMENT_PARSE_TIMEOUT_SECONDS", gte=1),
|
||||
Validator("DOCUMENT_PARSE_MEM_LIMIT_MB", gte=128),
|
||||
# Non-negative
|
||||
Validator("DOCUMENT_CHUNK_OVERLAP", gte=0),
|
||||
Validator("DOCUMENT_PDF_GRAPHICS_LIMIT", gte=0),
|
||||
# Non-empty strings
|
||||
Validator("VECTOR_SYNC_PDF_TAG", len_min=1),
|
||||
# Enum constraints
|
||||
@@ -701,6 +708,18 @@ class Settings:
|
||||
document_chunk_size: int = 2048 # Characters per chunk
|
||||
document_chunk_overlap: int = 200 # Overlapping characters between chunks
|
||||
|
||||
# PDF parse isolation (OOM guard). The parse runs in a subprocess so one
|
||||
# pathological file fails that doc, not the pod.
|
||||
document_pdf_graphics_limit: int = (
|
||||
5000 # to_markdown graphics cap; pages above skip graphics analysis
|
||||
)
|
||||
document_parse_timeout_seconds: int = (
|
||||
120 # wall-clock cap per parse; the worker subprocess is killed on timeout
|
||||
)
|
||||
document_parse_mem_limit_mb: int = (
|
||||
1536 # RLIMIT_AS in the parse subprocess (kept below the pod memory limit)
|
||||
)
|
||||
|
||||
# Observability settings
|
||||
metrics_enabled: bool = True
|
||||
metrics_port: int = 9090
|
||||
@@ -1309,6 +1328,9 @@ def get_settings() -> Settings:
|
||||
# Document chunking settings
|
||||
"document_chunk_size": "DOCUMENT_CHUNK_SIZE",
|
||||
"document_chunk_overlap": "DOCUMENT_CHUNK_OVERLAP",
|
||||
"document_pdf_graphics_limit": "DOCUMENT_PDF_GRAPHICS_LIMIT",
|
||||
"document_parse_timeout_seconds": "DOCUMENT_PARSE_TIMEOUT_SECONDS",
|
||||
"document_parse_mem_limit_mb": "DOCUMENT_PARSE_MEM_LIMIT_MB",
|
||||
# Observability settings
|
||||
"metrics_enabled": "METRICS_ENABLED",
|
||||
"metrics_port": "METRICS_PORT",
|
||||
|
||||
Reference in New Issue
Block a user