fix: isolate PDF parse in a subprocess so a bad file can't OOM the pod
The document processor crash-looped on one pathological PDF: pymupdf4llm's
table/graphics detection over a page with ~1M vector path items ballooned past
the 2 GiB pod limit. The parse ran in a thread, so nothing could interrupt or
memory-bound it -- a single bad file OOM-killed the whole pod.
Run the parse in an isolated worker subprocess (anyio.to_process, cancellable)
with an RLIMIT_AS memory cap and a wall-clock timeout, so a pathological file
fails THAT document instead of the pod (new document_processors/_isolation.py).
Also pass graphics_limit (default 5000) to to_markdown -- validated to cut the
known trigger page from 112 s to 23 s with bounded memory.
On a permanent parse failure the processor returns success=False (instead of
raising, which would retry 3x); vector/processor.py marks the placeholder
"failed" and skips indexing, and the scanner stops re-queuing failed placeholders
until the file changes -- so a doomed file no longer churns.
New per-tenant (per-pod env) settings: DOCUMENT_PDF_GRAPHICS_LIMIT,
DOCUMENT_PARSE_TIMEOUT_SECONDS, DOCUMENT_PARSE_MEM_LIMIT_MB. New metric
astrolabe_document_parse_failed_total{reason=timeout|oom|error} surfaces hard
failures that previously killed the process before any except ran.
First PR of the tiered document-processor effort (Deck #199); tier 0/1/3
pipeline tracked separately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
09e84783e5
commit
7db8d3e301
@@ -526,13 +526,24 @@ async def scan_user_documents(
|
||||
# File modified since last indexing
|
||||
needs_indexing = True
|
||||
elif existing_metadata.get("is_placeholder", False):
|
||||
# Placeholder exists - check if it's stale (processing may have failed)
|
||||
# Only requeue if placeholder is older than 5x scan interval
|
||||
# (Large PDFs can take 3-4 minutes to process)
|
||||
# Placeholder exists - check its status / staleness.
|
||||
queued_at = existing_metadata.get("queued_at", 0)
|
||||
placeholder_age = time.time() - queued_at
|
||||
stale_threshold = get_settings().vector_sync_scan_interval * 5
|
||||
if placeholder_age > stale_threshold:
|
||||
if existing_metadata.get("status") == "failed":
|
||||
# A permanent parse failure (e.g. an isolated-worker
|
||||
# OOM/timeout on a pathological PDF). Don't keep
|
||||
# re-queuing an unchanged file that will just fail
|
||||
# again -- the modified_at branch above still retries
|
||||
# it once the file actually changes.
|
||||
logger.debug(
|
||||
"Skipping file %s (ID: %s): previous parse failed permanently",
|
||||
file_path,
|
||||
file_id,
|
||||
)
|
||||
elif placeholder_age > stale_threshold:
|
||||
# Only requeue if placeholder is older than 5x scan
|
||||
# interval (large PDFs can take minutes to process).
|
||||
logger.debug(
|
||||
"Found stale placeholder for file %s (ID: %s) (age=%ss), requeuing",
|
||||
file_path,
|
||||
|
||||
Reference in New Issue
Block a user