feat(document): configurable OCR timeout and fail-fast PDF size guard
Two ingest-robustness fixes from card 309 (OHR-Bench smoke-test triage).
The OCR backend timeout was a hardcoded 180s module constant, so a tenant
whose gateway has its own shorter ceiling couldn't tune it. Promote it to
DOCUMENT_OCR_TIMEOUT_SECONDS (default 180), resolved per call via get_settings
so an override applies without a restart.
Large, awkward PDFs (e.g. a 42 MB scanned DUDE) were handed straight to the
fast/OCR tiers, where they burned the full OCR timeout for zero recovered
text. Add a pre-parse size guard in the tiered PDF pipeline: a PDF over
DOCUMENT_MAX_PDF_SIZE_MB (default 50, 0 disables) fails fast with
parse_failed_reason="oversize" before any tier runs, so the existing
permanent-failure path marks the placeholder failed and records
astrolabe_document_parse_failed_total{reason="oversize"} instead of retrying.
Both knobs go through Settings + dynaconf validators (env-var keys verified by
regression tests) and are documented under Background Indexing Configuration.
Refs: Deck board 12 card 309 (AC #3 OCR timeout + size guard).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
457c115ef4
commit
523e4cb7b5
@@ -202,6 +202,27 @@ class ProcessorRegistry:
|
||||
"""
|
||||
settings = get_settings()
|
||||
|
||||
# Pre-parse size guard: a pathologically large PDF (e.g. a 42 MB scanned
|
||||
# DUDE) burns the OCR timeout for 0 chars. Fail fast with an explicit
|
||||
# reason so the caller marks the placeholder "failed" instead of
|
||||
# retrying. 0 disables the cap.
|
||||
max_pdf_mb = settings.document_max_pdf_size_mb
|
||||
if max_pdf_mb > 0 and len(content) > max_pdf_mb * 1024 * 1024:
|
||||
size_mb = len(content) / (1024 * 1024)
|
||||
logger.warning(
|
||||
"PDF %s is %.1f MB (> %.1f MB cap); failing fast as oversize",
|
||||
filename or "<bytes>",
|
||||
size_mb,
|
||||
max_pdf_mb,
|
||||
)
|
||||
return ProcessingResult(
|
||||
text="",
|
||||
metadata={"parse_failed_reason": "oversize"},
|
||||
processor="size_guard",
|
||||
success=False,
|
||||
error=(f"PDF exceeds size cap: {size_mb:.1f} MB > {max_pdf_mb:.1f} MB"),
|
||||
)
|
||||
|
||||
if settings.document_tier1_engine == "pymupdf":
|
||||
processor = self._pdf_processor_for_tier("structured")
|
||||
if processor is None:
|
||||
|
||||
Reference in New Issue
Block a user