feat(document): configurable OCR timeout and fail-fast PDF size guard

Two ingest-robustness fixes from card 309 (OHR-Bench smoke-test triage).

The OCR backend timeout was a hardcoded 180s module constant, so a tenant
whose gateway has its own shorter ceiling couldn't tune it. Promote it to
DOCUMENT_OCR_TIMEOUT_SECONDS (default 180), resolved per call via get_settings
so an override applies without a restart.

Large, awkward PDFs (e.g. a 42 MB scanned DUDE) were handed straight to the
fast/OCR tiers, where they burned the full OCR timeout for zero recovered
text. Add a pre-parse size guard in the tiered PDF pipeline: a PDF over
DOCUMENT_MAX_PDF_SIZE_MB (default 50, 0 disables) fails fast with
parse_failed_reason="oversize" before any tier runs, so the existing
permanent-failure path marks the placeholder failed and records
astrolabe_document_parse_failed_total{reason="oversize"} instead of retrying.

Both knobs go through Settings + dynaconf validators (env-var keys verified by
regression tests) and are documented under Background Indexing Configuration.

Refs: Deck board 12 card 309 (AC #3 OCR timeout + size guard).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-11 05:09:58 +02:00
co-authored by Claude Opus 4.8
parent 457c115ef4
commit 523e4cb7b5
7 changed files with 163 additions and 2 deletions
@@ -31,7 +31,9 @@ from .base import DocumentProcessor, ProcessingResult
logger = logging.getLogger(__name__)
_OCR_TIMEOUT_SECONDS = 180.0
# Connect timeout for the OCR backend request. The overall (read) timeout is
# configurable via DOCUMENT_OCR_TIMEOUT_SECONDS and resolved per call.
_OCR_CONNECT_TIMEOUT_SECONDS = 10.0
def _pages_to_text(
@@ -93,8 +95,11 @@ class _GatewayOcrBackend(_OcrBackend):
"document_b64": base64.b64encode(content).decode("ascii"),
"mime_type": mime_type,
}
# Resolve the timeout per call (get_settings builds fresh, so a test or
# tenant override is honoured without a restart).
ocr_timeout = get_settings().document_ocr_timeout_seconds
async with httpx.AsyncClient(
timeout=httpx.Timeout(_OCR_TIMEOUT_SECONDS, connect=10.0)
timeout=httpx.Timeout(ocr_timeout, connect=_OCR_CONNECT_TIMEOUT_SECONDS)
) as client:
resp = await client.post(self._url, json=payload, headers=headers)
resp.raise_for_status()