feat(document): configurable OCR timeout and fail-fast PDF size guard
Two ingest-robustness fixes from card 309 (OHR-Bench smoke-test triage).
The OCR backend timeout was a hardcoded 180s module constant, so a tenant
whose gateway has its own shorter ceiling couldn't tune it. Promote it to
DOCUMENT_OCR_TIMEOUT_SECONDS (default 180), resolved per call via get_settings
so an override applies without a restart.
Large, awkward PDFs (e.g. a 42 MB scanned DUDE) were handed straight to the
fast/OCR tiers, where they burned the full OCR timeout for zero recovered
text. Add a pre-parse size guard in the tiered PDF pipeline: a PDF over
DOCUMENT_MAX_PDF_SIZE_MB (default 50, 0 disables) fails fast with
parse_failed_reason="oversize" before any tier runs, so the existing
permanent-failure path marks the placeholder failed and records
astrolabe_document_parse_failed_total{reason="oversize"} instead of retrying.
Both knobs go through Settings + dynaconf validators (env-var keys verified by
regression tests) and are documented under Background Indexing Configuration.
Refs: Deck board 12 card 309 (AC #3 OCR timeout + size guard).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
457c115ef4
commit
523e4cb7b5
@@ -213,6 +213,24 @@ class TestChunkConfigValidation:
|
||||
_reload_config()
|
||||
assert get_settings().document_chunk_page_aware is False
|
||||
|
||||
def test_ocr_timeout_default_and_env_override(self):
|
||||
"""document_ocr_timeout_seconds defaults to 180 and reads its env var.
|
||||
|
||||
Guards the _DEFAULTS-key-must-match-env-var footgun: a mismatch would
|
||||
leave the override silently ignored.
|
||||
"""
|
||||
assert Settings().document_ocr_timeout_seconds == 180.0
|
||||
with patch.dict(os.environ, {"DOCUMENT_OCR_TIMEOUT_SECONDS": "45"}, clear=True):
|
||||
_reload_config()
|
||||
assert get_settings().document_ocr_timeout_seconds == 45.0
|
||||
|
||||
def test_max_pdf_size_default_and_env_override(self):
|
||||
"""document_max_pdf_size_mb defaults to 50 and reads its env var."""
|
||||
assert Settings().document_max_pdf_size_mb == 50.0
|
||||
with patch.dict(os.environ, {"DOCUMENT_MAX_PDF_SIZE_MB": "12.5"}, clear=True):
|
||||
_reload_config()
|
||||
assert get_settings().document_max_pdf_size_mb == 12.5
|
||||
|
||||
def test_valid_chunk_settings(self):
|
||||
"""Test valid chunk size and overlap configuration."""
|
||||
settings = Settings(
|
||||
|
||||
Reference in New Issue
Block a user