admin/mcp-nextcloud
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(document): configurable OCR timeout and fail-fast PDF size guard

Browse Source 
Two ingest-robustness fixes from card 309 (OHR-Bench smoke-test triage).

The OCR backend timeout was a hardcoded 180s module constant, so a tenant
whose gateway has its own shorter ceiling couldn't tune it. Promote it to
DOCUMENT_OCR_TIMEOUT_SECONDS (default 180), resolved per call via get_settings
so an override applies without a restart.

Large, awkward PDFs (e.g. a 42 MB scanned DUDE) were handed straight to the
fast/OCR tiers, where they burned the full OCR timeout for zero recovered
text. Add a pre-parse size guard in the tiered PDF pipeline: a PDF over
DOCUMENT_MAX_PDF_SIZE_MB (default 50, 0 disables) fails fast with
parse_failed_reason="oversize" before any tier runs, so the existing
permanent-failure path marks the placeholder failed and records
astrolabe_document_parse_failed_total{reason="oversize"} instead of retrying.

Both knobs go through Settings + dynaconf validators (env-var keys verified by
regression tests) and are documented under Background Indexing Configuration.

Refs: Deck board 12 card 309 (AC #3 OCR timeout + size guard).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-11 05:09:58 +02:00
co-authored by Claude Opus 4.8
parent 457c115ef4
commit 523e4cb7b5
7 changed files with 163 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/configuration.md
+17
View File
@@ -536,6 +536,23 @@ DOCUMENT_CHUNK_OVERLAP=200 # Overlapping characters between chunks (d
> **Note:** The `VECTOR_SYNC_*` tuning parameters keep their names as they're implementation details. Only the user-facing feature flag was renamed to `ENABLE_SEMANTIC_SEARCH`.
#### Document parsing robustness (PDF)
These guard the parse/OCR tiers against pathological PDFs. Defaults are safe;
tune per tenant when a corpus has very large scans or a gateway with its own
shorter OCR ceiling:
```dotenv
DOCUMENT_PARSE_TIMEOUT_SECONDS=120 # Wall-clock cap per isolated parse (default: 120)
DOCUMENT_OCR_TIMEOUT_SECONDS=180 # OCR backend request timeout (default: 180)
DOCUMENT_MAX_PDF_SIZE_MB=50 # Pre-parse size cap; 0 disables (default: 50)
```
A PDF larger than `DOCUMENT_MAX_PDF_SIZE_MB` fails fast with reason `oversize`
(exported on `astrolabe_document_parse_failed_total{reason="oversize"}`) instead
of being handed to the tiers, where a 40+ MB scan would otherwise burn the full
OCR timeout for zero recovered text.
### Embedding Service Configuration
The server picks an embedding provider via auto-detection. Priority order