fix(review): _enum_fields validation, gate classify_from_text flags, OCR warnings

Address PR #858 review round 3:

- document_tier1_engine / document_ocr_provider now validate + normalize via
  Settings.__post_init__ _enum_fields (the repo's canonical opt-in-enum pattern;
  case-insensitive) instead of dynaconf Validators. A typo now raises ValueError
  at load and "Gateway" normalizes to "gateway".
- classify_from_text gates no_text_layer/bad_text_layer on ocr_frac >=
  OCR_PAGE_FRACTION, matching classify_pdf -- a "fast"-routed doc with a few junk
  pages no longer emits a misleading flag (keeps the shadow vs hot-path
  classification metrics consistent).
- build_ocr_backend warns when an EXPLICIT provider is misconfigured
  (gateway without EMBEDDING_GATEWAY_URL, mistral without MISTRAL_API_KEY)
  instead of silently returning None.
- Pypdfium2FastProcessor.health_check probes the import; documented why
  OcrProcessor.health_check is unconditionally True (lazy per-tenant backends).
- Removed the leftover per-boundary / per-chunk debug logging loops.

Tests: enum normalization + rejection for the two new settings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-05 02:35:26 +02:00
co-authored by Claude Opus 4.8
parent f1272dfe84
commit bc957b5bc7
6 changed files with 37 additions and 25 deletions
@@ -213,8 +213,12 @@ def classify_from_text(
# recorded classification metric accurate rather than a misleading "ocr").
ocr_frac = (sum(p.needs_ocr for p in pages) / sampled) if sampled else 0.0
# Flags gated on ocr_frac >= OCR_PAGE_FRACTION (matching classify_pdf): a
# doc that routes "fast" must not carry a junk-layer flag just because a few
# isolated pages are bad -- otherwise the classification metric diverges
# between this hot path and the standalone classify_pdf.
flags: set[str] = set()
if sampled:
if sampled and ocr_frac >= OCR_PAGE_FRACTION:
if total_chars == 0:
flags.add("no_text_layer")
elif mean_quality < MIN_TEXT_QUALITY: