fix(review): _enum_fields validation, gate classify_from_text flags, OCR warnings

Address PR #858 review round 3:

- document_tier1_engine / document_ocr_provider now validate + normalize via
  Settings.__post_init__ _enum_fields (the repo's canonical opt-in-enum pattern;
  case-insensitive) instead of dynaconf Validators. A typo now raises ValueError
  at load and "Gateway" normalizes to "gateway".
- classify_from_text gates no_text_layer/bad_text_layer on ocr_frac >=
  OCR_PAGE_FRACTION, matching classify_pdf -- a "fast"-routed doc with a few junk
  pages no longer emits a misleading flag (keeps the shadow vs hot-path
  classification metrics consistent).
- build_ocr_backend warns when an EXPLICIT provider is misconfigured
  (gateway without EMBEDDING_GATEWAY_URL, mistral without MISTRAL_API_KEY)
  instead of silently returning None.
- Pypdfium2FastProcessor.health_check probes the import; documented why
  OcrProcessor.health_check is unconditionally True (lazy per-tenant backends).
- Removed the leftover per-boundary / per-chunk debug logging loops.

Tests: enum normalization + rejection for the two new settings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-05 02:35:26 +02:00
co-authored by Claude Opus 4.8
parent f1272dfe84
commit bc957b5bc7
6 changed files with 37 additions and 25 deletions
+4 -5
View File
@@ -301,12 +301,9 @@ _dynaconf = Dynaconf(
Validator("DOCUMENT_CHUNK_OVERLAP", gte=0),
# Non-empty strings
Validator("VECTOR_SYNC_PDF_TAG", len_min=1),
# Enum constraints
# Enum constraints (document_* enums are validated + normalized in
# __post_init__ via _enum_fields instead, for case-insensitive input).
Validator("LOG_FORMAT", is_in=["text", "json"]),
Validator("DOCUMENT_TIER1_ENGINE", is_in=["pypdfium2", "pymupdf"]),
Validator(
"DOCUMENT_OCR_PROVIDER", is_in=["auto", "gateway", "mistral", "none"]
),
Validator(
"LOG_LEVEL",
is_in=["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"],
@@ -887,6 +884,8 @@ class Settings:
"embedding_provider": {"autodetect", "gateway"},
"mcp_role": {"api", "worker", "all"},
"collection_metadata_source": {"qdrant", "api"},
"document_tier1_engine": {"pypdfium2", "pymupdf"},
"document_ocr_provider": {"auto", "gateway", "mistral", "none"},
}
for _field, _allowed in _enum_fields.items():
_val = (getattr(self, _field) or "").strip().lower()