fix(review): cache OCR backend, drop asserts, real pipeline_tier, guard zero-page
Address PR #858 review: - 🔴 OcrProcessor now resolves its backend once and reuses it. Rebuilding per call created a fresh GatewayTokenProvider each time -- discarding its M2M-token cache, so every OCR'd document fetched a new token -- and a new Mistral client. - 🔴 build_ocr_backend uses explicit ValueError (not assert, which is stripped under `python -O`) for the gateway M2M triple. - PIPELINE_TIER in the Qdrant payload now reflects the tier that actually produced the doc: the registry stamps result.metadata["pipeline_tier"] and the processor reads it (was hardcoded "fast", wrong for OCR/structured). - Escalation now requires classification.page_count > 0, so a zero-page (empty/corrupt) PDF isn't pointlessly sent to OCR; documented that a fast FAILURE (encrypted/unopenable) is a hard failure and is not OCR-escalated. - Documented the OCR page_boundaries separator-attribution choice. - Downgraded the per-document page-boundary / page-assignment INFO logs to debug. New tests: zero-page no-escalation, pipeline_tier stamping. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
4dbf362261
commit
1634e8adc2
@@ -245,10 +245,16 @@ class ProcessorRegistry:
|
||||
)
|
||||
|
||||
# Escalate scanned / no-text-layer PDFs to OCR (tier-3) when enabled and
|
||||
# a provider is registered. The fast tier is terminal otherwise.
|
||||
# a provider is registered. The fast tier is terminal otherwise. Note: a
|
||||
# fast FAILURE (encrypted/corrupt -- result.success False, no
|
||||
# classification) is NOT escalated; a PDF pypdfium2 can't open is treated
|
||||
# as a hard failure (OCR reads the same bytes and would usually fail
|
||||
# too). The page_count guard skips a zero-page (empty/corrupt) PDF, which
|
||||
# OCR can't help either.
|
||||
if (
|
||||
classification is not None
|
||||
and classification.recommended_tier == "ocr"
|
||||
and classification.page_count > 0
|
||||
and settings.document_ocr_enabled
|
||||
):
|
||||
ocr = self._pdf_processor_for_tier("ocr")
|
||||
@@ -359,6 +365,10 @@ class ProcessorRegistry:
|
||||
raise
|
||||
|
||||
duration = time.time() - start_time
|
||||
# Record the tier that actually produced this result so downstream
|
||||
# (Qdrant payload pipeline_tier, analytics) reflects escalation
|
||||
# instead of a hardcoded "fast".
|
||||
result.metadata.setdefault("pipeline_tier", tier)
|
||||
pages = int(result.metadata.get("page_count", 0) or 0)
|
||||
chars = len(result.text)
|
||||
status = "success" if result.success else "error"
|
||||
|
||||
Reference in New Issue
Block a user