fix(review): cache OCR backend, drop asserts, real pipeline_tier, guard zero-page

Address PR #858 review:

- 🔴 OcrProcessor now resolves its backend once and reuses it. Rebuilding per
  call created a fresh GatewayTokenProvider each time -- discarding its M2M-token
  cache, so every OCR'd document fetched a new token -- and a new Mistral client.
- 🔴 build_ocr_backend uses explicit ValueError (not assert, which is stripped
  under `python -O`) for the gateway M2M triple.
- PIPELINE_TIER in the Qdrant payload now reflects the tier that actually
  produced the doc: the registry stamps result.metadata["pipeline_tier"] and the
  processor reads it (was hardcoded "fast", wrong for OCR/structured).
- Escalation now requires classification.page_count > 0, so a zero-page
  (empty/corrupt) PDF isn't pointlessly sent to OCR; documented that a fast
  FAILURE (encrypted/unopenable) is a hard failure and is not OCR-escalated.
- Documented the OCR page_boundaries separator-attribution choice.
- Downgraded the per-document page-boundary / page-assignment INFO logs to debug.

New tests: zero-page no-escalation, pipeline_tier stamping.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-05 02:13:09 +02:00
co-authored by Claude Opus 4.8
parent 4dbf362261
commit 1634e8adc2
4 changed files with 85 additions and 17 deletions
+7 -3
View File
@@ -589,7 +589,7 @@ async def _index_document(
# Diagnostic: Log page boundary information if available
if "page_boundaries" in file_metadata:
page_boundaries = file_metadata["page_boundaries"]
logger.info(
logger.debug(
"Page boundaries for %s: %s pages, text length: %s",
file_path,
len(page_boundaries),
@@ -652,7 +652,7 @@ async def _index_document(
# Diagnostic: Verify page number assignment
assigned_count = sum(1 for c in chunks if c.page_number is not None)
logger.info(
logger.debug(
"Assigned page numbers to %s/%s chunks for %s",
assigned_count,
len(chunks),
@@ -926,7 +926,11 @@ async def _index_document(
# Decomposition payload keys (design §10.2), additive.
payload_keys.PROCESSOR_VERSION: "monolith-v1",
payload_keys.PARSED_AT: indexed_at,
payload_keys.PIPELINE_TIER: "fast",
# Actual tier that produced this doc (registry stamps it on
# the result metadata); non-PDF doc types stay "fast".
payload_keys.PIPELINE_TIER: file_metadata.get(
"pipeline_tier", "fast"
),
payload_keys.EMBEDDING_IDENTITY: _embedding_identity,
payload_keys.ACL_HASH: _acl_hash,
# File-specific metadata (PDF, etc.)