fix(review): cache OCR backend, drop asserts, real pipeline_tier, guard zero-page
Address PR #858 review: - 🔴 OcrProcessor now resolves its backend once and reuses it. Rebuilding per call created a fresh GatewayTokenProvider each time -- discarding its M2M-token cache, so every OCR'd document fetched a new token -- and a new Mistral client. - 🔴 build_ocr_backend uses explicit ValueError (not assert, which is stripped under `python -O`) for the gateway M2M triple. - PIPELINE_TIER in the Qdrant payload now reflects the tier that actually produced the doc: the registry stamps result.metadata["pipeline_tier"] and the processor reads it (was hardcoded "fast", wrong for OCR/structured). - Escalation now requires classification.page_count > 0, so a zero-page (empty/corrupt) PDF isn't pointlessly sent to OCR; documented that a fast FAILURE (encrypted/unopenable) is a hard failure and is not OCR-escalated. - Documented the OCR page_boundaries separator-attribution choice. - Downgraded the per-document page-boundary / page-assignment INFO logs to debug. New tests: zero-page no-escalation, pipeline_tier stamping. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
4dbf362261
commit
1634e8adc2
@@ -589,7 +589,7 @@ async def _index_document(
|
||||
# Diagnostic: Log page boundary information if available
|
||||
if "page_boundaries" in file_metadata:
|
||||
page_boundaries = file_metadata["page_boundaries"]
|
||||
logger.info(
|
||||
logger.debug(
|
||||
"Page boundaries for %s: %s pages, text length: %s",
|
||||
file_path,
|
||||
len(page_boundaries),
|
||||
@@ -652,7 +652,7 @@ async def _index_document(
|
||||
|
||||
# Diagnostic: Verify page number assignment
|
||||
assigned_count = sum(1 for c in chunks if c.page_number is not None)
|
||||
logger.info(
|
||||
logger.debug(
|
||||
"Assigned page numbers to %s/%s chunks for %s",
|
||||
assigned_count,
|
||||
len(chunks),
|
||||
@@ -926,7 +926,11 @@ async def _index_document(
|
||||
# Decomposition payload keys (design §10.2), additive.
|
||||
payload_keys.PROCESSOR_VERSION: "monolith-v1",
|
||||
payload_keys.PARSED_AT: indexed_at,
|
||||
payload_keys.PIPELINE_TIER: "fast",
|
||||
# Actual tier that produced this doc (registry stamps it on
|
||||
# the result metadata); non-PDF doc types stay "fast".
|
||||
payload_keys.PIPELINE_TIER: file_metadata.get(
|
||||
"pipeline_tier", "fast"
|
||||
),
|
||||
payload_keys.EMBEDDING_IDENTITY: _embedding_identity,
|
||||
payload_keys.ACL_HASH: _acl_hash,
|
||||
# File-specific metadata (PDF, etc.)
|
||||
|
||||
Reference in New Issue
Block a user