refactor(ingest): doc legacy ocr queue; fix docstrings + getattr guard
Address claude-review round 2 on #922: - Legacy `ingest-ocr` tier resolution (important 1): document why it deliberately resolves to `fast` rather than mapping to `ocr-upstream` — stranded pre-split jobs re-extract empty and re-escalate via the ladder to the cheap `ocr-incluster` rung, keeping them OFF the paid upstream rung. Added a tier_for_queue(LEGACY_INGEST_QUEUE_OCR) == "fast" assertion. - Double get_settings() in `_get_batch_client` (important 2): bind once to a local. - Stale docstrings (important 3): OcrProcessor (serves both rungs now), _tier_available (both OCR rungs gated), evaluate_escalation (targets ocr-incluster, falls through to ocr-upstream). - Misconfigured model_setting (nit 5): OcrProcessor.__init__ raises ValueError on an unknown settings attr (fail-fast at startup vs AttributeError mid-OCR); also removes the dynamic-getattr static-analysis smell SonarCloud flagged. - Redundant guard (nit 4): kept `and ocr_tier is not None` — it's required for ty to narrow ocr_tier to str for record_document_escalation; added a comment. - Test gap (nit 6): added a test pinning the CURRENT incluster-failure -> tier-1 fallback (does NOT cascade to upstream) so the future 503-escalation change is an explicit diff. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
87b8edd139
commit
6e32bd9561
@@ -287,7 +287,9 @@ def build_ocr_backend(
|
||||
|
||||
|
||||
class OcrProcessor(DocumentProcessor):
|
||||
"""Tier-3 OCR processor (gateway or direct Mistral backend)."""
|
||||
"""OCR processor for both OCR rungs — tier2 in-cluster (gateway-only GPU) and
|
||||
tier3 upstream (gateway or direct Mistral backend). One class, two registered
|
||||
instances bound to different ``(tier, model_setting, gateway_only)``."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -303,6 +305,11 @@ class OcrProcessor(DocumentProcessor):
|
||||
# gateway-only (its model, e.g. surya, is reachable solely via the
|
||||
# gateway); the upstream rung keeps the configurable gateway/mistral
|
||||
# selection. surya is NEVER hard-coded here — only a config default.
|
||||
# Fail fast on a misconfigured model_setting (a typo in a constructor call)
|
||||
# so it surfaces at startup, not as an AttributeError mid-OCR. The string
|
||||
# only ever comes from hardcoded defaults in __init__.py, never user input.
|
||||
if not hasattr(Settings, model_setting):
|
||||
raise ValueError(f"Unknown model_setting: {model_setting!r}")
|
||||
self._name = name
|
||||
self._tier = tier
|
||||
self._model_setting = model_setting
|
||||
@@ -450,9 +457,10 @@ class OcrProcessor(DocumentProcessor):
|
||||
self._batch_client_lock = anyio.Lock()
|
||||
async with self._batch_client_lock:
|
||||
if not self._batch_client_resolved: # double-checked
|
||||
settings = get_settings()
|
||||
self._batch_client = build_gateway_batch_client(
|
||||
get_settings(),
|
||||
model=getattr(get_settings(), self._model_setting),
|
||||
settings,
|
||||
model=getattr(settings, self._model_setting),
|
||||
)
|
||||
self._batch_client_resolved = True
|
||||
return self._batch_client
|
||||
|
||||
@@ -360,6 +360,10 @@ class ProcessorRegistry:
|
||||
from_tier, settings, minimum="ocr-incluster"
|
||||
)
|
||||
ocr = self._pdf_processor_for_tier(ocr_tier) if ocr_tier else None
|
||||
# `ocr is not None` already implies `ocr_tier is not None` at runtime,
|
||||
# but the type checker can't infer that across the conditional above,
|
||||
# so the explicit guard narrows `ocr_tier` to `str` for the
|
||||
# record_document_escalation(from_tier, ocr_tier, reason) call below.
|
||||
if ocr is not None and ocr_tier is not None:
|
||||
reason = (
|
||||
"corrupt_glyphs"
|
||||
@@ -514,8 +518,8 @@ class ProcessorRegistry:
|
||||
``ignore_ocr_enabled`` drops only the OCR-enabled gate (not the registered-
|
||||
processor requirement): it answers "would this tier run if OCR were turned
|
||||
on?" — used to compute the *ideal* escalation target for the what-if-OCR
|
||||
suppressed-escalation signal. (Today only ``ocr`` has an enabled gate; a
|
||||
future per-tier gate would extend the condition below.)
|
||||
suppressed-escalation signal. (Both OCR rungs — ``ocr-incluster`` and
|
||||
``ocr-upstream`` — have their own enabled gate; non-OCR tiers have none.)
|
||||
"""
|
||||
if self._pdf_processor_for_tier(tier) is None:
|
||||
return False
|
||||
@@ -624,9 +628,11 @@ class ProcessorRegistry:
|
||||
|
||||
Target-tier routing:
|
||||
|
||||
- ``total_chars == 0`` (scanned / no text layer) -> target the ``ocr``
|
||||
tier directly. Text-extractor tiers (``structured``) cannot conjure
|
||||
text from a pure raster scan, so a structured hop would just be wasted.
|
||||
- ``total_chars == 0`` (scanned / no text layer) -> target the cheapest
|
||||
OCR rung (``ocr-incluster``) directly; ``next_available_tier`` falls
|
||||
through to ``ocr-upstream`` if in-cluster is disabled/unregistered.
|
||||
Text-extractor tiers (``structured``) cannot conjure text from a pure
|
||||
raster scan, so a structured hop would just be wasted.
|
||||
- glyph-corrupt text layer (``recommended_tier == "structured"``) -> target
|
||||
the ``structured`` tier; pymupdf re-extracts a broken-/ToUnicode layer
|
||||
correctly, so OCR is never the target for this case.
|
||||
|
||||
@@ -119,6 +119,15 @@ def tier_for_queue(queue: str | None) -> str:
|
||||
The queue-aware task uses this to pick which single tier to parse with: the
|
||||
job's current queue *is* its tier. A job on the legacy ``ingest`` queue (or
|
||||
any unrecognised queue) defaults to the cheapest tier.
|
||||
|
||||
The pre-split legacy OCR queue ``ingest-ocr`` (``LEGACY_INGEST_QUEUE_OCR``)
|
||||
is deliberately NOT mapped here, so it also resolves to ``fast``. These are
|
||||
in-flight jobs enqueued before the tier2/tier3 split; running them at fast
|
||||
re-extracts (an empty layer for a scanned doc), which re-enters the ladder
|
||||
and naturally re-escalates to ``ocr-incluster`` (the cheap GPU rung) — one
|
||||
extra cheap hop, but it keeps stranded legacy OCR jobs OFF the paid upstream
|
||||
rung rather than mapping them straight to ``ocr-upstream``. The set is
|
||||
transient (only during a single rollout window).
|
||||
"""
|
||||
return _QUEUE_TIERS.get(queue or "", "fast")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user