fix(ocr): round-3 review — guard unexpected batch status + tests/comments

Round 3 review (PR #910):
- Guard an unexpected terminal batch status in _process_batch: anything that
  isn't succeeded/failed (gateway version skew, a new lifecycle state) now marks
  the document parse-failed instead of falling through to _pages_to_text([]) — a
  0-chunk "success" that silently indexed empty text and re-submitted forever.
  Test added.
- gateway_batch_client.submit: raise an actionable ValueError on a 2xx response
  with no job_id (was a bare KeyError deep in the caller).
- Document that a _process_batch transport error intentionally propagates to
  procrastinate for retry rather than falling back to sync (opt-in batch wants
  the retry).
- Annotate _batch_client as GatewayBatchOcrClient | None (TYPE_CHECKING import
  already present); clarify the delete_stale_for_doc first-submit no-op comment.
- Add a parametrized build_gateway_batch_client test (the gateway-only invariant:
  mistral/none/no-URL -> None; gateway|auto + URL -> client).

1653 unit tests pass; ruff + ty green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-15 10:53:24 +02:00
co-authored by Claude Opus 4.8
parent 995e810d89
commit 55630ba25c
3 changed files with 69 additions and 3 deletions
@@ -277,7 +277,7 @@ class OcrProcessor(DocumentProcessor):
# documents. ``_batch_fallback_warned`` rate-limits the "can't batch,
# using sync" warning to once per pod.
self._batch_client_resolved = False
self._batch_client: Any = None
self._batch_client: GatewayBatchOcrClient | None = None
self._batch_client_lock: anyio.Lock | None = None
self._batch_fallback_warned = False
@@ -310,6 +310,11 @@ class OcrProcessor(DocumentProcessor):
# "pending" sentinel) when handled, or None to fall back to the
# synchronous path below (no gateway backend, or no per-doc identity — the
# inline/memory pool can't defer a poll).
#
# A transport error from _process_batch (e.g. the gateway briefly down)
# is intentionally NOT caught here: it propagates to procrastinate for a
# durable retry rather than silently falling back to sync. If you've opted
# into batch mode you want the retry, not an unexpected sync transcription.
if settings.document_ocr_mode == "batch":
batch_result = await self._process_batch(
content, content_type, filename, options, settings
@@ -446,7 +451,8 @@ class OcrProcessor(DocumentProcessor):
)
if job is None:
# New submission. Drop any superseded-version rows for this doc first
# (a re-edited file changes etag), then submit + record.
# (a re-edited file changes etag) — a no-op on the very first submit,
# one cheap DELETE on a resubmit. Then submit + record.
await store.delete_stale_for_doc(
user_id=user_id, doc_id=doc_id, doc_type=doc_type, keep_etag=etag
)
@@ -500,6 +506,24 @@ class OcrProcessor(DocumentProcessor):
success=False,
error=f"batch OCR failed: {result.error or 'unknown'}",
)
if not result.is_succeeded:
# Defensive: poll() maps anything that isn't "succeeded" to its raw
# status, and only pending/succeeded/failed are handled above. An
# unexpected terminal status (gateway version skew, a new lifecycle
# state) must NOT fall through to _pages_to_text([]) -> a 0-chunk
# "success" that silently indexes empty text and re-submits forever.
logger.warning(
"batch OCR job %s returned unexpected status %r; marking failed",
job.job_id,
result.status,
)
return ProcessingResult(
text="",
metadata={"parse_failed_reason": "error"},
processor=self.name,
success=False,
error=f"unexpected batch status: {result.status}",
)
text, boundaries = _pages_to_text(result.pages)
return ProcessingResult(
text=text,