Chris CoutinhoandClaude Opus 4.8
3b7e8d779b
Add DOCUMENT_OCR_MODE=sync|batch (default sync). In batch mode the tier-3 OCR
processor submits documents to the embedding gateway's async Batch OCR routes
(POST /v1/ocr/batch + GET /v1/ocr/batch/{job_id}, astrolabe-cloud-website#372)
for ~50% cheaper large-corpus backfill. The direct Mistral OCR path is left
untouched. Tracked on Deck #332.
Batch jobs run minutes-hours, so the OCR tier cannot block (the procrastinate
worker reclaims jobs in `doing` after INGEST_STALLED_JOB_SECONDS). Instead it
submits, records the gateway job id in a new per-tenant `batch_ocr_jobs` table
(procrastinate args are immutable across retries), and raises a BatchPending
signal that TieredEscalationStrategy turns into a same-queue deferred re-poll —
releasing the worker slot between polls. On completion the per-page markdown is
indexed like the sync path; a failure or a job past
DOCUMENT_OCR_BATCH_MAX_WAIT_SECONDS marks the document parse-failed.
Batch is opt-in and gateway-only: with the direct mistral backend, no gateway
URL, or the inline/memory pipeline (which can't defer), it falls back to sync.
One batch job per document (coalescing N docs/job is a follow-up).
- embedding/gateway_batch_client.py: submit/poll client (reuses GatewayTokenProvider).
- vector/batch_ocr_store.py + migration 008: job tracking (portable SQLite+PG).
- document_processors/escalation.py: BatchPending control-flow signal.
- document_processors/ocr.py: batch state machine + sync fallback.
- vector/processor.py: thread doc identity to the OCR tier; raise BatchPending
from the pending sentinel; propagate it as control flow (not a failure).
- vector/queue/procrastinate.py: BatchPending -> same-queue retry_in, exempt
from the transient cap (bounded by the processor's deadline).
- config + docs; tests across client/store/processor/strategy/parse-tier.
1653 unit tests pass; ruff + ty green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
79 lines
2.5 KiB
Python
79 lines
2.5 KiB
Python
"""Unit tests for the batch OCR job-tracking store (Deck #332).
|
|
|
|
Runs against a real temp-SQLite ``RefreshTokenStorage`` (its ``initialize()``
|
|
applies the migrations, incl. ``batch_ocr_jobs``).
|
|
"""
|
|
|
|
import tempfile
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
|
|
from nextcloud_mcp_server.vector.batch_ocr_store import BatchOcrJobStore
|
|
|
|
pytestmark = pytest.mark.unit
|
|
|
|
|
|
@pytest.fixture
|
|
async def store():
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
storage = RefreshTokenStorage(db_path=str(Path(tmp) / "batch.db"))
|
|
await storage.initialize()
|
|
yield BatchOcrJobStore(storage)
|
|
|
|
|
|
_DOC = dict(user_id="u1", doc_id="d1", doc_type="file", etag="v1")
|
|
|
|
|
|
async def test_get_missing_returns_none(store):
|
|
assert await store.get(**_DOC) is None
|
|
|
|
|
|
async def test_insert_then_get(store):
|
|
await store.insert_pending(**_DOC, job_id="mistral/j1")
|
|
job = await store.get(**_DOC)
|
|
assert job is not None
|
|
assert job.job_id == "mistral/j1"
|
|
assert job.status == "pending"
|
|
assert job.submitted_at > 0
|
|
|
|
|
|
async def test_insert_is_idempotent_on_conflict(store):
|
|
await store.insert_pending(**_DOC, job_id="mistral/j1", submitted_at=100)
|
|
# A racing re-submit must not overwrite the first row's job id.
|
|
await store.insert_pending(**_DOC, job_id="mistral/j2", submitted_at=200)
|
|
job = await store.get(**_DOC)
|
|
assert job.job_id == "mistral/j1"
|
|
assert job.submitted_at == 100
|
|
|
|
|
|
async def test_delete(store):
|
|
await store.insert_pending(**_DOC, job_id="mistral/j1")
|
|
await store.delete(**_DOC)
|
|
assert await store.get(**_DOC) is None
|
|
|
|
|
|
async def test_delete_stale_for_doc_keeps_current_etag(store):
|
|
await store.insert_pending(
|
|
user_id="u1", doc_id="d1", doc_type="file", etag="old", job_id="mistral/old"
|
|
)
|
|
await store.insert_pending(
|
|
user_id="u1", doc_id="d1", doc_type="file", etag="new", job_id="mistral/new"
|
|
)
|
|
await store.delete_stale_for_doc(
|
|
user_id="u1", doc_id="d1", doc_type="file", keep_etag="new"
|
|
)
|
|
# Old version row gone; current one kept.
|
|
assert (
|
|
await store.get(user_id="u1", doc_id="d1", doc_type="file", etag="old")
|
|
) is None
|
|
kept = await store.get(user_id="u1", doc_id="d1", doc_type="file", etag="new")
|
|
assert kept is not None and kept.job_id == "mistral/new"
|
|
|
|
|
|
async def test_rows_are_scoped_per_document(store):
|
|
await store.insert_pending(**_DOC, job_id="mistral/j1")
|
|
other = await store.get(user_id="u1", doc_id="d2", doc_type="file", etag="v1")
|
|
assert other is None # different doc_id
|