fix(ingest): address review round 1 (reclaim queue + tests)
- Register the periodic stalled-job reclaim on a dedicated ingest-maintenance queue that every worker drains (any --tier), so reclaim still fires when the fast fleet is scaled to zero and only ocr workers run. procrastinate's periodic-defer dedup keeps it single-run across drainers. - escalation: mark `unsupported`/`forced` reason labels as reserved (not raised). - processor: note that options/progress_callback are intentionally not threaded through _parse_pdf_tier yet (symmetric with the inline path). - tests: assert TieredEscalationStrategy backoff progression (4/8/16/…/300s); cover get_ingest_pending per-queue aggregation + the legacy job_counts fallback; add an external-path zero-page no-escalation case; use the canonical INGEST_QUEUE_FAST instead of the back-compat alias. Deck #323. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
9676bb3106
commit
35f8204a16
@@ -87,6 +87,14 @@ LEGACY_INGEST_QUEUE = "ingest"
|
||||
# Back-compat alias for callers that imported the old single-queue constant.
|
||||
INGEST_QUEUE_NAME = DEFAULT_INGEST_QUEUE
|
||||
|
||||
# Maintenance queue carrying ONLY the periodic stalled-job reclaim (no document
|
||||
# jobs). Every worker drains it regardless of --tier, so the reclaim fires even
|
||||
# in an asymmetric deployment where the fast fleet is scaled to zero and only
|
||||
# ocr workers run. procrastinate's periodic-defer dedup ensures exactly one
|
||||
# worker runs each tick even when many drain this queue. Kept off document
|
||||
# queues so tier isolation (which fleet processes which docs) is preserved.
|
||||
INGEST_QUEUE_MAINTENANCE = "ingest-maintenance"
|
||||
|
||||
# Queues the job-count + reclaim helpers sweep (tier queues + the legacy one).
|
||||
_MANAGED_QUEUES: tuple[str, ...] = (*ALL_INGEST_QUEUES, LEGACY_INGEST_QUEUE)
|
||||
|
||||
@@ -368,8 +376,12 @@ def _build_ingest_blueprint() -> Blueprint:
|
||||
max_transient_attempts=get_settings().ingest_transient_max_attempts
|
||||
),
|
||||
)(process_document_task)
|
||||
# Reclaim runs on the dedicated maintenance queue (every worker drains it),
|
||||
# not a tier queue -- otherwise an ocr-only deployment (fast scaled to zero)
|
||||
# would never fire the periodic and orphaned ``doing`` jobs would never be
|
||||
# reclaimed. The task itself sweeps ALL queues (get_stalled_jobs(queue=None)).
|
||||
reclaim = bp.task(
|
||||
name="reclaim_stalled_jobs", queue=DEFAULT_INGEST_QUEUE, pass_context=True
|
||||
name="reclaim_stalled_jobs", queue=INGEST_QUEUE_MAINTENANCE, pass_context=True
|
||||
)(reclaim_stalled_ingest_jobs)
|
||||
bp.periodic(cron="*/5 * * * *", periodic_id="reclaim_stalled_ingest")(reclaim)
|
||||
return bp
|
||||
|
||||
Reference in New Issue
Block a user