fix(ingest): address review round 1 (reclaim queue + tests)
- Register the periodic stalled-job reclaim on a dedicated ingest-maintenance queue that every worker drains (any --tier), so reclaim still fires when the fast fleet is scaled to zero and only ocr workers run. procrastinate's periodic-defer dedup keeps it single-run across drainers. - escalation: mark `unsupported`/`forced` reason labels as reserved (not raised). - processor: note that options/progress_callback are intentionally not threaded through _parse_pdf_tier yet (symmetric with the inline path). - tests: assert TieredEscalationStrategy backoff progression (4/8/16/…/300s); cover get_ingest_pending per-queue aggregation + the legacy job_counts fallback; add an external-path zero-page no-escalation case; use the canonical INGEST_QUEUE_FAST instead of the back-compat alias. Deck #323. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
9676bb3106
commit
35f8204a16
@@ -384,6 +384,7 @@ def worker(concurrency: int | None, tier: str | None):
|
||||
|
||||
from nextcloud_mcp_server.vector.queue.procrastinate import ( # noqa: PLC0415
|
||||
ALL_INGEST_QUEUES,
|
||||
INGEST_QUEUE_MAINTENANCE,
|
||||
LEGACY_INGEST_QUEUE,
|
||||
TIER_QUEUES,
|
||||
apply_ingest_queue_schema,
|
||||
@@ -392,11 +393,14 @@ def worker(concurrency: int | None, tier: str | None):
|
||||
|
||||
# Which queues this process drains. A single tier -> just its queue; no tier
|
||||
# -> every tier queue PLUS the legacy single queue, so a rolling upgrade
|
||||
# never strands jobs deferred under the pre-#323 name.
|
||||
# never strands jobs deferred under the pre-#323 name. Every worker also
|
||||
# drains the maintenance queue so the periodic stalled-job reclaim fires
|
||||
# regardless of which tier(s) are scaled up (procrastinate dedups the
|
||||
# periodic, so multiple drainers don't multiply the reclaim).
|
||||
if tier is not None:
|
||||
queues = [TIER_QUEUES[tier]]
|
||||
queues = [TIER_QUEUES[tier], INGEST_QUEUE_MAINTENANCE]
|
||||
else:
|
||||
queues = [*ALL_INGEST_QUEUES, LEGACY_INGEST_QUEUE]
|
||||
queues = [*ALL_INGEST_QUEUES, LEGACY_INGEST_QUEUE, INGEST_QUEUE_MAINTENANCE]
|
||||
|
||||
# This is the consumer side of the distributed (postgres) ingest backend.
|
||||
# Unlike the in-process anyio pool, the worker talks to procrastinate's App
|
||||
|
||||
Reference in New Issue
Block a user