fix(ingest): zero queue-depth gauge on all-queues-drained (review round 4)

- metrics: update_ingest_queue_depth guarded on `not by_queue`, which conflated
  None (memory backend no-op) with {} (postgres, ALL queues drained). When every
  queue drains at once, get_ingest_job_counts_by_queue returns {} and the
  pre-zero loop was skipped, leaving a stale ghost backlog in the gauge. Guard on
  `by_queue is None` only; add an all-drained regression test.
- procrastinate: note that INGEST_TRANSIENT_MAX_ATTEMPTS is snapshotted at
  blueprint-build time (restart to pick up changes).

Deck #323.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-13 14:10:01 +02:00
co-authored by Claude Opus 4.8
parent 44f72839ed
commit ce53e21ead
3 changed files with 27 additions and 6 deletions
@@ -379,6 +379,9 @@ def _build_ingest_blueprint() -> Blueprint:
# BaseRetryStrategy subclass is the documented extension point (and is
# accepted at runtime by get_retry_strategy). The annotation is just too
# narrow, hence the ignore.
# Settings are snapshotted here at blueprint-build time (build_app, first
# use), so a restart is needed to pick up INGEST_TRANSIENT_MAX_ATTEMPTS
# changes -- intentional: the strategy lives for the App's lifetime.
retry=TieredEscalationStrategy(
max_transient_attempts=get_settings().ingest_transient_max_attempts
),