fix: make procrastinate ingest queue opt-in (default to in-process anyio)

An unset INGEST_QUEUE auto-derived "postgres" whenever DATABASE_URL was
PostgreSQL, silently starting the procrastinate ingest worker (schema
migration, reclaim cron, deferred jobs) on every Postgres-backed tenant —
even though none had opted into the api/worker split. Observed on
tenant-blackbox-demo (:0.98.0): ~600 "Deferred 1 job" log lines / 24h.

Resolve an unset INGEST_QUEUE to "memory" (the in-process anyio queue)
regardless of the database backend. procrastinate is now strictly opt-in
via an explicit INGEST_QUEUE=postgres; the existing guard still rejects
postgres against a SQLite DATABASE_URL. Docs + unit test updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-04 01:59:24 +02:00
co-authored by Claude Opus 4.8
parent 1e615c2bf1
commit ad211ee2da
3 changed files with 35 additions and 15 deletions
+11 -8
View File
@@ -171,9 +171,10 @@ _DEFAULTS: dict[str, Any] = {
# the current monolithic behavior; self-hosters who set none are
# unaffected. See docs/architecture/mcp-decomposition.md (sibling repo).
"embedding_provider": "autodetect", # autodetect | gateway
# Ingest queue backend (Deck #183). None → auto: ``postgres`` (procrastinate)
# when DATABASE_URL is Postgres, else ``memory`` (the in-process anyio queue
# for SQLite/dev). procrastinate requires PostgreSQL.
# Ingest queue backend (Deck #183). None → ``memory`` (the in-process anyio
# queue): procrastinate is strictly opt-in, even on a Postgres DATABASE_URL.
# Set ``postgres`` explicitly to split ingest into a procrastinate worker;
# that requires a PostgreSQL DATABASE_URL.
"ingest_queue": None, # memory | postgres
# Process role for the per-tenant two-pod model (Deck #183). ``api`` runs the
# MCP/query server + scanner (defers jobs); the ``worker`` role is the
@@ -829,13 +830,15 @@ class Settings:
f"{_field.upper()} must be one of {sorted(_allowed)}; got {_val!r}"
)
# Ingest queue backend (Deck #183). Unset → auto-derive from the
# database backend: procrastinate needs PostgreSQL, so SQLite/dev falls
# back to the in-process anyio queue. An explicit ``postgres`` against a
# SQLite DATABASE_URL is a misconfiguration — fail loudly.
# Ingest queue backend (Deck #183). Procrastinate is opt-in: unset →
# ``memory`` (the in-process anyio queue) regardless of DB backend, so a
# Postgres DATABASE_URL alone never silently spins up a procrastinate
# worker. ``postgres`` must be set explicitly, and an explicit
# ``postgres`` against a SQLite DATABASE_URL is a misconfiguration —
# fail loudly below.
_queue = (self.ingest_queue or "").strip().lower()
if not _queue:
_queue = "memory" if is_sqlite_url(get_database_url()) else "postgres"
_queue = "memory"
if _queue not in {"memory", "postgres"}:
raise ValueError(
f"INGEST_QUEUE must be one of ['memory', 'postgres']; got {_queue!r}"