feat: replace NATS ingest with procrastinate Postgres queue (#183)

Re-architect document ingest from the shared NATS-glued document-processor to a
per-tenant, in-process model owned by nextcloud-mcp-server (Deck #183). The MCP
server now owns both sides of ingest:

- Producer (api role): the scanner defers one job per changed document into the
  app's Postgres via procrastinate (queueing_lock dedup; no execution lock, so a
  crashed worker can't deadlock a doc — Qdrant upserts are idempotent).
- Consumer (worker role): `nextcloud-mcp-server worker` drains the queue and runs
  the existing process_document pipeline; a periodic task reclaims jobs orphaned
  in `doing` by a crash.

INGEST_QUEUE selects the transport (auto: postgres when DATABASE_URL is Postgres,
else the in-process anyio queue for SQLite/dev). procrastinate manages its own
tables (applied on a fresh DB at startup and by `db upgrade`). The vector-sync
status surface reads job counts from Postgres in postgres mode. procrastinate +
psycopg3 ship in the [postgres] extra; the app's own engine still uses asyncpg
(driver unification is a follow-up handled in the rendered Helm chart).

NATS JetStream, the Postgres-queue stub, the bus status subscriber, and nats-py
are removed.

BREAKING CHANGE: the external-NATS-ingest env vars are removed
(INGEST_MODE, STATUS_BACKEND, INGEST_BUS_URL, INGEST_BUS_NUM_REPLICAS,
FACT_EVENT_EMITTER). Use INGEST_QUEUE (memory|postgres) and the `worker`
command instead. TENANT_ID is retained (no longer NATS-subject-charset-validated).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs/configuration.md

@@ -785,9 +785,9 @@ docker-compose up
 
 ## Decomposition Hook Points (Optional, Advanced)
 
-The server can optionally offload document processing and embeddings to external
-services (the Astrolabe Cloud document-processor and embedding-gateway). These
-are **opt-in**; every default reproduces the in-process monolith behavior, so
+The server can optionally offload embeddings to an external gateway and split
+ingest into a separate scale-to-zero worker process (Deck #183). These are
+**opt-in**; every default reproduces the in-process monolith behavior, so
 self-hosters can ignore this section.
 
 ```bash
@@ -799,21 +799,41 @@ EMBEDDING_GATEWAY_TOKEN_URL=...
 EMBEDDING_GATEWAY_CLIENT_ID=...
 EMBEDDING_GATEWAY_CLIENT_SECRET=...
 
-# External ingest: publish to NATS instead of the in-process processor pool
-INGEST_MODE=external          # local (default) | external
-STATUS_BACKEND=bus            # local (default) | bus — REQUIRED with external
-INGEST_BUS_URL=nats://nats:4222
-TENANT_ID=<uuid>              # NATS per-tenant subject token
+# Ingest queue backend. Default (unset) auto-derives from DATABASE_URL:
+#   - PostgreSQL DATABASE_URL → "postgres" (the procrastinate queue)
+#   - SQLite / unset          → "memory"   (the in-process anyio queue)
+INGEST_QUEUE=postgres         # memory | postgres
+# Process role (informational; the worker is launched via the `worker` command):
+MCP_ROLE=all                  # api | worker | all (default)
+TENANT_ID=<uuid>              # per-tenant identity (used in collection naming)
+```
+
+### Postgres ingest queue + worker (api/worker split)
+
+When `INGEST_QUEUE=postgres` (a PostgreSQL `DATABASE_URL`), the scanner **defers**
+one job per changed document into the app's Postgres via
+[procrastinate](https://procrastinate.readthedocs.io); a separate **worker**
+process drains the queue (fetch → chunk → embed → upsert Qdrant). Run the two
+roles as separate Deployments from the same image:
+
+```bash
+# API pod (always-on): serves MCP/query + runs the scanner (defers jobs)
+nextcloud-mcp-server run
+
+# Ingest worker (scale-to-zero on queue depth via KEDA): drains the queue
+nextcloud-mcp-server worker -c 4
 ```
 
 Notes:
-- `STATUS_BACKEND=local` with `INGEST_MODE=external` is rejected at startup
-  (the in-process job state is empty for externally-dispatched work).
-- **`nats-py` ships as a core dependency** (small, pure-Python) and is imported
-  lazily — only when `INGEST_MODE=external`. Self-hosters who never enable
-  external ingest pay no runtime cost.
-- `INGEST_MODE=external` + `STATUS_BACKEND=bus` opens **two** NATS connections
-  per pod (the ingest producer and the status subscriber are separate roles).
+
+- **procrastinate manages its own tables** (`procrastinate_jobs`, …) in the same
+  database. They are created on a fresh DB by the API pod at startup and by
+  `nextcloud-mcp-server db upgrade` — a migration lineage independent of the
+  app's Alembic schema. procrastinate is Postgres-only (psycopg3); it ships in
+  the `[postgres]` extra and is imported lazily.
+- KEDA scales the worker on
+  `SELECT count(*) FROM procrastinate_jobs WHERE queue_name='ingest' AND status='todo'`.
+- `INGEST_QUEUE=postgres` with a SQLite `DATABASE_URL` is rejected at startup.
 
 ---