feat: replace NATS ingest with procrastinate Postgres queue (#183)

Re-architect document ingest from the shared NATS-glued document-processor to a
per-tenant, in-process model owned by nextcloud-mcp-server (Deck #183). The MCP
server now owns both sides of ingest:

- Producer (api role): the scanner defers one job per changed document into the
  app's Postgres via procrastinate (queueing_lock dedup; no execution lock, so a
  crashed worker can't deadlock a doc — Qdrant upserts are idempotent).
- Consumer (worker role): `nextcloud-mcp-server worker` drains the queue and runs
  the existing process_document pipeline; a periodic task reclaims jobs orphaned
  in `doing` by a crash.

INGEST_QUEUE selects the transport (auto: postgres when DATABASE_URL is Postgres,
else the in-process anyio queue for SQLite/dev). procrastinate manages its own
tables (applied on a fresh DB at startup and by `db upgrade`). The vector-sync
status surface reads job counts from Postgres in postgres mode. procrastinate +
psycopg3 ship in the [postgres] extra; the app's own engine still uses asyncpg
(driver unification is a follow-up handled in the rendered Helm chart).

NATS JetStream, the Postgres-queue stub, the bus status subscriber, and nats-py
are removed.

BREAKING CHANGE: the external-NATS-ingest env vars are removed
(INGEST_MODE, STATUS_BACKEND, INGEST_BUS_URL, INGEST_BUS_NUM_REPLICAS,
FACT_EVENT_EMITTER). Use INGEST_QUEUE (memory|postgres) and the `worker`
command instead. TENANT_ID is retained (no longer NATS-subject-charset-validated).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

nextcloud_mcp_server/api/management.py

@@ -293,52 +293,21 @@ async def get_vector_sync_status(request: Request) -> JSONResponse:
         )
 
     try:
-        # Bus status backend (INGEST_MODE=external): there is no in-process
-        # queue; pending/terminal state comes from the NATS status subscriber's
-        # store. indexed_documents stays the mode-independent Qdrant count.
-        if settings.status_backend == "bus":
-            store = getattr(request.app.state, "status_store", None)
-            indexed_count = 0
-            try:
-                qdrant_client = await get_qdrant_client()
-                count_result = await qdrant_client.count(
-                    collection_name=settings.get_collection_name(),
-                    count_filter=Filter(must=[get_placeholder_filter()]),
-                )
-                indexed_count = count_result.count
-            except Exception as e:
-                logger.warning("Failed to query Qdrant for indexed count: %s", e)
-            return JSONResponse(
-                {
-                    "status": "idle",
-                    "indexed_documents": indexed_count,
-                    "pending_documents": 0,
-                    "status_backend": "bus",
-                    "recent_states": store.counts() if store is not None else {},
-                }
-            )
-
-        # Get document receive stream from app state (set by starlette_lifespan in app.py)
-        document_receive_stream = getattr(
-            request.app.state, "document_receive_stream", None
+        # Outstanding-work view depends on the queue backend (Deck #183):
+        # memory → stream buffer depth; postgres → procrastinate job counts.
+        from nextcloud_mcp_server.vector.ingest_status import (  # noqa: PLC0415
+            get_ingest_pending,
         )
 
-        if document_receive_stream is None:
-            logger.debug("document_receive_stream not available in app state")
-            return JSONResponse(
-                {
-                    "status": "unknown",
-                    "indexed_documents": 0,
-                    "pending_documents": 0,
-                    "message": "Vector sync stream not initialized",
-                }
-            )
+        pending = await get_ingest_pending(
+            task_producer=getattr(request.app.state, "task_producer", None),
+            document_receive_stream=getattr(
+                request.app.state, "document_receive_stream", None
+            ),
+            ingest_queue=settings.ingest_queue,
+        )
 
-        # Get pending count from stream statistics
-        stream_stats = document_receive_stream.statistics()
-        pending_count = stream_stats.current_buffer_used
-
-        # Get Qdrant client and query indexed count
+        # Get Qdrant client and query indexed count (backend-independent)
         indexed_count = 0
         try:
             qdrant_client = await get_qdrant_client()
@@ -355,15 +324,18 @@ async def get_vector_sync_status(request: Request) -> JSONResponse:
             # Continue with indexed_count = 0
 
         # Determine status
-        status = "syncing" if pending_count > 0 else "idle"
+        status = "syncing" if pending.pending > 0 else "idle"
 
-        return JSONResponse(
-            {
-                "status": status,
-                "indexed_documents": indexed_count,
-                "pending_documents": pending_count,
-            }
-        )
+        body: dict[str, object] = {
+            "status": status,
+            "indexed_documents": indexed_count,
+            "pending_documents": pending.pending,
+            "ingest_queue": settings.ingest_queue,
+        }
+        if pending.job_counts is not None:
+            # Per-status breakdown (todo/doing/failed/…) on the postgres backend.
+            body["job_counts"] = pending.job_counts
+        return JSONResponse(body)
 
     except Exception as e:
         error_msg = _sanitize_error_for_client(e, "get_vector_sync_status")