feat: replace NATS ingest with procrastinate Postgres queue (#183)

Re-architect document ingest from the shared NATS-glued document-processor to a
per-tenant, in-process model owned by nextcloud-mcp-server (Deck #183). The MCP
server now owns both sides of ingest:

- Producer (api role): the scanner defers one job per changed document into the
  app's Postgres via procrastinate (queueing_lock dedup; no execution lock, so a
  crashed worker can't deadlock a doc — Qdrant upserts are idempotent).
- Consumer (worker role): `nextcloud-mcp-server worker` drains the queue and runs
  the existing process_document pipeline; a periodic task reclaims jobs orphaned
  in `doing` by a crash.

INGEST_QUEUE selects the transport (auto: postgres when DATABASE_URL is Postgres,
else the in-process anyio queue for SQLite/dev). procrastinate manages its own
tables (applied on a fresh DB at startup and by `db upgrade`). The vector-sync
status surface reads job counts from Postgres in postgres mode. procrastinate +
psycopg3 ship in the [postgres] extra; the app's own engine still uses asyncpg
(driver unification is a follow-up handled in the rendered Helm chart).

NATS JetStream, the Postgres-queue stub, the bus status subscriber, and nats-py
are removed.

BREAKING CHANGE: the external-NATS-ingest env vars are removed
(INGEST_MODE, STATUS_BACKEND, INGEST_BUS_URL, INGEST_BUS_NUM_REPLICAS,
FACT_EVENT_EMITTER). Use INGEST_QUEUE (memory|postgres) and the `worker`
command instead. TENANT_ID is retained (no longer NATS-subject-charset-validated).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-03 04:11:11 +02:00
co-authored by Claude Opus 4.8
parent b91af923d2
commit 21b7922bac
27 changed files with 1369 additions and 1120 deletions
+18 -18
View File
@@ -945,23 +945,21 @@ def configure_semantic_tools(mcp: FastMCP):
# missing attribute is a typo that should fail loudly. The
# value itself can legitimately be ``None`` before sync starts,
# which the check below handles.
# Outstanding-work view depends on the queue backend (Deck #183):
# memory → stream buffer depth; postgres → procrastinate job counts.
# Direct attribute access matches the eviction_task_group pattern at
# ``nc_semantic_search``: both AppContext and OAuthAppContext define
# these, so a missing attribute is a typo that should fail loudly.
from nextcloud_mcp_server.vector.ingest_status import ( # noqa: PLC0415
get_ingest_pending,
)
lifespan_ctx = ctx.request_context.lifespan_context
document_receive_stream = lifespan_ctx.document_receive_stream
if document_receive_stream is None:
logger.debug(
"document_receive_stream not available in lifespan context"
)
return VectorSyncStatusResponse(
indexed_count=0,
pending_count=0,
status="unknown",
enabled=True,
)
# Get pending count from stream statistics
stream_stats = document_receive_stream.statistics()
pending_count = stream_stats.current_buffer_used
pending = await get_ingest_pending(
task_producer=lifespan_ctx.task_producer,
document_receive_stream=lifespan_ctx.document_receive_stream,
ingest_queue=settings.ingest_queue,
)
# Get Qdrant client and query indexed count
indexed_count = 0
@@ -981,13 +979,15 @@ def configure_semantic_tools(mcp: FastMCP):
# Continue with indexed_count = 0
# Determine status
status = "syncing" if pending_count > 0 else "idle"
status = "syncing" if pending.pending > 0 else "idle"
return VectorSyncStatusResponse(
indexed_count=indexed_count,
pending_count=pending_count,
pending_count=pending.pending,
status=status,
enabled=True,
ingest_queue=settings.ingest_queue,
job_counts=pending.job_counts,
)
except Exception as e: