feat: replace NATS ingest with procrastinate Postgres queue (#183)
Re-architect document ingest from the shared NATS-glued document-processor to a per-tenant, in-process model owned by nextcloud-mcp-server (Deck #183). The MCP server now owns both sides of ingest: - Producer (api role): the scanner defers one job per changed document into the app's Postgres via procrastinate (queueing_lock dedup; no execution lock, so a crashed worker can't deadlock a doc — Qdrant upserts are idempotent). - Consumer (worker role): `nextcloud-mcp-server worker` drains the queue and runs the existing process_document pipeline; a periodic task reclaims jobs orphaned in `doing` by a crash. INGEST_QUEUE selects the transport (auto: postgres when DATABASE_URL is Postgres, else the in-process anyio queue for SQLite/dev). procrastinate manages its own tables (applied on a fresh DB at startup and by `db upgrade`). The vector-sync status surface reads job counts from Postgres in postgres mode. procrastinate + psycopg3 ship in the [postgres] extra; the app's own engine still uses asyncpg (driver unification is a follow-up handled in the rendered Helm chart). NATS JetStream, the Postgres-queue stub, the bus status subscriber, and nats-py are removed. BREAKING CHANGE: the external-NATS-ingest env vars are removed (INGEST_MODE, STATUS_BACKEND, INGEST_BUS_URL, INGEST_BUS_NUM_REPLICAS, FACT_EVENT_EMITTER). Use INGEST_QUEUE (memory|postgres) and the `worker` command instead. TENANT_ID is retained (no longer NATS-subject-charset-validated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
b91af923d2
commit
21b7922bac
@@ -1,18 +1,18 @@
|
||||
"""Ingest-path ports (design §10, hexagonal).
|
||||
"""Ingest-path ports (design §10, hexagonal; Deck #183).
|
||||
|
||||
A ``TaskProducer`` is where the scanner + webhook receiver send a
|
||||
``DocumentTask``. The transport behind it is swappable:
|
||||
|
||||
- the in-process anyio ``MemoryObjectSendStream`` (local ingest — the default),
|
||||
- ``NatsTaskProducer`` (external ingest → the document-processor), and
|
||||
- a future Postgres-queue producer (seam only; the *external* processor owns the
|
||||
consume side — see ``postgres.py``).
|
||||
- ``MemoryTaskProducer`` over the in-process anyio ``MemoryObjectSendStream``
|
||||
(``INGEST_QUEUE=memory`` — the SQLite/dev default), and
|
||||
- ``ProcrastinateTaskProducer`` (``INGEST_QUEUE=postgres``), which defers jobs
|
||||
into the per-tenant Postgres for the out-of-process ``worker`` role to drain.
|
||||
|
||||
The protocol is exactly the surface the scanner/oauth_sync already use on the
|
||||
memory stream (``send`` + ``clone`` + ``async with``), so both adapters drop in
|
||||
with only a type-annotation change at the call sites. There is intentionally NO
|
||||
consumer port: the MCP server's only in-process consumer is the memory stream;
|
||||
when ingest is external the document-processor is the consumer, not this server.
|
||||
with only a type-annotation change at the call sites. There is no consumer port:
|
||||
in memory mode the in-process processor pool is the consumer; in postgres mode
|
||||
the procrastinate worker is.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
Reference in New Issue
Block a user