Chris CoutinhoandClaude Opus 4.8
21b7922bac
Re-architect document ingest from the shared NATS-glued document-processor to a per-tenant, in-process model owned by nextcloud-mcp-server (Deck #183). The MCP server now owns both sides of ingest: - Producer (api role): the scanner defers one job per changed document into the app's Postgres via procrastinate (queueing_lock dedup; no execution lock, so a crashed worker can't deadlock a doc — Qdrant upserts are idempotent). - Consumer (worker role): `nextcloud-mcp-server worker` drains the queue and runs the existing process_document pipeline; a periodic task reclaims jobs orphaned in `doing` by a crash. INGEST_QUEUE selects the transport (auto: postgres when DATABASE_URL is Postgres, else the in-process anyio queue for SQLite/dev). procrastinate manages its own tables (applied on a fresh DB at startup and by `db upgrade`). The vector-sync status surface reads job counts from Postgres in postgres mode. procrastinate + psycopg3 ship in the [postgres] extra; the app's own engine still uses asyncpg (driver unification is a follow-up handled in the rendered Helm chart). NATS JetStream, the Postgres-queue stub, the bus status subscriber, and nats-py are removed. BREAKING CHANGE: the external-NATS-ingest env vars are removed (INGEST_MODE, STATUS_BACKEND, INGEST_BUS_URL, INGEST_BUS_NUM_REPLICAS, FACT_EVENT_EMITTER). Use INGEST_QUEUE (memory|postgres) and the `worker` command instead. TENANT_ID is retained (no longer NATS-subject-charset-validated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
38 lines
1.3 KiB
Python
38 lines
1.3 KiB
Python
"""Composition root for the ingest producer (Deck #183).
|
|
|
|
The transport is selected from ``INGEST_QUEUE``:
|
|
|
|
- ``postgres`` → :class:`ProcrastinateTaskProducer`, which defers jobs into the
|
|
per-tenant Postgres for the out-of-process ``worker`` role to drain.
|
|
- ``memory`` (SQLite/dev default) → the in-process anyio stream, built inline by
|
|
the server lifespan (it owns both the send and receive ends), so it is not
|
|
produced here.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
|
|
from ...config import Settings
|
|
from .ports import TaskProducer
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
async def build_producer(settings: Settings) -> TaskProducer:
|
|
"""Build the Postgres (procrastinate) ingest producer.
|
|
|
|
Precondition: ``settings.ingest_queue == "postgres"`` (the memory transport
|
|
is constructed inline by the lifespan because it needs the paired receive
|
|
stream for the in-process processor pool).
|
|
"""
|
|
if settings.ingest_queue != "postgres":
|
|
raise ValueError(
|
|
"build_producer is only for INGEST_QUEUE=postgres; the memory "
|
|
f"transport is built inline by the lifespan (got {settings.ingest_queue!r})"
|
|
)
|
|
|
|
from .procrastinate import ProcrastinateTaskProducer # noqa: PLC0415
|
|
|
|
return await ProcrastinateTaskProducer.connect()
|