feat: replace NATS ingest with procrastinate Postgres queue (#183)
Re-architect document ingest from the shared NATS-glued document-processor to a per-tenant, in-process model owned by nextcloud-mcp-server (Deck #183). The MCP server now owns both sides of ingest: - Producer (api role): the scanner defers one job per changed document into the app's Postgres via procrastinate (queueing_lock dedup; no execution lock, so a crashed worker can't deadlock a doc — Qdrant upserts are idempotent). - Consumer (worker role): `nextcloud-mcp-server worker` drains the queue and runs the existing process_document pipeline; a periodic task reclaims jobs orphaned in `doing` by a crash. INGEST_QUEUE selects the transport (auto: postgres when DATABASE_URL is Postgres, else the in-process anyio queue for SQLite/dev). procrastinate manages its own tables (applied on a fresh DB at startup and by `db upgrade`). The vector-sync status surface reads job counts from Postgres in postgres mode. procrastinate + psycopg3 ship in the [postgres] extra; the app's own engine still uses asyncpg (driver unification is a follow-up handled in the rendered Helm chart). NATS JetStream, the Postgres-queue stub, the bus status subscriber, and nats-py are removed. BREAKING CHANGE: the external-NATS-ingest env vars are removed (INGEST_MODE, STATUS_BACKEND, INGEST_BUS_URL, INGEST_BUS_NUM_REPLICAS, FACT_EVENT_EMITTER). Use INGEST_QUEUE (memory|postgres) and the `worker` command instead. TENANT_ID is retained (no longer NATS-subject-charset-validated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
b91af923d2
commit
21b7922bac
@@ -154,7 +154,9 @@ async def processor_task(
|
||||
logger.info("Processor %s stopped", worker_id)
|
||||
|
||||
|
||||
async def process_document(doc_task: DocumentTask, nc_client: NextcloudClient):
|
||||
async def process_document(
|
||||
doc_task: DocumentTask, nc_client: NextcloudClient, *, max_retries: int = 3
|
||||
):
|
||||
"""
|
||||
Process a single document: fetch, tokenize, embed, store in Qdrant.
|
||||
|
||||
@@ -163,6 +165,10 @@ async def process_document(doc_task: DocumentTask, nc_client: NextcloudClient):
|
||||
Args:
|
||||
doc_task: Document task to process
|
||||
nc_client: Authenticated Nextcloud client
|
||||
max_retries: In-process indexing attempts before re-raising. The default
|
||||
(3) suits the in-process SQLite pool, which has no durable retry. The
|
||||
procrastinate worker passes ``1`` so durable retry is owned by the
|
||||
queue (and survives worker crashes), avoiding compounding 3×N retries.
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
@@ -230,7 +236,6 @@ async def process_document(doc_task: DocumentTask, nc_client: NextcloudClient):
|
||||
return
|
||||
|
||||
# Handle indexing with retry
|
||||
max_retries = 3
|
||||
retry_delay = 1.0
|
||||
|
||||
for attempt in range(max_retries):
|
||||
|
||||
Reference in New Issue
Block a user