Chris CoutinhoandClaude Opus 4.8
21b7922bac
Re-architect document ingest from the shared NATS-glued document-processor to a per-tenant, in-process model owned by nextcloud-mcp-server (Deck #183). The MCP server now owns both sides of ingest: - Producer (api role): the scanner defers one job per changed document into the app's Postgres via procrastinate (queueing_lock dedup; no execution lock, so a crashed worker can't deadlock a doc — Qdrant upserts are idempotent). - Consumer (worker role): `nextcloud-mcp-server worker` drains the queue and runs the existing process_document pipeline; a periodic task reclaims jobs orphaned in `doing` by a crash. INGEST_QUEUE selects the transport (auto: postgres when DATABASE_URL is Postgres, else the in-process anyio queue for SQLite/dev). procrastinate manages its own tables (applied on a fresh DB at startup and by `db upgrade`). The vector-sync status surface reads job counts from Postgres in postgres mode. procrastinate + psycopg3 ship in the [postgres] extra; the app's own engine still uses asyncpg (driver unification is a follow-up handled in the rendered Helm chart). NATS JetStream, the Postgres-queue stub, the bus status subscriber, and nats-py are removed. BREAKING CHANGE: the external-NATS-ingest env vars are removed (INGEST_MODE, STATUS_BACKEND, INGEST_BUS_URL, INGEST_BUS_NUM_REPLICAS, FACT_EVENT_EMITTER). Use INGEST_QUEUE (memory|postgres) and the `worker` command instead. TENANT_ID is retained (no longer NATS-subject-charset-validated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
60 lines
2.2 KiB
Python
60 lines
2.2 KiB
Python
"""Ingest-path ports (design §10, hexagonal; Deck #183).
|
|
|
|
A ``TaskProducer`` is where the scanner + webhook receiver send a
|
|
``DocumentTask``. The transport behind it is swappable:
|
|
|
|
- ``MemoryTaskProducer`` over the in-process anyio ``MemoryObjectSendStream``
|
|
(``INGEST_QUEUE=memory`` — the SQLite/dev default), and
|
|
- ``ProcrastinateTaskProducer`` (``INGEST_QUEUE=postgres``), which defers jobs
|
|
into the per-tenant Postgres for the out-of-process ``worker`` role to drain.
|
|
|
|
The protocol is exactly the surface the scanner/oauth_sync already use on the
|
|
memory stream (``send`` + ``clone`` + ``async with``), so both adapters drop in
|
|
with only a type-annotation change at the call sites. There is no consumer port:
|
|
in memory mode the in-process processor pool is the consumer; in postgres mode
|
|
the procrastinate worker is.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from types import TracebackType
|
|
from typing import TYPE_CHECKING, Protocol, runtime_checkable
|
|
|
|
if TYPE_CHECKING:
|
|
from ..scanner import DocumentTask
|
|
|
|
|
|
@runtime_checkable
|
|
class TaskProducer(Protocol):
|
|
"""Sink for scanner/webhook ``DocumentTask``s (see module docstring)."""
|
|
|
|
# Positional-only so anyio's MemoryObjectSendStream.send(item) structurally
|
|
# satisfies this protocol (its parameter is named "item", not "task").
|
|
async def send(self, task: DocumentTask, /) -> None: ...
|
|
|
|
def clone(self) -> TaskProducer:
|
|
"""Return a producer handle for one user's scanner (multi-user mode).
|
|
|
|
For the memory stream this is a real clone (each closed independently);
|
|
for the bus it returns ``self`` (one shared connection).
|
|
"""
|
|
...
|
|
|
|
async def __aenter__(self) -> TaskProducer: ...
|
|
|
|
async def __aexit__(
|
|
self,
|
|
exc_type: type[BaseException] | None,
|
|
exc: BaseException | None,
|
|
tb: TracebackType | None,
|
|
) -> None: ...
|
|
|
|
async def aclose(self) -> None:
|
|
"""Close *this* handle (e.g. a per-user clone when its scanner exits).
|
|
|
|
For the memory stream this closes the clone; for the shared bus
|
|
connection it is a no-op (the connection is owned by the lifespan,
|
|
which drains it once on shutdown).
|
|
"""
|
|
...
|