Chris CoutinhoandClaude Opus 4.8
6c7b679b52
LocalTransport.aclose() (added in round 3) closes its owned stream ends; the ADR still described aclose() as a no-op for the memory stream. Update the prose to match the shipped behaviour. Doc-only. Refs: Deck #196 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
127 lines
6.2 KiB
Markdown
127 lines
6.2 KiB
Markdown
# ADR-028: Ingest transport port (local anyio vs distributed procrastinate)
|
|
|
|
## Status
|
|
|
|
Accepted — 2026-06-04
|
|
|
|
## Context
|
|
|
|
Document ingest (scanner/webhook → fetch → chunk → embed → upsert to Qdrant)
|
|
runs in one of two modes, selected by `INGEST_QUEUE`:
|
|
|
|
- `memory` (the SQLite/dev default): an in-process anyio
|
|
`MemoryObjectStream` drained by a pool of in-process worker tasks.
|
|
- `postgres`: jobs are deferred into the per-tenant Postgres via
|
|
[procrastinate](https://procrastinate.readthedocs.io/) and drained by a
|
|
*separate* `nextcloud-mcp-server worker` process (the scale-to-zero
|
|
api/worker split — KEDA scales the worker Deployment on queue depth).
|
|
|
|
ADR-007 introduced the in-process model; Deck #183 / PR #836 added the
|
|
postgres backend and a `TaskProducer` **Protocol** (`vector/queue/ports.py`)
|
|
so the scanner and webhook receiver send a `DocumentTask` to a backend-agnostic
|
|
sink — `MemoryTaskProducer` (anyio) or `ProcrastinateTaskProducer` (Postgres).
|
|
|
|
That abstracted the **producer** side, but left two gaps:
|
|
|
|
1. **The consumer side was not abstracted.** Memory mode spun up an in-process
|
|
pool inside the server lifespan; postgres mode relied on the external worker
|
|
CLI. Nothing tied these together.
|
|
2. **Mode selection leaked into the lifespan.** `app.py` carried a duplicated
|
|
`if use_postgres: build producer + ensure_schema … else: create stream …`
|
|
branch, plus a conditional N-worker startup and a
|
|
`getattr(task_producer, "drain", None)` shutdown probe — repeated across the
|
|
two near-identical lifespan paths (single-user BasicAuth and multi-user
|
|
OAuth/BasicAuth). Adding a third backend (Redis/NATS/SQS) would have meant
|
|
editing both blocks.
|
|
|
|
## Decision
|
|
|
|
Introduce an `IngestTransport` abstraction that owns **both** sides of one
|
|
ingest backend — the producer to wire into `app.state`/the scanner, and how (or
|
|
whether) the in-process consumer pool runs — built by a single
|
|
`build_transport(settings)` factory.
|
|
|
|
```
|
|
build_transport(settings) ->
|
|
INGEST_QUEUE=postgres -> DistributedTransport(build_producer(settings))
|
|
INGEST_QUEUE=memory -> LocalTransport(vector_sync_queue_max_size)
|
|
```
|
|
|
|
`IngestTransport` (`vector/queue/transport.py`) exposes:
|
|
|
|
- `producer` — the `TaskProducer` to wire into `app.state` / hand to the scanner.
|
|
- `send_stream` / `receive_stream` — the raw anyio stream ends in memory mode,
|
|
`None` for distributed backends (the latter keeps `ingest_status` queue-depth
|
|
and the integration conftest's stream-singleton handling working unchanged).
|
|
- `run_consumers(task_group, spawn_worker, count)` — start the in-process pool;
|
|
a **no-op** for distributed backends, whose consumer is the external worker.
|
|
- `aclose()` — tear down backend-owned resources once on shutdown.
|
|
`DistributedTransport` closes the procrastinate connector pool;
|
|
`LocalTransport` closes its owned send/receive stream ends (belt-and-suspenders
|
|
— task-group cancellation already closes the per-worker clones, and anyio
|
|
`aclose` is idempotent). The base default is a no-op.
|
|
|
|
The lifespan supplies a `spawn_worker` closure so the transport never learns
|
|
about auth modes — the single-user closure binds a shared `nc_client`+username,
|
|
the multi-user closure binds the Nextcloud host for per-document credential
|
|
resolution. Both forward anyio's injected `task_status` so `tg.start` observes
|
|
each worker's readiness.
|
|
|
|
### Why an ABC for the transport but a Protocol for the producer
|
|
|
|
`TaskProducer` is a `Protocol` specifically so anyio's third-party
|
|
`MemoryObjectSendStream` satisfies it structurally. The transport has exactly
|
|
two in-house adapters that share the `producer` storage and the
|
|
`receive_stream`/`run_consumers`/`aclose` defaults, so a concrete `abc.ABC` is
|
|
simpler, gives shared default implementations, and checks more cleanly under
|
|
`ty`. We keep `TaskProducer`/`build_producer` unchanged; the transport *wraps* a
|
|
producer.
|
|
|
|
### No consumer port
|
|
|
|
Deliberately, there is no consumer *port* (mirroring `ports.py`): in memory mode
|
|
the in-process pool is the consumer; in postgres mode the external worker is.
|
|
The worker CLI (`cli.py worker`) talks to procrastinate's `App` directly
|
|
(`run_worker_async`) — a different control surface from the in-process pool — so
|
|
it does not route through `IngestTransport`; `DistributedTransport.run_consumers`
|
|
is a no-op precisely because that separate process is the consumer.
|
|
|
|
### Single-tenant parallelism invariant
|
|
|
|
A single tenant must process its users' files **in parallel**, never one user
|
|
fully then the next. This holds by construction and is documented here as a
|
|
contract:
|
|
|
|
- **Local backend:** `LocalTransport.run_consumers` hands each of N workers
|
|
(`VECTOR_SYNC_PROCESSOR_WORKERS`, default 3) an independent `clone()` of *one*
|
|
shared receive stream. All users' `DocumentTask`s are multiplexed onto that
|
|
single queue and dispatched **per-document**, so N documents — from any mix of
|
|
users — are in flight at once.
|
|
- **Distributed backend:** the worker runs `run_worker_async(concurrency=N)`
|
|
(default N = `VECTOR_SYNC_PROCESSOR_WORKERS`) over the single `ingest` queue,
|
|
and procrastinate's only lock is a per-**document** `queueing_lock`
|
|
(`user_id:doc_type:doc_id`) — there is no per-user lock — so different users'
|
|
jobs run concurrently across worker slots and pods.
|
|
|
|
An explicit anyio overlap test for this invariant is tracked as a follow-up
|
|
(Deck #197).
|
|
|
|
## Consequences
|
|
|
|
- The two `app.py` lifespan paths are backend-agnostic: `build_transport()` +
|
|
`_wire_vector_sync_state()` + `transport.run_consumers()` +
|
|
`transport.aclose()`, with no `INGEST_QUEUE` branching and no `getattr` drain
|
|
probe.
|
|
- Adding a future queue backend is one new `IngestTransport` adapter + one
|
|
`build_transport` arm — no change to `app.py`, the scanner, or the webhook
|
|
receiver.
|
|
- `app.state.task_producer` / `_vector_sync_state.task_producer` and the queue
|
|
depth surface (`ingest_status.py`) keep their existing contracts.
|
|
|
|
## References
|
|
|
|
- ADR-007 — Background vector database synchronization (in-process anyio model)
|
|
- ADR-010 — Webhook-based vector database synchronization
|
|
- Deck #183 / PR #836 — procrastinate Postgres ingest queue + `TaskProducer` port
|
|
- Deck #196 — this work; Deck #197 — explicit parallelism test follow-up
|