feat: add IngestTransport port for local/distributed ingest backends

Finish the hexagonal ports-&-adapters split started in #183. The producer side
already had a TaskProducer port + adapters, but the consumer side was
unabstracted and the INGEST_QUEUE selection leaked into a duplicated
`if use_postgres:` branch across both app.py lifespan paths.

Introduce an IngestTransport ABC (vector/queue/transport.py) that bundles the
producer with running (or not running) the in-process consumer pool, built by a
single build_transport() factory:

- LocalTransport (INGEST_QUEUE=memory): in-process anyio stream drained by an
  N-worker pool that run_consumers starts.
- DistributedTransport (INGEST_QUEUE=postgres): wraps ProcrastinateTaskProducer;
  run_consumers is a no-op because the consumer is the external `worker` role.

Both lifespan paths now call build_transport + _wire_vector_sync_state (new
helper that centralizes the app.state / module-singleton / browser-app writes) +
transport.run_consumers + transport.aclose(), with no INGEST_QUEUE branching and
no getattr drain probe. Adding a future backend (Redis/NATS/SQS) is one new
adapter + one build_transport arm, with no app.py or scanner change.

Preserves the single-tenant parallelism invariant (one shared multiplexed queue
+ N-worker pool, per-document not per-user dispatch) and documents it in
ADR-028. The worker CLI is unchanged (it is the external consumer).

Refs: Deck #196 (Deck #197 tracks the explicit parallelism regression test)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs/ADR-028-ingest-transport-port.md

@@ -0,0 +1,124 @@
+# ADR-028: Ingest transport port (local anyio vs distributed procrastinate)
+
+## Status
+
+Accepted — 2026-06-04
+
+## Context
+
+Document ingest (scanner/webhook → fetch → chunk → embed → upsert to Qdrant)
+runs in one of two modes, selected by `INGEST_QUEUE`:
+
+- `memory` (the SQLite/dev default): an in-process anyio
+  `MemoryObjectStream` drained by a pool of in-process worker tasks.
+- `postgres`: jobs are deferred into the per-tenant Postgres via
+  [procrastinate](https://procrastinate.readthedocs.io/) and drained by a
+  *separate* `nextcloud-mcp-server worker` process (the scale-to-zero
+  api/worker split — KEDA scales the worker Deployment on queue depth).
+
+ADR-007 introduced the in-process model; Deck #183 / PR #836 added the
+postgres backend and a `TaskProducer` **Protocol** (`vector/queue/ports.py`)
+so the scanner and webhook receiver send a `DocumentTask` to a backend-agnostic
+sink — `MemoryTaskProducer` (anyio) or `ProcrastinateTaskProducer` (Postgres).
+
+That abstracted the **producer** side, but left two gaps:
+
+1. **The consumer side was not abstracted.** Memory mode spun up an in-process
+   pool inside the server lifespan; postgres mode relied on the external worker
+   CLI. Nothing tied these together.
+2. **Mode selection leaked into the lifespan.** `app.py` carried a duplicated
+   `if use_postgres: build producer + ensure_schema … else: create stream …`
+   branch, plus a conditional N-worker startup and a
+   `getattr(task_producer, "drain", None)` shutdown probe — repeated across the
+   two near-identical lifespan paths (single-user BasicAuth and multi-user
+   OAuth/BasicAuth). Adding a third backend (Redis/NATS/SQS) would have meant
+   editing both blocks.
+
+## Decision
+
+Introduce an `IngestTransport` abstraction that owns **both** sides of one
+ingest backend — the producer to wire into `app.state`/the scanner, and how (or
+whether) the in-process consumer pool runs — built by a single
+`build_transport(settings)` factory.
+
+```
+build_transport(settings) ->
+    INGEST_QUEUE=postgres -> DistributedTransport(build_producer(settings))
+    INGEST_QUEUE=memory   -> LocalTransport(vector_sync_queue_max_size)
+```
+
+`IngestTransport` (`vector/queue/transport.py`) exposes:
+
+- `producer` — the `TaskProducer` to wire into `app.state` / hand to the scanner.
+- `send_stream` / `receive_stream` — the raw anyio stream ends in memory mode,
+  `None` for distributed backends (the latter keeps `ingest_status` queue-depth
+  and the integration conftest's stream-singleton handling working unchanged).
+- `run_consumers(task_group, spawn_worker, count)` — start the in-process pool;
+  a **no-op** for distributed backends, whose consumer is the external worker.
+- `aclose()` — tear down backend-owned resources once on shutdown (closes the
+  procrastinate connector pool; a no-op for the memory stream, which task-group
+  cancellation closes).
+
+The lifespan supplies a `spawn_worker` closure so the transport never learns
+about auth modes — the single-user closure binds a shared `nc_client`+username,
+the multi-user closure binds the Nextcloud host for per-document credential
+resolution. Both forward anyio's injected `task_status` so `tg.start` observes
+each worker's readiness.
+
+### Why an ABC for the transport but a Protocol for the producer
+
+`TaskProducer` is a `Protocol` specifically so anyio's third-party
+`MemoryObjectSendStream` satisfies it structurally. The transport has exactly
+two in-house adapters that share the `producer` storage and the
+`receive_stream`/`run_consumers`/`aclose` defaults, so a concrete `abc.ABC` is
+simpler, gives shared default implementations, and checks more cleanly under
+`ty`. We keep `TaskProducer`/`build_producer` unchanged; the transport *wraps* a
+producer.
+
+### No consumer port
+
+Deliberately, there is no consumer *port* (mirroring `ports.py`): in memory mode
+the in-process pool is the consumer; in postgres mode the external worker is.
+The worker CLI (`cli.py worker`) talks to procrastinate's `App` directly
+(`run_worker_async`) — a different control surface from the in-process pool — so
+it does not route through `IngestTransport`; `DistributedTransport.run_consumers`
+is a no-op precisely because that separate process is the consumer.
+
+### Single-tenant parallelism invariant
+
+A single tenant must process its users' files **in parallel**, never one user
+fully then the next. This holds by construction and is documented here as a
+contract:
+
+- **Local backend:** `LocalTransport.run_consumers` hands each of N workers
+  (`VECTOR_SYNC_PROCESSOR_WORKERS`, default 3) an independent `clone()` of *one*
+  shared receive stream. All users' `DocumentTask`s are multiplexed onto that
+  single queue and dispatched **per-document**, so N documents — from any mix of
+  users — are in flight at once.
+- **Distributed backend:** the worker runs `run_worker_async(concurrency=N)`
+  (default N = `VECTOR_SYNC_PROCESSOR_WORKERS`) over the single `ingest` queue,
+  and procrastinate's only lock is a per-**document** `queueing_lock`
+  (`user_id:doc_type:doc_id`) — there is no per-user lock — so different users'
+  jobs run concurrently across worker slots and pods.
+
+An explicit anyio overlap test for this invariant is tracked as a follow-up
+(Deck #197).
+
+## Consequences
+
+- The two `app.py` lifespan paths are backend-agnostic: `build_transport()` +
+  `_wire_vector_sync_state()` + `transport.run_consumers()` +
+  `transport.aclose()`, with no `INGEST_QUEUE` branching and no `getattr` drain
+  probe.
+- Adding a future queue backend is one new `IngestTransport` adapter + one
+  `build_transport` arm — no change to `app.py`, the scanner, or the webhook
+  receiver.
+- `app.state.task_producer` / `_vector_sync_state.task_producer` and the queue
+  depth surface (`ingest_status.py`) keep their existing contracts.
+
+## References
+
+- ADR-007 — Background vector database synchronization (in-process anyio model)
+- ADR-010 — Webhook-based vector database synchronization
+- Deck #183 / PR #836 — procrastinate Postgres ingest queue + `TaskProducer` port
+- Deck #196 — this work; Deck #197 — explicit parallelism test follow-up