feat: add IngestTransport port for local/distributed ingest backends

Finish the hexagonal ports-&-adapters split started in #183. The producer side
already had a TaskProducer port + adapters, but the consumer side was
unabstracted and the INGEST_QUEUE selection leaked into a duplicated
`if use_postgres:` branch across both app.py lifespan paths.

Introduce an IngestTransport ABC (vector/queue/transport.py) that bundles the
producer with running (or not running) the in-process consumer pool, built by a
single build_transport() factory:

- LocalTransport (INGEST_QUEUE=memory): in-process anyio stream drained by an
  N-worker pool that run_consumers starts.
- DistributedTransport (INGEST_QUEUE=postgres): wraps ProcrastinateTaskProducer;
  run_consumers is a no-op because the consumer is the external `worker` role.

Both lifespan paths now call build_transport + _wire_vector_sync_state (new
helper that centralizes the app.state / module-singleton / browser-app writes) +
transport.run_consumers + transport.aclose(), with no INGEST_QUEUE branching and
no getattr drain probe. Adding a future backend (Redis/NATS/SQS) is one new
adapter + one build_transport arm, with no app.py or scanner change.

Preserves the single-tenant parallelism invariant (one shared multiplexed queue
+ N-worker pool, per-document not per-user dispatch) and documents it in
ADR-028. The worker CLI is unchanged (it is the external consumer).

Refs: Deck #196 (Deck #197 tracks the explicit parallelism regression test)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-04 19:43:31 +02:00
co-authored by Claude Opus 4.8
parent 55ea8dd358
commit e7bcdb1950
6 changed files with 593 additions and 142 deletions
+17 -1
View File
@@ -3,5 +3,21 @@
from .factory import build_producer
from .memory import MemoryTaskProducer
from .ports import TaskProducer
from .transport import (
DistributedTransport,
IngestTransport,
LocalTransport,
SpawnWorker,
build_transport,
)
__all__ = ["MemoryTaskProducer", "TaskProducer", "build_producer"]
__all__ = [
"DistributedTransport",
"IngestTransport",
"LocalTransport",
"MemoryTaskProducer",
"SpawnWorker",
"TaskProducer",
"build_producer",
"build_transport",
]