feat: add IngestTransport port for local/distributed ingest backends
Finish the hexagonal ports-&-adapters split started in #183. The producer side already had a TaskProducer port + adapters, but the consumer side was unabstracted and the INGEST_QUEUE selection leaked into a duplicated `if use_postgres:` branch across both app.py lifespan paths. Introduce an IngestTransport ABC (vector/queue/transport.py) that bundles the producer with running (or not running) the in-process consumer pool, built by a single build_transport() factory: - LocalTransport (INGEST_QUEUE=memory): in-process anyio stream drained by an N-worker pool that run_consumers starts. - DistributedTransport (INGEST_QUEUE=postgres): wraps ProcrastinateTaskProducer; run_consumers is a no-op because the consumer is the external `worker` role. Both lifespan paths now call build_transport + _wire_vector_sync_state (new helper that centralizes the app.state / module-singleton / browser-app writes) + transport.run_consumers + transport.aclose(), with no INGEST_QUEUE branching and no getattr drain probe. Adding a future backend (Redis/NATS/SQS) is one new adapter + one build_transport arm, with no app.py or scanner change. Preserves the single-tenant parallelism invariant (one shared multiplexed queue + N-worker pool, per-document not per-user dispatch) and documents it in ADR-028. The worker CLI is unchanged (it is the external consumer). Refs: Deck #196 (Deck #197 tracks the explicit parallelism regression test) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
55ea8dd358
commit
e7bcdb1950
@@ -3,5 +3,21 @@
|
||||
from .factory import build_producer
|
||||
from .memory import MemoryTaskProducer
|
||||
from .ports import TaskProducer
|
||||
from .transport import (
|
||||
DistributedTransport,
|
||||
IngestTransport,
|
||||
LocalTransport,
|
||||
SpawnWorker,
|
||||
build_transport,
|
||||
)
|
||||
|
||||
__all__ = ["MemoryTaskProducer", "TaskProducer", "build_producer"]
|
||||
__all__ = [
|
||||
"DistributedTransport",
|
||||
"IngestTransport",
|
||||
"LocalTransport",
|
||||
"MemoryTaskProducer",
|
||||
"SpawnWorker",
|
||||
"TaskProducer",
|
||||
"build_producer",
|
||||
"build_transport",
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user