feat: add opt-in MCP decomposition hook points (design §10)
Adds the seven §10.2 hook-point modules + five env vars so Astrolabe Cloud can offload document processing to the external document-processor / embedding gateway. Purely additive: with every setting unset the server behaves exactly as today, so self-hosters are unaffected (Deck #92). Hook points (all default to current monolith behavior): - config: EMBEDDING_PROVIDER, INGEST_MODE, STATUS_BACKEND, COLLECTION_METADATA_SOURCE, FACT_EVENT_EMITTER (+ supporting settings), validated in Settings.__post_init__ (fail-fast STATUS_BACKEND=local with INGEST_MODE=external); shared canonical.py. - vector/payload_keys.py + acl_hash.py: cross-impl NAMESPACE/point_id (§2.2) and BLAKE2b-128 ACL hash (§11), pinned by fixtures shared with the document-processor repo. - embedding/gateway_client.py: OpenAI-compatible GatewayProvider authenticating via M2M OIDC client-credentials (separate realm); manual-only registry entry. - vector/collection_metadata.py: sentinel-point / API metadata source with env fallback. - vector/queue/: hexagonal ingest producer ports + memory/NATS adapters (Postgres seam); INGEST_MODE=external publishes mcp.ingest.requested.{tenant} instead of the in-memory stream and skips the in-process processor pool. The lifespan becomes a composition root across both deployment branches. - vector/queue/status.py: STATUS_BACKEND=bus subscriber feeding a StatusStore the vector-sync status endpoint reads. - admin/payload_backfill.py: POST /api/v1/admin/payload-backfill (admin scope); processor writes the new payload keys; query-side ACL pre-filter gated behind ACL_PREFILTER_ENABLED (default off). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
c7da612f20
commit
d883052fb8
@@ -0,0 +1,46 @@
|
||||
"""Composition root for the ingest producer (design §10).
|
||||
|
||||
``build_external_producer`` is called by the lifespan only when
|
||||
``INGEST_MODE=external``; local mode uses the in-memory stream directly (it
|
||||
already satisfies :class:`TaskProducer`). The transport under ``external`` is
|
||||
selected from the ``INGEST_BUS_URL`` scheme (``nats://`` now, ``postgres://``
|
||||
later) so moving the external processor to Postgres needs no new INGEST_MODE.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from ...config import Settings
|
||||
from .ports import TaskProducer
|
||||
|
||||
|
||||
def _transport_for(url: str) -> str:
|
||||
scheme = urlsplit(url).scheme.lower()
|
||||
if scheme.startswith("postgres"):
|
||||
return "postgres"
|
||||
return "nats"
|
||||
|
||||
|
||||
async def build_external_producer(settings: Settings) -> TaskProducer:
|
||||
"""Build the external-ingest producer for the configured transport.
|
||||
|
||||
Precondition: ``settings.ingest_mode == "external"`` (so __post_init__ has
|
||||
guaranteed ``ingest_bus_url`` and ``tenant_id`` are set).
|
||||
"""
|
||||
assert settings.ingest_bus_url is not None
|
||||
assert settings.tenant_id is not None
|
||||
|
||||
transport = _transport_for(settings.ingest_bus_url)
|
||||
if transport == "postgres":
|
||||
from .postgres import PostgresTaskProducer # noqa: PLC0415
|
||||
|
||||
return await PostgresTaskProducer.connect(settings)
|
||||
|
||||
from .nats import NatsTaskProducer # noqa: PLC0415
|
||||
|
||||
return await NatsTaskProducer.connect(
|
||||
url=settings.ingest_bus_url,
|
||||
tenant_id=settings.tenant_id,
|
||||
num_replicas=settings.ingest_bus_num_replicas,
|
||||
)
|
||||
Reference in New Issue
Block a user