feat: add opt-in MCP decomposition hook points (design §10)

Adds the seven §10.2 hook-point modules + five env vars so Astrolabe Cloud can
offload document processing to the external document-processor / embedding
gateway. Purely additive: with every setting unset the server behaves exactly
as today, so self-hosters are unaffected (Deck #92).

Hook points (all default to current monolith behavior):
- config: EMBEDDING_PROVIDER, INGEST_MODE, STATUS_BACKEND,
  COLLECTION_METADATA_SOURCE, FACT_EVENT_EMITTER (+ supporting settings),
  validated in Settings.__post_init__ (fail-fast STATUS_BACKEND=local with
  INGEST_MODE=external); shared canonical.py.
- vector/payload_keys.py + acl_hash.py: cross-impl NAMESPACE/point_id (§2.2)
  and BLAKE2b-128 ACL hash (§11), pinned by fixtures shared with the
  document-processor repo.
- embedding/gateway_client.py: OpenAI-compatible GatewayProvider authenticating
  via M2M OIDC client-credentials (separate realm); manual-only registry entry.
- vector/collection_metadata.py: sentinel-point / API metadata source with env
  fallback.
- vector/queue/: hexagonal ingest producer ports + memory/NATS adapters
  (Postgres seam); INGEST_MODE=external publishes mcp.ingest.requested.{tenant}
  instead of the in-memory stream and skips the in-process processor pool. The
  lifespan becomes a composition root across both deployment branches.
- vector/queue/status.py: STATUS_BACKEND=bus subscriber feeding a StatusStore
  the vector-sync status endpoint reads.
- admin/payload_backfill.py: POST /api/v1/admin/payload-backfill (admin scope);
  processor writes the new payload keys; query-side ACL pre-filter gated behind
  ACL_PREFILTER_ENABLED (default off).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-05-29 13:13:25 +02:00
co-authored by Claude Opus 4.8
parent c7da612f20
commit d883052fb8
37 changed files with 2534 additions and 55 deletions
@@ -41,8 +41,9 @@ def _warn_missing_secret_once() -> None:
async def handle_nextcloud_webhook(request: Request) -> JSONResponse:
"""Receive a Nextcloud webhook and queue a DocumentTask for vector sync.
Returns quickly so NC's webhook worker is not blocked. The send-stream is
read from ``request.app.state.document_send_stream``; when vector sync
Returns quickly so NC's webhook worker is not blocked. The task producer is
read from ``request.app.state.task_producer`` (the in-memory send stream in
local mode, or the NATS bus producer in external mode); when vector sync
isn't running we return 503 so NC retries delivery.
When ``WEBHOOK_SECRET`` is set, the request must carry
@@ -93,8 +94,8 @@ async def handle_nextcloud_webhook(request: Request) -> JSONResponse:
status_code=200,
)
send_stream = getattr(request.app.state, "document_send_stream", None)
if send_stream is None:
producer = getattr(request.app.state, "task_producer", None)
if producer is None:
logger.warning(
"Webhook received but vector sync is not running; rejecting so NC retries"
)
@@ -105,7 +106,7 @@ async def handle_nextcloud_webhook(request: Request) -> JSONResponse:
try:
with anyio.fail_after(1.0):
await send_stream.send(task)
await producer.send(task)
except TimeoutError:
# Queue is saturated (default 10 000 tasks). Returning 503 lets NC
# retry rather than pinning this handler until its outbound timeout