feat: add opt-in MCP decomposition hook points (design §10)

Adds the seven §10.2 hook-point modules + five env vars so Astrolabe Cloud can
offload document processing to the external document-processor / embedding
gateway. Purely additive: with every setting unset the server behaves exactly
as today, so self-hosters are unaffected (Deck #92).

Hook points (all default to current monolith behavior):
- config: EMBEDDING_PROVIDER, INGEST_MODE, STATUS_BACKEND,
  COLLECTION_METADATA_SOURCE, FACT_EVENT_EMITTER (+ supporting settings),
  validated in Settings.__post_init__ (fail-fast STATUS_BACKEND=local with
  INGEST_MODE=external); shared canonical.py.
- vector/payload_keys.py + acl_hash.py: cross-impl NAMESPACE/point_id (§2.2)
  and BLAKE2b-128 ACL hash (§11), pinned by fixtures shared with the
  document-processor repo.
- embedding/gateway_client.py: OpenAI-compatible GatewayProvider authenticating
  via M2M OIDC client-credentials (separate realm); manual-only registry entry.
- vector/collection_metadata.py: sentinel-point / API metadata source with env
  fallback.
- vector/queue/: hexagonal ingest producer ports + memory/NATS adapters
  (Postgres seam); INGEST_MODE=external publishes mcp.ingest.requested.{tenant}
  instead of the in-memory stream and skips the in-process processor pool. The
  lifespan becomes a composition root across both deployment branches.
- vector/queue/status.py: STATUS_BACKEND=bus subscriber feeding a StatusStore
  the vector-sync status endpoint reads.
- admin/payload_backfill.py: POST /api/v1/admin/payload-backfill (admin scope);
  processor writes the new payload keys; query-side ACL pre-filter gated behind
  ACL_PREFILTER_ENABLED (default off).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-05-29 13:13:25 +02:00
co-authored by Claude Opus 4.8
parent c7da612f20
commit d883052fb8
37 changed files with 2534 additions and 55 deletions
+44
View File
@@ -122,6 +122,25 @@ async def validate_token_and_get_user(
return user_id, validated
class AdminScopeRequired(Exception):
"""Raised when an authenticated caller lacks the ``admin`` scope."""
async def require_admin_scope(request: Request) -> str:
"""Authenticate the caller and require the ``admin`` scope.
This is the first ``/api/v1/admin/*`` guard; it establishes the pattern
(auth via :func:`validate_token_and_get_user`, then an explicit scope check).
Raises ``ValueError`` on auth failure and :class:`AdminScopeRequired` on a
valid token that lacks ``admin``; callers map these to 401 / 403.
"""
user_id, validated = await validate_token_and_get_user(request)
scopes = validated.get("scopes") or []
if "admin" not in scopes:
raise AdminScopeRequired("admin scope required")
return user_id
def _sanitize_error_for_client(error: Exception, context: str = "") -> str:
"""
Return a safe, generic error message for clients.
@@ -274,6 +293,31 @@ async def get_vector_sync_status(request: Request) -> JSONResponse:
)
try:
# Bus status backend (INGEST_MODE=external): there is no in-process
# queue; pending/terminal state comes from the NATS status subscriber's
# store. indexed_documents stays the mode-independent Qdrant count.
if settings.status_backend == "bus":
store = getattr(request.app.state, "status_store", None)
indexed_count = 0
try:
qdrant_client = await get_qdrant_client()
count_result = await qdrant_client.count(
collection_name=settings.get_collection_name(),
count_filter=Filter(must=[get_placeholder_filter()]),
)
indexed_count = count_result.count
except Exception as e:
logger.warning("Failed to query Qdrant for indexed count: %s", e)
return JSONResponse(
{
"status": "idle",
"indexed_documents": indexed_count,
"pending_documents": 0,
"status_backend": "bus",
"recent_states": store.counts() if store is not None else {},
}
)
# Get document receive stream from app state (set by starlette_lifespan in app.py)
document_receive_stream = getattr(
request.app.state, "document_receive_stream", None