feat: add opt-in MCP decomposition hook points (design §10)
Adds the seven §10.2 hook-point modules + five env vars so Astrolabe Cloud can offload document processing to the external document-processor / embedding gateway. Purely additive: with every setting unset the server behaves exactly as today, so self-hosters are unaffected (Deck #92). Hook points (all default to current monolith behavior): - config: EMBEDDING_PROVIDER, INGEST_MODE, STATUS_BACKEND, COLLECTION_METADATA_SOURCE, FACT_EVENT_EMITTER (+ supporting settings), validated in Settings.__post_init__ (fail-fast STATUS_BACKEND=local with INGEST_MODE=external); shared canonical.py. - vector/payload_keys.py + acl_hash.py: cross-impl NAMESPACE/point_id (§2.2) and BLAKE2b-128 ACL hash (§11), pinned by fixtures shared with the document-processor repo. - embedding/gateway_client.py: OpenAI-compatible GatewayProvider authenticating via M2M OIDC client-credentials (separate realm); manual-only registry entry. - vector/collection_metadata.py: sentinel-point / API metadata source with env fallback. - vector/queue/: hexagonal ingest producer ports + memory/NATS adapters (Postgres seam); INGEST_MODE=external publishes mcp.ingest.requested.{tenant} instead of the in-memory stream and skips the in-process processor pool. The lifespan becomes a composition root across both deployment branches. - vector/queue/status.py: STATUS_BACKEND=bus subscriber feeding a StatusStore the vector-sync status endpoint reads. - admin/payload_backfill.py: POST /api/v1/admin/payload-backfill (admin scope); processor writes the new payload keys; query-side ACL pre-filter gated behind ACL_PREFILTER_ENABLED (default off). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
c7da612f20
commit
d883052fb8
@@ -14,6 +14,7 @@ from anyio.streams.memory import MemoryObjectReceiveStream
|
||||
from httpx import HTTPStatusError
|
||||
from qdrant_client.models import FieldCondition, Filter, MatchValue, PointStruct
|
||||
|
||||
from nextcloud_mcp_server.acl_hash import compute_acl_hash
|
||||
from nextcloud_mcp_server.client import NextcloudClient
|
||||
from nextcloud_mcp_server.config import get_settings
|
||||
from nextcloud_mcp_server.document_processors import get_registry
|
||||
@@ -25,6 +26,7 @@ from nextcloud_mcp_server.observability.metrics import (
|
||||
)
|
||||
from nextcloud_mcp_server.observability.tracing import trace_operation
|
||||
from nextcloud_mcp_server.search.pdf_highlighter import PDFHighlighter
|
||||
from nextcloud_mcp_server.vector import payload_keys
|
||||
from nextcloud_mcp_server.vector.document_chunker import DocumentChunker
|
||||
from nextcloud_mcp_server.vector.html_processor import html_to_markdown
|
||||
from nextcloud_mcp_server.vector.placeholder import delete_placeholder_point
|
||||
@@ -672,6 +674,15 @@ async def _index_document(
|
||||
indexed_at = int(time.time())
|
||||
points = []
|
||||
|
||||
# Decomposition payload keys (design §10.2) — written even in local mode so
|
||||
# a future migration to the external processor is friction-free. Computed
|
||||
# once per document (not per chunk). The local processor has no triage, so
|
||||
# PIPELINE_TIER is "fast"; ACL hash records at least the owner principal
|
||||
# (full share enumeration is a follow-up — a missing/partial acl_hash is
|
||||
# safe because the query-side pre-filter only applies when present + enabled).
|
||||
_embedding_identity = get_settings().get_embedding_model_name()
|
||||
_acl_hash = compute_acl_hash([("user", doc_task.user_id)])
|
||||
|
||||
# Surface deck card data quality issues at indexing time rather than
|
||||
# only at verification time (where _verify_deck_cards falls through to
|
||||
# legacy-data pass-through when board_id/stack_id are missing). This is
|
||||
@@ -719,6 +730,12 @@ async def _index_document(
|
||||
"chunk_start_offset": chunk.start_offset,
|
||||
"chunk_end_offset": chunk.end_offset,
|
||||
"metadata_version": 2, # v2 includes position metadata
|
||||
# Decomposition payload keys (design §10.2), additive.
|
||||
payload_keys.PROCESSOR_VERSION: "monolith-v1",
|
||||
payload_keys.PARSED_AT: indexed_at,
|
||||
payload_keys.PIPELINE_TIER: "fast",
|
||||
payload_keys.EMBEDDING_IDENTITY: _embedding_identity,
|
||||
payload_keys.ACL_HASH: _acl_hash,
|
||||
# File-specific metadata (PDF, etc.)
|
||||
**(
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user