Chris CoutinhoandClaude Opus 4.8
d883052fb8
Adds the seven §10.2 hook-point modules + five env vars so Astrolabe Cloud can offload document processing to the external document-processor / embedding gateway. Purely additive: with every setting unset the server behaves exactly as today, so self-hosters are unaffected (Deck #92). Hook points (all default to current monolith behavior): - config: EMBEDDING_PROVIDER, INGEST_MODE, STATUS_BACKEND, COLLECTION_METADATA_SOURCE, FACT_EVENT_EMITTER (+ supporting settings), validated in Settings.__post_init__ (fail-fast STATUS_BACKEND=local with INGEST_MODE=external); shared canonical.py. - vector/payload_keys.py + acl_hash.py: cross-impl NAMESPACE/point_id (§2.2) and BLAKE2b-128 ACL hash (§11), pinned by fixtures shared with the document-processor repo. - embedding/gateway_client.py: OpenAI-compatible GatewayProvider authenticating via M2M OIDC client-credentials (separate realm); manual-only registry entry. - vector/collection_metadata.py: sentinel-point / API metadata source with env fallback. - vector/queue/: hexagonal ingest producer ports + memory/NATS adapters (Postgres seam); INGEST_MODE=external publishes mcp.ingest.requested.{tenant} instead of the in-memory stream and skips the in-process processor pool. The lifespan becomes a composition root across both deployment branches. - vector/queue/status.py: STATUS_BACKEND=bus subscriber feeding a StatusStore the vector-sync status endpoint reads. - admin/payload_backfill.py: POST /api/v1/admin/payload-backfill (admin scope); processor writes the new payload keys; query-side ACL pre-filter gated behind ACL_PREFILTER_ENABLED (default off). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
60 lines
2.2 KiB
Python
60 lines
2.2 KiB
Python
"""Ingest-path ports (design §10, hexagonal).
|
|
|
|
A ``TaskProducer`` is where the scanner + webhook receiver send a
|
|
``DocumentTask``. The transport behind it is swappable:
|
|
|
|
- the in-process anyio ``MemoryObjectSendStream`` (local ingest — the default),
|
|
- ``NatsTaskProducer`` (external ingest → the document-processor), and
|
|
- a future Postgres-queue producer (seam only; the *external* processor owns the
|
|
consume side — see ``postgres.py``).
|
|
|
|
The protocol is exactly the surface the scanner/oauth_sync already use on the
|
|
memory stream (``send`` + ``clone`` + ``async with``), so both adapters drop in
|
|
with only a type-annotation change at the call sites. There is intentionally NO
|
|
consumer port: the MCP server's only in-process consumer is the memory stream;
|
|
when ingest is external the document-processor is the consumer, not this server.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from types import TracebackType
|
|
from typing import TYPE_CHECKING, Protocol, runtime_checkable
|
|
|
|
if TYPE_CHECKING:
|
|
from ..scanner import DocumentTask
|
|
|
|
|
|
@runtime_checkable
|
|
class TaskProducer(Protocol):
|
|
"""Sink for scanner/webhook ``DocumentTask``s (see module docstring)."""
|
|
|
|
# Positional-only so anyio's MemoryObjectSendStream.send(item) structurally
|
|
# satisfies this protocol (its parameter is named "item", not "task").
|
|
async def send(self, task: DocumentTask, /) -> None: ...
|
|
|
|
def clone(self) -> TaskProducer:
|
|
"""Return a producer handle for one user's scanner (multi-user mode).
|
|
|
|
For the memory stream this is a real clone (each closed independently);
|
|
for the bus it returns ``self`` (one shared connection).
|
|
"""
|
|
...
|
|
|
|
async def __aenter__(self) -> TaskProducer: ...
|
|
|
|
async def __aexit__(
|
|
self,
|
|
exc_type: type[BaseException] | None,
|
|
exc: BaseException | None,
|
|
tb: TracebackType | None,
|
|
) -> None: ...
|
|
|
|
async def aclose(self) -> None:
|
|
"""Close *this* handle (e.g. a per-user clone when its scanner exits).
|
|
|
|
For the memory stream this closes the clone; for the shared bus
|
|
connection it is a no-op (the connection is owned by the lifespan,
|
|
which drains it once on shutdown).
|
|
"""
|
|
...
|