Chris CoutinhoandClaude Opus 4.8
21b7922bac
Re-architect document ingest from the shared NATS-glued document-processor to a per-tenant, in-process model owned by nextcloud-mcp-server (Deck #183). The MCP server now owns both sides of ingest: - Producer (api role): the scanner defers one job per changed document into the app's Postgres via procrastinate (queueing_lock dedup; no execution lock, so a crashed worker can't deadlock a doc — Qdrant upserts are idempotent). - Consumer (worker role): `nextcloud-mcp-server worker` drains the queue and runs the existing process_document pipeline; a periodic task reclaims jobs orphaned in `doing` by a crash. INGEST_QUEUE selects the transport (auto: postgres when DATABASE_URL is Postgres, else the in-process anyio queue for SQLite/dev). procrastinate manages its own tables (applied on a fresh DB at startup and by `db upgrade`). The vector-sync status surface reads job counts from Postgres in postgres mode. procrastinate + psycopg3 ship in the [postgres] extra; the app's own engine still uses asyncpg (driver unification is a follow-up handled in the rendered Helm chart). NATS JetStream, the Postgres-queue stub, the bus status subscriber, and nats-py are removed. BREAKING CHANGE: the external-NATS-ingest env vars are removed (INGEST_MODE, STATUS_BACKEND, INGEST_BUS_URL, INGEST_BUS_NUM_REPLICAS, FACT_EVENT_EMITTER). Use INGEST_QUEUE (memory|postgres) and the `worker` command instead. TENANT_ID is retained (no longer NATS-subject-charset-validated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
57 lines
2.1 KiB
Python
57 lines
2.1 KiB
Python
"""Shared read model for the vector-sync status surface (Deck #183).
|
|
|
|
The status endpoints (``/api/v1/vector-sync/status``, the userinfo route, and
|
|
the ``nc_get_vector_sync_status`` MCP tool) all need the same "how much work is
|
|
outstanding" figure, computed differently per ``INGEST_QUEUE`` backend:
|
|
|
|
- ``memory`` — the in-process anyio stream's buffer depth (today's behavior).
|
|
- ``postgres`` — procrastinate job counts read from the per-tenant Postgres
|
|
(``todo`` + ``doing``), plus the per-status breakdown for observability.
|
|
|
|
``indexed_documents`` (the Qdrant placeholder count) is backend-independent and
|
|
stays at each call site.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
from dataclasses import dataclass
|
|
from typing import Any
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
@dataclass
|
|
class IngestPending:
|
|
"""Outstanding-work view for the active ingest queue backend."""
|
|
|
|
pending: int
|
|
# Per-status counts (todo/doing/failed/…) on the postgres backend; None on
|
|
# the memory backend, which has no durable per-status breakdown.
|
|
job_counts: dict[str, int] | None = None
|
|
|
|
|
|
async def get_ingest_pending(
|
|
*, task_producer: Any, document_receive_stream: Any, ingest_queue: str | None
|
|
) -> IngestPending:
|
|
"""Compute outstanding ingest work for the configured queue backend.
|
|
|
|
Never raises — a status surface must stay available even if the queue is
|
|
unreachable; failures degrade to ``pending=0``.
|
|
"""
|
|
if ingest_queue == "postgres":
|
|
counts: dict[str, int] = {}
|
|
if task_producer is not None and hasattr(task_producer, "job_counts"):
|
|
try:
|
|
counts = await task_producer.job_counts()
|
|
except Exception as e:
|
|
logger.warning("Failed to read ingest job counts: %s", e)
|
|
pending = counts.get("todo", 0) + counts.get("doing", 0)
|
|
return IngestPending(pending=pending, job_counts=counts)
|
|
|
|
if document_receive_stream is None:
|
|
return IngestPending(pending=0)
|
|
return IngestPending(
|
|
pending=document_receive_stream.statistics().current_buffer_used
|
|
)
|