Chris CoutinhoandClaude Opus 4.8
21b7922bac
Re-architect document ingest from the shared NATS-glued document-processor to a per-tenant, in-process model owned by nextcloud-mcp-server (Deck #183). The MCP server now owns both sides of ingest: - Producer (api role): the scanner defers one job per changed document into the app's Postgres via procrastinate (queueing_lock dedup; no execution lock, so a crashed worker can't deadlock a doc — Qdrant upserts are idempotent). - Consumer (worker role): `nextcloud-mcp-server worker` drains the queue and runs the existing process_document pipeline; a periodic task reclaims jobs orphaned in `doing` by a crash. INGEST_QUEUE selects the transport (auto: postgres when DATABASE_URL is Postgres, else the in-process anyio queue for SQLite/dev). procrastinate manages its own tables (applied on a fresh DB at startup and by `db upgrade`). The vector-sync status surface reads job counts from Postgres in postgres mode. procrastinate + psycopg3 ship in the [postgres] extra; the app's own engine still uses asyncpg (driver unification is a follow-up handled in the rendered Helm chart). NATS JetStream, the Postgres-queue stub, the bus status subscriber, and nats-py are removed. BREAKING CHANGE: the external-NATS-ingest env vars are removed (INGEST_MODE, STATUS_BACKEND, INGEST_BUS_URL, INGEST_BUS_NUM_REPLICAS, FACT_EVENT_EMITTER). Use INGEST_QUEUE (memory|postgres) and the `worker` command instead. TENANT_ID is retained (no longer NATS-subject-charset-validated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
58 lines
1.8 KiB
Python
58 lines
1.8 KiB
Python
"""Unit tests for the shared ingest-status read model (Deck #183)."""
|
|
|
|
from types import SimpleNamespace
|
|
from unittest.mock import AsyncMock
|
|
|
|
import pytest
|
|
|
|
from nextcloud_mcp_server.vector.ingest_status import get_ingest_pending
|
|
|
|
pytestmark = pytest.mark.unit
|
|
|
|
|
|
class TestGetIngestPending:
|
|
async def test_postgres_reads_job_counts(self):
|
|
producer = AsyncMock()
|
|
producer.job_counts.return_value = {"todo": 5, "doing": 2, "failed": 1}
|
|
|
|
result = await get_ingest_pending(
|
|
task_producer=producer,
|
|
document_receive_stream=None,
|
|
ingest_queue="postgres",
|
|
)
|
|
assert result.pending == 7 # todo + doing
|
|
assert result.job_counts == {"todo": 5, "doing": 2, "failed": 1}
|
|
|
|
async def test_postgres_degrades_to_zero_on_error(self):
|
|
producer = AsyncMock()
|
|
producer.job_counts.side_effect = RuntimeError("db down")
|
|
|
|
result = await get_ingest_pending(
|
|
task_producer=producer,
|
|
document_receive_stream=None,
|
|
ingest_queue="postgres",
|
|
)
|
|
assert result.pending == 0
|
|
assert result.job_counts == {}
|
|
|
|
async def test_memory_reads_stream_buffer(self):
|
|
stream = SimpleNamespace(
|
|
statistics=lambda: SimpleNamespace(current_buffer_used=3)
|
|
)
|
|
result = await get_ingest_pending(
|
|
task_producer=None,
|
|
document_receive_stream=stream,
|
|
ingest_queue="memory",
|
|
)
|
|
assert result.pending == 3
|
|
assert result.job_counts is None
|
|
|
|
async def test_memory_without_stream_is_zero(self):
|
|
result = await get_ingest_pending(
|
|
task_producer=None,
|
|
document_receive_stream=None,
|
|
ingest_queue="memory",
|
|
)
|
|
assert result.pending == 0
|
|
assert result.job_counts is None
|