feat: backend-agnostic vector-sync gauges (pending/documents/chunks)
The only queue metric, mcp_vector_sync_queue_size, was updated inline by the single-user consumer (processor_task) but never by the multi-user consumer (oauth_processor_task). On multi-user tenants (e.g. blackbox-demo, 5 users) the gauge read 0 for 24h while the live anyio buffer held ~2214 pending documents (shown by /api/v1/vector-sync/status). The "indexed" figure was also a chunk count (16039 points ≈ 480 docs) mislabelled as documents. Publish a consumer-independent snapshot from a periodic task (vector/metrics_publisher.vector_sync_metrics_task), spawned in BOTH lifespan task groups (single-user and multi-user) and every queue backend: - mcp_vector_sync_pending_documents — outstanding work via ingest_status.get_ingest_pending() (anyio buffer depth or procrastinate todo+doing); also keeps the legacy queue_size gauge meaningful on all paths. - mcp_vector_sync_indexed_documents — distinct documents, counted exactly and cheaply via the one chunk_index=0 point per document (no facet). - mcp_vector_sync_indexed_chunks — total non-placeholder points. The /api/v1/vector-sync/status endpoint now returns indexed_documents (distinct docs) AND indexed_chunks separately, so documents and chunks are no longer conflated. The publisher uses approximate Qdrant counts (every-N-seconds gauge); the on-demand endpoint counts exactly. New knob: VECTOR_SYNC_METRICS_REFRESH_INTERVAL (default 20s). Fail-safe: a metrics refresh never disturbs ingest. BREAKING CHANGE: /api/v1/vector-sync/status field `indexed_documents` now holds the distinct-document count (was the chunk count); the chunk count moved to the new `indexed_chunks` field. The Astrolabe UI + the nc_get_vector_sync_status MCP tool / userinfo page are harmonized in a follow-up (Deck #195). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
55ea8dd358
commit
fbe70ecd9c
@@ -126,6 +126,7 @@ from nextcloud_mcp_server.server import (
|
||||
)
|
||||
from nextcloud_mcp_server.server.auth_tools import register_auth_tools
|
||||
from nextcloud_mcp_server.server.oauth_tools import register_oauth_tools
|
||||
from nextcloud_mcp_server.vector.metrics_publisher import vector_sync_metrics_task
|
||||
from nextcloud_mcp_server.vector.oauth_sync import (
|
||||
oauth_processor_task,
|
||||
user_manager_task,
|
||||
@@ -1744,6 +1745,16 @@ def get_app(transport: str = "streamable-http", enabled_apps: list[str] | None =
|
||||
username,
|
||||
)
|
||||
|
||||
# Publish outstanding-work + corpus gauges on a fixed cadence,
|
||||
# independent of the consumer path and queue backend (fixes the
|
||||
# gauge reading 0 on the multi-user path; see metrics_publisher).
|
||||
await tg.start(
|
||||
vector_sync_metrics_task,
|
||||
task_producer,
|
||||
receive_stream,
|
||||
shutdown_event,
|
||||
)
|
||||
|
||||
# Expose this long-lived task group to request-path code that
|
||||
# wants to spawn background work (e.g. ADR-019 verify-on-read
|
||||
# eviction). Eviction coroutines have their own try/except, so
|
||||
@@ -1973,6 +1984,18 @@ def get_app(transport: str = "streamable-http", enabled_apps: list[str] | None =
|
||||
nextcloud_host_for_sync,
|
||||
)
|
||||
|
||||
# Publish outstanding-work + corpus gauges on a fixed
|
||||
# cadence. Critical on this multi-user path: the consumer is
|
||||
# oauth_processor_task, which never updated the queue gauge,
|
||||
# so without this the gauge read 0 while the buffer held
|
||||
# thousands of pending docs (see metrics_publisher).
|
||||
await tg.start(
|
||||
vector_sync_metrics_task,
|
||||
task_producer,
|
||||
receive_stream,
|
||||
shutdown_event,
|
||||
)
|
||||
|
||||
# Expose this long-lived task group to request-path code
|
||||
# that wants to spawn background work (e.g. ADR-019
|
||||
# verify-on-read eviction). Eviction coroutines have their
|
||||
|
||||
Reference in New Issue
Block a user