feat: backend-agnostic vector-sync gauges (pending/documents/chunks)

The only queue metric, mcp_vector_sync_queue_size, was updated inline by the
single-user consumer (processor_task) but never by the multi-user consumer
(oauth_processor_task). On multi-user tenants (e.g. blackbox-demo, 5 users) the
gauge read 0 for 24h while the live anyio buffer held ~2214 pending documents
(shown by /api/v1/vector-sync/status). The "indexed" figure was also a chunk
count (16039 points ≈ 480 docs) mislabelled as documents.

Publish a consumer-independent snapshot from a periodic task
(vector/metrics_publisher.vector_sync_metrics_task), spawned in BOTH lifespan
task groups (single-user and multi-user) and every queue backend:
- mcp_vector_sync_pending_documents — outstanding work via
  ingest_status.get_ingest_pending() (anyio buffer depth or procrastinate
  todo+doing); also keeps the legacy queue_size gauge meaningful on all paths.
- mcp_vector_sync_indexed_documents — distinct documents, counted exactly and
  cheaply via the one chunk_index=0 point per document (no facet).
- mcp_vector_sync_indexed_chunks — total non-placeholder points.

The /api/v1/vector-sync/status endpoint now returns indexed_documents (distinct
docs) AND indexed_chunks separately, so documents and chunks are no longer
conflated. The publisher uses approximate Qdrant counts (every-N-seconds gauge);
the on-demand endpoint counts exactly. New knob:
VECTOR_SYNC_METRICS_REFRESH_INTERVAL (default 20s). Fail-safe: a metrics refresh
never disturbs ingest.

BREAKING CHANGE: /api/v1/vector-sync/status field `indexed_documents` now holds
the distinct-document count (was the chunk count); the chunk count moved to the
new `indexed_chunks` field. The Astrolabe UI + the nc_get_vector_sync_status MCP
tool / userinfo page are harmonized in a follow-up (Deck #195).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-04 19:30:10 +02:00
co-authored by Claude Opus 4.8
parent 55ea8dd358
commit fbe70ecd9c
6 changed files with 384 additions and 14 deletions
+6
View File
@@ -88,6 +88,7 @@ _DEFAULTS: dict[str, Any] = {
"vector_sync_scan_interval": 300,
"vector_sync_processor_workers": 3,
"vector_sync_queue_max_size": 10000,
"vector_sync_metrics_refresh_interval": 20,
"vector_sync_user_poll_interval": 60,
# Orphan-sweep at Pod startup (card #101). When True, delete any
# placeholders carrying a different / absent ``instance_id`` before
@@ -645,6 +646,10 @@ class Settings:
vector_sync_scan_interval: int = 300 # seconds (5 minutes)
vector_sync_processor_workers: int = 3
vector_sync_queue_max_size: int = 10000
# Cadence for the periodic gauge publisher (vector/metrics_publisher.py):
# outstanding-work + indexed documents/chunks. Decoupled from the consumer
# so the gauges are correct on every deployment mode and queue backend.
vector_sync_metrics_refresh_interval: int = 20 # seconds
vector_sync_user_poll_interval: int = 60 # seconds - OAuth mode user discovery
vector_sync_orphan_sweep_enabled: bool = True # card #101
# System tag marking files for vector indexing. The scanner indexes files
@@ -1267,6 +1272,7 @@ def get_settings() -> Settings:
"vector_sync_scan_interval": "VECTOR_SYNC_SCAN_INTERVAL",
"vector_sync_processor_workers": "VECTOR_SYNC_PROCESSOR_WORKERS",
"vector_sync_queue_max_size": "VECTOR_SYNC_QUEUE_MAX_SIZE",
"vector_sync_metrics_refresh_interval": "VECTOR_SYNC_METRICS_REFRESH_INTERVAL",
"vector_sync_user_poll_interval": "VECTOR_SYNC_USER_POLL_INTERVAL",
"vector_sync_orphan_sweep_enabled": "VECTOR_SYNC_ORPHAN_SWEEP_ENABLED",
"vector_sync_pdf_tag": "VECTOR_SYNC_PDF_TAG",