feat: backend-agnostic vector-sync gauges (pending/documents/chunks)
The only queue metric, mcp_vector_sync_queue_size, was updated inline by the single-user consumer (processor_task) but never by the multi-user consumer (oauth_processor_task). On multi-user tenants (e.g. blackbox-demo, 5 users) the gauge read 0 for 24h while the live anyio buffer held ~2214 pending documents (shown by /api/v1/vector-sync/status). The "indexed" figure was also a chunk count (16039 points ≈ 480 docs) mislabelled as documents. Publish a consumer-independent snapshot from a periodic task (vector/metrics_publisher.vector_sync_metrics_task), spawned in BOTH lifespan task groups (single-user and multi-user) and every queue backend: - mcp_vector_sync_pending_documents — outstanding work via ingest_status.get_ingest_pending() (anyio buffer depth or procrastinate todo+doing); also keeps the legacy queue_size gauge meaningful on all paths. - mcp_vector_sync_indexed_documents — distinct documents, counted exactly and cheaply via the one chunk_index=0 point per document (no facet). - mcp_vector_sync_indexed_chunks — total non-placeholder points. The /api/v1/vector-sync/status endpoint now returns indexed_documents (distinct docs) AND indexed_chunks separately, so documents and chunks are no longer conflated. The publisher uses approximate Qdrant counts (every-N-seconds gauge); the on-demand endpoint counts exactly. New knob: VECTOR_SYNC_METRICS_REFRESH_INTERVAL (default 20s). Fail-safe: a metrics refresh never disturbs ingest. BREAKING CHANGE: /api/v1/vector-sync/status field `indexed_documents` now holds the distinct-document count (was the chunk count); the chunk count moved to the new `indexed_chunks` field. The Astrolabe UI + the nc_get_vector_sync_status MCP tool / userinfo page are harmonized in a follow-up (Deck #195). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
55ea8dd358
commit
fbe70ecd9c
@@ -88,6 +88,7 @@ _DEFAULTS: dict[str, Any] = {
|
||||
"vector_sync_scan_interval": 300,
|
||||
"vector_sync_processor_workers": 3,
|
||||
"vector_sync_queue_max_size": 10000,
|
||||
"vector_sync_metrics_refresh_interval": 20,
|
||||
"vector_sync_user_poll_interval": 60,
|
||||
# Orphan-sweep at Pod startup (card #101). When True, delete any
|
||||
# placeholders carrying a different / absent ``instance_id`` before
|
||||
@@ -645,6 +646,10 @@ class Settings:
|
||||
vector_sync_scan_interval: int = 300 # seconds (5 minutes)
|
||||
vector_sync_processor_workers: int = 3
|
||||
vector_sync_queue_max_size: int = 10000
|
||||
# Cadence for the periodic gauge publisher (vector/metrics_publisher.py):
|
||||
# outstanding-work + indexed documents/chunks. Decoupled from the consumer
|
||||
# so the gauges are correct on every deployment mode and queue backend.
|
||||
vector_sync_metrics_refresh_interval: int = 20 # seconds
|
||||
vector_sync_user_poll_interval: int = 60 # seconds - OAuth mode user discovery
|
||||
vector_sync_orphan_sweep_enabled: bool = True # card #101
|
||||
# System tag marking files for vector indexing. The scanner indexes files
|
||||
@@ -1267,6 +1272,7 @@ def get_settings() -> Settings:
|
||||
"vector_sync_scan_interval": "VECTOR_SYNC_SCAN_INTERVAL",
|
||||
"vector_sync_processor_workers": "VECTOR_SYNC_PROCESSOR_WORKERS",
|
||||
"vector_sync_queue_max_size": "VECTOR_SYNC_QUEUE_MAX_SIZE",
|
||||
"vector_sync_metrics_refresh_interval": "VECTOR_SYNC_METRICS_REFRESH_INTERVAL",
|
||||
"vector_sync_user_poll_interval": "VECTOR_SYNC_USER_POLL_INTERVAL",
|
||||
"vector_sync_orphan_sweep_enabled": "VECTOR_SYNC_ORPHAN_SWEEP_ENABLED",
|
||||
"vector_sync_pdf_tag": "VECTOR_SYNC_PDF_TAG",
|
||||
|
||||
Reference in New Issue
Block a user