fix(vector-sync): sweep placeholder orphans at Pod startup (#101)
When the per-tenant nextcloud-mcp-server Pod OOMKills mid-batch, the in-memory anyio processor queue is lost but the placeholder Qdrant points (is_placeholder=true, status=pending) survive. The next Pod's scanner re-runs, sees the existing placeholders, applies the 5 × VECTOR_SYNC_SCAN_INTERVAL staleness gate (~5h with the deployed 1h scan interval), and skips them. Result: 0 documents indexed for the duration of the gate after every restart. Stamps a process-level instance_id (UUID per Pod-process) onto every placeholder write. A new sweep_orphan_placeholders helper, called once from starlette_lifespan after the Qdrant client is initialised and before the scanner / user-manager spawns, scrolls the collection and deletes any placeholder whose instance_id doesn't match the current Pod's (including placeholders with no instance_id field — back-compat for pre-fix Pod versions). The scanner's next cycle naturally re-creates fresh placeholders and queues work normally; no DocumentTask reconstruction needed. Sweep is one-shot at startup, not periodic — the existing staleness gate still covers same-Pod recovery, and the cross-Pod-restart gap was the only failure mode. Failure is non-fatal (logged via vector_sync.orphan_sweep_failed) so a transient Qdrant hiccup at boot doesn't prevent the scanner from running. Both lifespan branches (single-user BasicAuth, OAuth / multi-user BasicAuth) call the sweep via a module-local helper. A new VECTOR_SYNC_ORPHAN_SWEEP_ENABLED setting (default True) provides an escape hatch. Closes Deck #101. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
32fc03cf67
commit
a5cbe91b29
@@ -87,6 +87,12 @@ _DEFAULTS: dict[str, Any] = {
|
||||
"vector_sync_processor_workers": 3,
|
||||
"vector_sync_queue_max_size": 10000,
|
||||
"vector_sync_user_poll_interval": 60,
|
||||
# Orphan-sweep at Pod startup (card #101). When True, delete any
|
||||
# placeholders carrying a different / absent ``instance_id`` before
|
||||
# the scanner's first cycle, so a Pod restart mid-batch doesn't
|
||||
# leave work stuck behind the 5x-scan-interval staleness gate.
|
||||
# Escape hatch only — leave on by default.
|
||||
"vector_sync_orphan_sweep_enabled": True,
|
||||
# Verify-on-read concurrency cap (ADR-019)
|
||||
"verification_concurrency": 20,
|
||||
# Qdrant
|
||||
@@ -585,6 +591,7 @@ class Settings:
|
||||
vector_sync_processor_workers: int = 3
|
||||
vector_sync_queue_max_size: int = 10000
|
||||
vector_sync_user_poll_interval: int = 60 # seconds - OAuth mode user discovery
|
||||
vector_sync_orphan_sweep_enabled: bool = True # card #101
|
||||
|
||||
# Verify-on-read concurrency (ADR-019). Cap on parallel Nextcloud
|
||||
# round-trips during search-result verification fan-out. Lower this if the
|
||||
@@ -1076,6 +1083,7 @@ def get_settings() -> Settings:
|
||||
"vector_sync_processor_workers": "VECTOR_SYNC_PROCESSOR_WORKERS",
|
||||
"vector_sync_queue_max_size": "VECTOR_SYNC_QUEUE_MAX_SIZE",
|
||||
"vector_sync_user_poll_interval": "VECTOR_SYNC_USER_POLL_INTERVAL",
|
||||
"vector_sync_orphan_sweep_enabled": "VECTOR_SYNC_ORPHAN_SWEEP_ENABLED",
|
||||
# Verify-on-read (ADR-019)
|
||||
"verification_concurrency": "VERIFICATION_CONCURRENCY",
|
||||
# Qdrant settings
|
||||
|
||||
Reference in New Issue
Block a user