fix(vector-sync): sweep placeholder orphans at Pod startup (#101)
When the per-tenant nextcloud-mcp-server Pod OOMKills mid-batch, the in-memory anyio processor queue is lost but the placeholder Qdrant points (is_placeholder=true, status=pending) survive. The next Pod's scanner re-runs, sees the existing placeholders, applies the 5 × VECTOR_SYNC_SCAN_INTERVAL staleness gate (~5h with the deployed 1h scan interval), and skips them. Result: 0 documents indexed for the duration of the gate after every restart. Stamps a process-level instance_id (UUID per Pod-process) onto every placeholder write. A new sweep_orphan_placeholders helper, called once from starlette_lifespan after the Qdrant client is initialised and before the scanner / user-manager spawns, scrolls the collection and deletes any placeholder whose instance_id doesn't match the current Pod's (including placeholders with no instance_id field — back-compat for pre-fix Pod versions). The scanner's next cycle naturally re-creates fresh placeholders and queues work normally; no DocumentTask reconstruction needed. Sweep is one-shot at startup, not periodic — the existing staleness gate still covers same-Pod recovery, and the cross-Pod-restart gap was the only failure mode. Failure is non-fatal (logged via vector_sync.orphan_sweep_failed) so a transient Qdrant hiccup at boot doesn't prevent the scanner from running. Both lifespan branches (single-user BasicAuth, OAuth / multi-user BasicAuth) call the sweep via a module-local helper. A new VECTOR_SYNC_ORPHAN_SWEEP_ENABLED setting (default True) provides an escape hatch. Closes Deck #101. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
32fc03cf67
commit
a5cbe91b29
@@ -129,6 +129,7 @@ from nextcloud_mcp_server.vector.oauth_sync import (
|
||||
oauth_processor_task,
|
||||
user_manager_task,
|
||||
)
|
||||
from nextcloud_mcp_server.vector.placeholder import sweep_orphan_placeholders
|
||||
from nextcloud_mcp_server.vector.processor import processor_task
|
||||
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
|
||||
from nextcloud_mcp_server.vector.scanner import scanner_task
|
||||
@@ -1437,6 +1438,34 @@ def get_app(transport: str = "streamable-http", enabled_apps: list[str] | None =
|
||||
await stack.enter_async_context(_maybe_login_flow_cleanup(app))
|
||||
yield
|
||||
|
||||
async def _sweep_orphan_placeholders_if_enabled() -> None:
|
||||
"""One-shot Pod-startup sweep of cross-restart placeholder orphans.
|
||||
|
||||
See ``vector.placeholder.sweep_orphan_placeholders`` and Deck
|
||||
card #101. Both lifespan branches (single-user BasicAuth and
|
||||
OAuth / multi-user BasicAuth) call this after the qdrant
|
||||
client is initialised and before the scanner / user-manager
|
||||
tasks spawn. Failures are non-fatal — the existing staleness
|
||||
gate will eventually re-queue orphans on the slow ~5h path.
|
||||
"""
|
||||
if not settings.vector_sync_orphan_sweep_enabled:
|
||||
return
|
||||
try:
|
||||
qdrant_client = await get_qdrant_client()
|
||||
swept, kept = await sweep_orphan_placeholders(
|
||||
qdrant_client, settings.qdrant_collection
|
||||
)
|
||||
logger.info(
|
||||
"vector_sync.orphan_sweep",
|
||||
extra={
|
||||
"swept": swept,
|
||||
"kept": kept,
|
||||
"collection": settings.qdrant_collection,
|
||||
},
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("vector_sync.orphan_sweep_failed")
|
||||
|
||||
@asynccontextmanager
|
||||
async def starlette_lifespan(app: Starlette):
|
||||
# Set OAuth context for OAuth login routes (ADR-004)
|
||||
@@ -1612,6 +1641,9 @@ def get_app(transport: str = "streamable-http", enabled_apps: list[str] | None =
|
||||
f"Cannot start vector sync - Qdrant initialization failed: {e}"
|
||||
) from e
|
||||
|
||||
# Orphan-sweep before scanner starts — card #101.
|
||||
await _sweep_orphan_placeholders_if_enabled()
|
||||
|
||||
# Initialize shared state
|
||||
send_stream, receive_stream = anyio.create_memory_object_stream(
|
||||
max_buffer_size=settings.vector_sync_queue_max_size
|
||||
@@ -1774,6 +1806,11 @@ def get_app(transport: str = "streamable-http", enabled_apps: list[str] | None =
|
||||
f"Cannot start vector sync - Qdrant initialization failed: {e}"
|
||||
) from e
|
||||
|
||||
# Orphan-sweep before scanners spawn — card #101. Runs once
|
||||
# across the shared (per-tenant) collection regardless of
|
||||
# how many per-user scanners the user-manager later starts.
|
||||
await _sweep_orphan_placeholders_if_enabled()
|
||||
|
||||
# Clean up stale app passwords at startup (BasicAuth mode only)
|
||||
if not oauth_enabled:
|
||||
try:
|
||||
|
||||
Reference in New Issue
Block a user