feat: implement vector sync scanner and processor (ADR-007 Phase 2)

Implements background vector database synchronization using anyio
TaskGroups for BasicAuth mode with single-user credentials.

Scanner Implementation:
- Periodic document discovery (hourly, configurable)
- Timestamp-based change detection (Nextcloud vs Qdrant)
- Wake event for immediate scanning on-demand
- Supports both initial sync (all docs) and incremental sync (changes only)
- Detects deleted documents and queues for removal

Processor Implementation:
- Concurrent document processing pool (3 workers default)
- I/O-bound embedding generation via Ollama API
- Retry logic with exponential backoff (3 retries)
- Document chunking (512 words, 50-word overlap)
- Handles both index and delete operations
- Upserts vectors to Qdrant with rich metadata

App Lifespan Integration:
- Extended AppContext with background task state
- Modified app_lifespan_basic() to start tasks via anyio TaskGroups
- Graceful shutdown with coordinated task cancellation
- Only activates when VECTOR_SYNC_ENABLED=true

Embedding Service:
- OllamaEmbeddingProvider with TLS support
- Singleton pattern for shared client instances
- Batch embedding support for efficiency
- Auto-detects embedding dimension (768 for nomic-embed-text)

Qdrant Client:
- Async client wrapper with singleton pattern
- Auto-creates collection on first use
- COSINE distance metric for semantic similarity
- Integrates with embedding service for dimension detection

Health Check Enhancement:
- Added Qdrant status check to /health/ready endpoint
- Only checks when VECTOR_SYNC_ENABLED=true
- 2-second timeout for health probe
- Reports connection errors with details

Configuration:
- VECTOR_SYNC_ENABLED: Enable background sync
- VECTOR_SYNC_SCAN_INTERVAL: Scanner frequency (3600s default)
- VECTOR_SYNC_PROCESSOR_WORKERS: Concurrent processors (3 default)
- QDRANT_URL, QDRANT_API_KEY, QDRANT_COLLECTION: Vector DB config
- OLLAMA_BASE_URL, OLLAMA_EMBEDDING_MODEL: Embedding service config

Dependencies Added:
- qdrant-client>=1.7.0: Vector database client

Docker Compose:
- Added Qdrant service with health check
- Exposed ports 6333 (REST) and 6334 (gRPC)
- Configured MCP service with vector sync environment
- Added qdrant-data volume for persistence

Known Issue:
- FastMCP lifespan not triggering for streamable-http transport
- Background tasks will start once lifespan integration is complete
- Lifespan triggers on MCP session establishment, not server startup

Related: ADR-007 Background Vector Database Synchronization

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2025-11-08 21:14:38 +01:00
co-authored by Claude
parent dc93da2ea0
commit 8f45e996e8
16 changed files with 1122 additions and 11 deletions
@@ -79,19 +79,22 @@ async def register_client(
client_name: str = "Nextcloud MCP Server",
redirect_uris: list[str] | None = None,
scopes: str = "openid profile email",
token_type: str = "Bearer",
token_type: str | None = "Bearer",
resource_url: str | None = None,
) -> ClientInfo:
"""
Register a new OAuth client with Nextcloud OIDC using dynamic client registration.
Register a new OAuth client using RFC 7591 Dynamic Client Registration.
This function supports both Nextcloud OIDC and standard OIDC providers like Keycloak.
Args:
nextcloud_url: Base URL of the Nextcloud instance
nextcloud_url: Base URL of the OIDC provider
registration_endpoint: Full URL to the registration endpoint
client_name: Name of the client application
redirect_uris: List of redirect URIs (default: http://localhost:8000/oauth/callback)
scopes: Space-separated list of scopes to request
token_type: Type of access tokens to issue (default: "Bearer", also supports "JWT")
token_type: Type of access tokens (default: "Bearer", supports "JWT" for Nextcloud).
Set to None to omit this field (required for Keycloak and other standard providers).
resource_url: OAuth 2.0 Protected Resource URL (RFC 9728) - used for token introspection authorization
Returns:
@@ -100,6 +103,11 @@ async def register_client(
Raises:
httpx.HTTPStatusError: If registration fails
ValueError: If response is invalid
Note:
The token_type parameter is a Nextcloud-specific extension and is not part of RFC 7591.
Standard OIDC providers like Keycloak do not accept this field and will return a 400 error
if it's included. Set token_type=None when registering with Keycloak or other standard providers.
"""
if redirect_uris is None:
redirect_uris = ["http://localhost:8000/oauth/callback"]
@@ -111,9 +119,12 @@ async def register_client(
"grant_types": ["authorization_code", "refresh_token"],
"response_types": ["code"],
"scope": scopes,
"token_type": token_type,
}
# Add token_type if provided (Nextcloud-specific, not RFC 7591 standard)
if token_type is not None:
client_metadata["token_type"] = token_type
# Add resource_url if provided (RFC 9728)
if resource_url:
client_metadata["resource_url"] = resource_url