- viz_routes.py: Extract "dense" vector from named vector dict
- semantic.py: Specify using="dense" for BM25 hybrid collections
- Fixes "X must be 2D array" error in hybrid search
- Fixes "Dense vector is not found" error in semantic search
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The visualization UI was still using the old 'hybrid' algorithm name and
weight parameters that were replaced by the BM25 hybrid search refactor.
This caused "Unknown algorithm: hybrid" errors when using the search
& visualize feature.
Changes:
- Update default algorithm from 'hybrid' to 'bm25_hybrid'
- Update default scoreThreshold from 0.7 to 0.0 to match backend
- Remove deprecated semanticWeight, keywordWeight, fuzzyWeight parameters
- Remove weight parameters from search request
Fixes the visualization search functionality after BM25 hybrid refactor.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Remove obsolete search algorithm imports (Fuzzy, Keyword, Hybrid)
- Update UI to only show Semantic and BM25 Hybrid algorithms
- Replace manual weight controls with RRF fusion info message
- Update default algorithm from "hybrid" to "bm25_hybrid"
- Remove weight parameters (semantic_weight, keyword_weight, fuzzy_weight)
- Update score_threshold default from 0.7 to 0.0 for RRF scoring
- Document ty type checker in CLAUDE.md
Fixes unresolved-import type errors after BM25 refactor.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit addresses critical performance issues with vector visualization
search (reducing time from 40s to ~2s) and improves result visualization
through better visual encoding.
## Performance Fixes
### 1. Fix blocking sleep in retry decorator (base.py:51)
- Changed `time.sleep(5)` to `await anyio.sleep(5)` in @retry_on_429
- Prevents entire event loop from freezing during rate limit retries
- Impact: Reduced search time from 22s to 16s initially
### 2. Add concurrency limiting for verification (verification.py:77-93)
- Added `anyio.Semaphore(20)` to limit concurrent HTTP requests
- Prevents connection pool exhaustion (RequestError) from 90+ simultaneous requests
- Fixes false filtering (was filtering 77/90 results incorrectly)
- Note: Semaphore still in code but verification removed from viz endpoint
### 3. Remove unnecessary verification from viz endpoint (viz_routes.py:483-486)
- Visualization only needs Qdrant metadata (title, excerpt), not full content
- Verification only required for sampling (LLM needs full note content)
- Impact: Reduced search time from 43.7s to ~2s (final fix)
### 4. Restore streaming scanner pattern (scanner.py)
- Process notes one-at-a-time using async generator
- Avoids loading all notes into memory
## Visualization Improvements
### 5. Result-relative score normalization (viz_routes.py:489-504)
- Normalize scores within result set: best=1.0, worst=0.0
- Removes arbitrary RRF normalization (theoretical max didn't make sense)
- Makes visual encoding meaningful regardless of algorithm scores
### 6. Power scaling for marker sizes (userinfo_routes.py:743)
- Changed from linear `8 + (score * 12)` to power `6 + (score² * 14)`
- Creates dramatic visual contrast: 0.0→6px, 0.5→9.5px, 1.0→20px
- Combined with opacity (0.2-1.0) for clear visual hierarchy
### 7. Multi-channel visual encoding (userinfo_routes.py:740-745)
- Size: Exponentially scaled with score²
- Opacity: Linear 0.2-1.0 (keeps all points visible)
- Color: Viridis gradient (blue→yellow)
- Effect: Top results are large/bright/opaque, context results small/dim/transparent
## Result
- Search time: 40s → ~2s (20x faster)
- Visual contrast: Subtle → dramatic (clear result hierarchy)
- No arbitrary cutoffs: All results visible, best naturally highlighted
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Replace asyncio primitives with anyio equivalents throughout the codebase
to establish a single async pattern. This provides better structured
concurrency with automatic cancellation on errors and aligns with the
pytest anyio configuration.
Changes:
- hybrid.py: Replace asyncio.gather() with anyio task groups
- token_broker.py: Replace asyncio.Lock() with anyio.Lock()
- storage.py: Replace asyncio.run() with anyio.run()
- app.py: Replace tg.start_soon() with await tg.start() for task status
- processor.py: Add task_status parameter for structured startup
- scanner.py: Add task_status parameter for structured startup
- CLAUDE.md: Update async/await patterns guidance
The change from start_soon() to await tg.start() enables proper task
initialization signaling, ensuring background tasks are ready before
proceeding. This follows anyio best practices for structured concurrency.
All 118 unit tests pass with the new implementation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Collect all notes to delete first, then delete concurrently
- Use anyio task group with semaphore (20 concurrent deletions)
- Add progress reporting and error tracking for deletions
- Show count of notes found before deletion starts
This significantly improves --force performance when refreshing large
corpuses (e.g., 3,633 notes now delete in ~1 minute instead of ~5 minutes).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add --force flag to delete all existing notes in target category before upload
- Implement concurrent uploads using anyio task groups (20 concurrent max)
- Add semaphore to limit concurrent requests and avoid overwhelming server
- Improve progress reporting with upload count and error tracking
- Update README with --force flag documentation
Performance improvement: Concurrent uploads significantly reduce upload time
from ~10-15 minutes to ~2-3 minutes for 3,633 documents.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- HuggingFace BeIR/nfcorpus only has 'corpus' and 'queries' configs
- Download qrels from original BEIR ZIP file (nfcorpus.zip)
- Use synchronous httpx.Client for download (simpler than async)
- Remove deprecated trust_remote_code parameter
Tested with successful corpus download and qrels extraction.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Use NextcloudClient with BasicAuth instead of raw httpx
- Replace direct HTTP POST with notes.create_note() method
- Add close() method to LLMProvider Protocol for proper cleanup
- Fix type annotations for dataset iteration
This improves code reuse and consistency with the rest of the codebase.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add ADR-013 documenting RAG evaluation architecture
- Implement two-part evaluation: Context Recall (retrieval) + Answer Correctness (generation)
- Create Click CLI for ground truth generation and corpus upload
- Add pytest fixtures and tests for retrieval/generation quality
- Use BeIR/nfcorpus dataset with 5 selected test queries
- Support Ollama and Anthropic LLM providers
- Generate synthetic ground truth answers offline
- Add comprehensive documentation in tests/rag_evaluation/README.md
The framework separates one-time setup (generate/upload) from test execution,
making tests much faster (~6-12 min vs ~15-25 min per run).
Tests are manual only (not in CI) and require external LLM access.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Improve user comprehension by scaling RRF scores to match the intuitive
0-1 range used by other search algorithms.
## Problem
RRF (Reciprocal Rank Fusion) scores had a drastically different scale
than semantic/keyword/fuzzy scores:
- Semantic similarity: 0.0 to 1.0 (typical: 0.5-0.9)
- RRF scores: 0.0 to ~0.016 (typical: 0.005-0.015)
This caused user confusion - a score of 0.0078 looked terrible but was
actually excellent (near theoretical maximum).
## Solution
Normalize RRF scores using the formula:
`normalized_score = rrf_score * (rrf_k + 1) / total_weight`
Where:
- rrf_k = 60 (RRF constant)
- total_weight = sum of algorithm weights (default: 1.0)
**Example transformation:**
- Before: 0.0078 (confusing)
- After: 0.477 (intuitive)
## Changes
**nextcloud_mcp_server/search/hybrid.py:**
- Store total_weight as instance variable (line 63)
- Calculate normalization factor in _reciprocal_rank_fusion() (line 209)
- Apply normalization to all RRF scores (line 217)
- Preserve raw RRF score in metadata for debugging (line 222)
## Impact
**User Experience:**
- Hybrid search scores now comparable with semantic/keyword/fuzzy
- Score of 0.5 indicates good match across all algorithms
- Consistent scale improves score threshold usability
**Backward Compatibility:**
- Raw RRF scores preserved in metadata["rrf_score_raw"]
- Result ordering unchanged (normalization is linear transformation)
- Breaking change: Existing score thresholds need adjustment
**Performance:**
- Negligible overhead (single multiplication per result)
## Testing
Verified with nc_semantic_search and nc_semantic_search_answer:
- Hybrid scores now 0.47-0.7 range (was 0.003-0.011)
- Semantic scores unchanged (0.75)
- Result ordering preserved
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Move access verification from individual search algorithms to final output
stage, eliminating redundant API calls and improving performance.
## Changes
**New:**
- `search/verification.py`: Centralized verification using anyio task groups
- Deduplicates results by (doc_id, doc_type) before verification
- Verifies all unique documents in parallel using structured concurrency
- Filters out inaccessible documents in single pass
**Modified Search Algorithms:**
- `search/semantic.py`: Removed _deduplicate_and_verify() and _verify_document_access()
- `search/keyword.py`: Removed _verify_access() and parallel verification
- `search/fuzzy.py`: Removed _verify_access() and parallel verification
- `search/hybrid.py`: Removed nextcloud_client parameter passing
All algorithms now return unverified results from Qdrant payload.
**Modified Output Stages:**
- `server/semantic.py`: Added verify_search_results() call after search
- `auth/viz_routes.py`: Added verify_search_results() call after search
Both endpoints now verify access once at final stage with deduplication.
## Performance Impact
**Before:**
- Hybrid mode (limit=10): 30 API calls (10 per algorithm × 3 algorithms)
- Single algorithm: 10-20 API calls (with verification buffer)
**After:**
- Hybrid mode (limit=10): 10 API calls (deduplicated verification)
- Single algorithm: 10 API calls (deduplicated verification)
**Performance Gain:** 3x reduction in API calls for hybrid search
## Architecture Benefits
- **Separation of concerns**: Algorithms handle scoring, output stage handles security
- **Deduplication**: Each document verified exactly once
- **Parallel execution**: All verifications run concurrently via anyio task groups
- **Consistency**: Same verification logic across MCP tools and viz endpoints
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Vector Visualization Improvements:
- Add interactive vector viz tab with Alpine.js and Plotly.js to user info page
- Refactor viz route CSS for better scoping and maintainability
- Remove unused nextcloud_host variable
Performance Optimizations:
- Parallelize access verification in fuzzy and keyword search algorithms
- Use asyncio.gather() to verify multiple documents concurrently
- Add exception handling with return_exceptions=True for resilience
Dependencies:
- Update third_party/oidc submodule to include RFC 9728 resource_url support
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Skip tracing for /app/vector-sync/status to reduce noise from HTMX polling.
Metrics collection continues for this endpoint.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Move Webhooks tab to the right (User Info | Vector Sync | Vector Viz | Webhooks)
- Use request.user.display_name instead of session for viz routes
- Fixes session middleware error when accessing via iframe
- Add /app/vector-viz endpoint for interactive search testing
- Implement server-side PCA dimensionality reduction (768-dim → 2D)
- Support multi-select document type filter for cross-app search
- Support all search algorithms: semantic, keyword, fuzzy, hybrid
- Display 2D scatter plot of vector embeddings using Plotly
- Show search results with scores and document types
- Register viz routes in app.py
- Add custom PCA implementation using numpy eigendecomposition
- Replace sklearn.decomposition.PCA with custom implementation
- Maintains same API (fit, transform, fit_transform)
- Supports explained_variance_ratio_ for variance analysis
- Removes scikit-learn dependency from project
- Add type hints and assertion for type safety
BREAKING CHANGE: Search algorithms now require Qdrant to be populated.
Vector sync must be enabled and documents indexed for search to work.
- Keyword and fuzzy search now query Qdrant scroll API for title/excerpt
- Remove inefficient Nextcloud API fetching pattern
- Add optional Nextcloud verification for security
- Deduplicate by (doc_id, doc_type) tuple, keeping chunk_index=0
- Align with document processor pattern that already stores text in Qdrant
Implements NextcloudClientProtocol for multi-document type search following
user requirement that document types are not 1:1 with apps (e.g., Notes app
specializes in markdown, while Files/WebDAV handles multiple file types).
Key Changes:
- NextcloudClientProtocol: Generic protocol with app-specific client properties
- get_indexed_doc_types(): Query Qdrant for actually-indexed document types
- Document dispatch: All algorithms check Qdrant before attempting access
- Cross-type deduplication: Use (doc_id, doc_type) tuples in hybrid RRF
Search Algorithm Updates:
- Semantic: Added _verify_document_access() with dispatch to appropriate client
- Deduplication by (doc_id, doc_type) tuple
- Only "note" verification implemented, others return None with info log
- Keyword: Added _fetch_documents() dispatch method
- Queries Qdrant for available types before fetching
- Supports cross-app search when doc_type=None
- Fuzzy: Same pattern as keyword search
- Hybrid: Already uses (doc_id, doc_type) for deduplication (no changes needed)
Future-Proof Design:
- File/calendar verification stubs in place
- Clear logging when unsupported types found
- Easy to extend when processor indexes new document types
Currently Supported:
- "note" documents fully implemented and tested
- Other types gracefully handled (logged but skipped)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implements ADR-012 by adding multi-algorithm support to the MCP tool.
Key changes:
- Added algorithm parameter: "semantic"|"keyword"|"fuzzy"|"hybrid" (default: "hybrid")
- Added weight parameters for hybrid mode configuration
- Replaced direct Qdrant/embedding calls with search module abstractions
- Updated docstring to describe all four algorithms
- Simplified implementation: ~50 lines vs ~150 lines (67% reduction)
- Better error handling for missing vector sync
Algorithm selection:
- semantic: Pure vector similarity (requires VECTOR_SYNC_ENABLED=true)
- keyword: Token-based matching with weighted title/content scoring
- fuzzy: Character overlap for typo tolerance
- hybrid: RRF fusion with configurable weights (default: 0.5/0.3/0.2)
Backward compatibility:
- Tool name unchanged (nc_semantic_search)
- New parameters have sensible defaults
- Existing clients get hybrid search automatically (better than pure semantic)
- search_method field in response reflects actual algorithm used
Weight validation:
- Performed in HybridSearchAlgorithm constructor
- Must sum to ≤1.0 and all non-negative
- At least one weight must be > 0
- Clear error messages on validation failure
Next: Update viz pane to use same algorithms
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Updates ADR-012 to clarify that all search and filtering operations
must happen server-side, not in the browser.
Key changes:
- Enhanced viz pane data flow showing server-side processing
- Added performance benefits section (384x bandwidth reduction)
- Detailed server-side filtering approach:
* Query execution via search/algorithms.py
* User ID filtering (multi-tenant security)
* Document type filtering
* PCA reduction (768-dim → 2D) on server
* Only 2D coordinates + metadata sent to client
- Updated Phase 3 implementation plan:
* Remove ALL client-side search logic
* Implement /app/vector-viz server endpoint
* htmx form submission for queries
* Performance optimizations (caching, streaming)
This ensures:
- Minimal bandwidth usage (only 2 floats per doc vs 768)
- Client handles only visualization, not computation
- Can visualize 10,000+ documents without client lag
- Raw vectors never leave server (security)
- Same search logic as MCP tool (consistency)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Enhances ADR-012 with detailed architecture visualization and UI mockup
for the vector visualization pane.
Added sections:
- Architecture diagram showing MCP tool and viz pane integration
- Data flow diagrams for both MCP requests and viz pane interactions
- Detailed UI mockup with ASCII art showing:
* Search configuration controls
* Algorithm selector with weight sliders
* Interactive 2D scatter plot (Plotly.js)
* Results panel with scores
* Performance comparison table
- Technology stack details (htmx, Alpine.js, Plotly.js, Tailwind CSS)
The diagrams illustrate how the viz pane and MCP tool share the same
search algorithm implementations from search/algorithms.py, ensuring
consistency between user testing interface and programmatic API.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Changes:
- Remove streamable-http transport override from mcp service in docker-compose.yml
- Service now uses CLI default SSE transport on /sse endpoint
- Add create_mcp_client_session_sse() helper for SSE connections
- Update nc_mcp_client fixture to use SSE transport
- Fix unpacking for SSE client (yields 2 values vs 3 for streamable-http)
Testing:
- All 4 smoke tests pass with SSE transport
- 32/34 affected tests pass (2 skipped for vector sync)
- OAuth services remain on streamable-http (unchanged)
Note: SSE transport is being deprecated in favor of streamable-http.
This enables minimal validation testing before deprecation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
After comprehensive research, the hybrid OAuth + AppAPI architecture is NOT
being implemented due to fundamental architectural incompatibilities.
Key updates:
- Status: Proposed → Not Planned
- Added validation from Nextcloud Context Agent project
- Context Agent (official NC ExApp with MCP) faces IDENTICAL limitations
- Proves constraints are architectural, not implementation-specific
Context Agent findings:
- ExApp with MCP server endpoint (~28 tools exposed)
- Uses Task Processing API for confirmations (NOT MCP elicitation)
- Works around AppAPI proxy limitations by changing protocol
- MCP endpoint is secondary feature with documented constraints
- Primary use: In-app Assistant integration, not external MCP clients
Critical features impossible through AppAPI proxy:
- ❌ MCP sampling (eliminates RAG/LLM features)
- ❌ MCP elicitation (user prompts)
- ❌ Real-time progress updates
- ❌ Bidirectional streaming
- Validated by Context Agent facing same limitations
Decision rationale:
- MCP requires multi-turn nested interactions
- AppAPI provides stateless request/response proxy only
- No implementation effort can bridge this fundamental gap
- Would require complete AppAPI redesign (WebSocket, message routing)
- Even official Nextcloud projects work around these limitations
Alternative considered for future:
- Register as Task Processing provider (different product)
- Use Nextcloud Assistant UI (not external MCP clients)
- Accept different capabilities (no sampling, custom flows)
OAuth mode remains sole solution for external MCP client integration.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Previously, an empty query string to nc_notes_search_notes would return
zero results due to an early return when no query tokens were present.
This was counterintuitive - users expect an empty query to list all
notes, not return nothing.
Changes:
- Modified NotesSearchController.search_notes() to return all notes
when query is empty
- Added documentation to clarify this behavior
- Empty query results have _score: None (no relevance scoring)
- Non-empty query results continue to have relevance scores
Fixes behavior where listing all notes was impossible via the search tool.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fixes#296
The application code was looking for OIDC_CLIENT_ID and OIDC_CLIENT_SECRET
(without NEXTCLOUD_ prefix), but the Helm chart, documentation, and CLI
all use NEXTCLOUD_OIDC_CLIENT_ID and NEXTCLOUD_OIDC_CLIENT_SECRET.
This mismatch caused OAuth deployments via Helm to fail with crashloops
because the credentials weren't being found.
Changes:
- app.py: Use NEXTCLOUD_OIDC_CLIENT_ID/SECRET in setup_oauth_config()
- config.py: Use NEXTCLOUD_OIDC_CLIENT_ID/SECRET in get_settings()
- Updated documentation comments and error messages
This aligns with the documented naming convention where all Nextcloud-related
environment variables use the NEXTCLOUD_ prefix.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Created @instrument_tool decorator for automatic MCP tool metrics collection.
Applied to all 7 tools in notes.py.
Changes:
- observability/metrics.py:
* New instrument_tool() decorator for automatic timing and error tracking
* Compatible with @mcp.tool() and @require_scopes() decorators
* Records tool_name, duration, and success/error status
- server/notes.py:
* Applied @instrument_tool to all 7 tool functions
* nc_notes_create_note, nc_notes_update_note, nc_notes_append_content
* nc_notes_search_notes, nc_notes_get_note, nc_notes_get_attachment
* nc_notes_delete_note
These metrics will populate the MCP Tool Calls dashboard panels.
Part of PR #295 - Complete metrics instrumentation (Phase 5)
Remaining: 86 tools across 8 server files
This ADR documents the architectural decision to support both OAuth and
AppAPI (ExApp) deployment modes in a single codebase with 90%+ code sharing.
Key additions:
- Comprehensive analysis of AppAPI limitations and challenges
- Feature parity matrix comparing OAuth vs AppAPI modes
- Resolution of critical open questions via research:
* Non-browser client authentication (app passwords/OAuth)
* Streaming transport compatibility (buffered, not real-time)
* Callbacks/webhooks (MCP notifications not possible in AppAPI)
- Detailed implementation plan with 4 phases (10 days)
- Mode-aware architecture with abstraction layer
Critical findings:
- AppAPI mode does NOT support MCP sampling (RAG features)
- No real-time progress updates (use Nextcloud notifications)
- Buffered streaming only (Streamable HTTP works, WebSocket doesn't)
- Requires app password support in AppAPI proxy
Deployment mode selection:
- OAuth: Multi-tenant, external clients, sampling/RAG, real-time updates
- AppAPI: Single-tenant, simplified install, native UI, admin-controlled
Related to investigation of ~/Software/app_api/ and ~/Software/nc_py_api/
for AppAPI integration patterns.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fixes Kubernetes label validation error when deploying dashboard ConfigMap.
Problem:
- Kubernetes labels cannot contain spaces (validation regex: [A-Za-z0-9][-A-Za-z0-9_.]*[A-Za-z0-9])
- Previous implementation had grafana_folder: "Nextcloud MCP" as a label
- Deployment failed with: "Invalid value: 'Nextcloud MCP'"
Solution:
- Move grafana_folder from labels to annotations (annotations allow spaces)
- Keep grafana_dashboard="1" as label for ConfigMap discovery
- Grafana sidecar reads folder name from folderAnnotation parameter
Changes:
- dashboard-configmap.yaml: Move grafana_folder to annotations section
- dashboards/README.md: Fix kubectl commands to use annotations
- values.yaml: Update comments to clarify annotation usage
This follows the standard kube-prometheus-stack pattern where:
- Labels are used for ConfigMap discovery (strict validation)
- Annotations are used for metadata like folder names (relaxed validation)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This fixes dimension mismatch errors when using embedding models with
non-standard dimensions (e.g., qwen3-embedding:4b produces 2560-dim
vectors instead of the hardcoded 768).
Changes:
- OllamaEmbeddingProvider: Detect dimensions dynamically by generating
test embedding instead of hardcoding to 768
- qdrant_client: Call dimension detection before collection creation
- app.py: Initialize Qdrant collection before starting background tasks
in streamable-http transport path
- tests: Fix integration tests to properly mock EmbeddingService wrapper
Fixes dimension mismatch error:
"could not broadcast input array from shape (2560,) into shape (768,)"
All integration tests passing (6/6).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fixes layout issues on the webhooks admin tab:
- Add min-height to container to fill viewport consistently
- Use CSS Grid to overlay tab panes without jumpiness
- Add smooth htmx fade transitions for content swaps
- Adjust vector sync polling interval from 3s to 10s
- Add .playwright-mcp/ to gitignore for test screenshots
The CSS Grid approach allows tabs to overlay without absolute positioning,
preventing content cutoff while maintaining smooth transitions without
container resizing jumps.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implement real-time vector sync status updates in the /app UI without
requiring page refreshes. The status (indexed documents, pending
documents, sync state) now updates automatically every 3 seconds.
Changes:
- Add vector_sync_status_fragment() endpoint that returns HTML fragment
with current vector sync status
- Modify user_info_html() to use htmx loading for vector sync section
with hx-trigger="load" on initial render
- Status fragment includes hx-trigger="every 3s" for continuous polling
- Add /app/vector-sync/status route to browser_routes
The implementation uses htmx (already loaded on page) to poll the status
endpoint, providing near real-time updates with minimal overhead. The
endpoint queries Qdrant for indexed count and reads from memory streams
for pending count, returning only the status HTML fragment.
Pattern follows existing webhook management UI which also uses htmx
for dynamic loading.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Simplified the webapp routing structure by consolidating the admin UI
to a single clean endpoint.
Changes:
- Moved webapp from /user/page to /app (root of mount)
- Removed /user JSON endpoint (no longer needed)
- Updated mount point from /user to /app in app.py
- Updated all route path checks (3 locations)
- Updated OAuth redirects to point to /app
- Updated all HTMX endpoint references
- Updated documentation (ADR-007, CHANGELOG)
- Added redirect from /app to /app/ for trailing slash handling
New Route Structure:
- /app - Main webapp (HTML UI with tabs)
- /app/revoke - Revoke background access
- /app/webhooks - Webhook management UI
- /app/webhooks/enable/{preset_id} - Enable webhook preset
- /app/webhooks/disable/{preset_id} - Disable webhook preset
Breaking Change: Existing bookmarks to /user or /user/page will no longer work.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Refactored the storage system to use a unified SQLite database for both
webhook tracking and OAuth token storage, available in both BasicAuth
and OAuth modes.
Changes:
- Renamed refresh_token_storage.py → storage.py
- Made TOKEN_ENCRYPTION_KEY optional (only required for OAuth token ops)
- Added registered_webhooks table with schema versioning
- Added webhook storage methods (store, get, delete, list, clear)
- Initialize storage in both BasicAuth and OAuth modes
- Updated webhook routes to persist registrations in database
- Database-first pattern for webhook status checks (performance)
- Updated all imports across codebase
Storage Behavior:
- Database created automatically at startup if needed
- Existing databases detected and reused
- Server fails fast if database initialization fails
- No migrations needed (OAuth feature is experimental)
Testing:
- Added 13 comprehensive unit tests for webhook storage
- All 118 unit tests pass
- All 5 smoke tests pass
- Verified fail-fast behavior on initialization errors
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Manual testing of Nextcloud webhook_listeners app to validate webhook
payloads against ADR-010 expected schemas and document implementation
requirements for webhook-based vector synchronization.
## Changes
- Add test webhook endpoint at /webhooks/nextcloud in app.py
- Captures and logs webhook payloads for analysis
- Returns 200 OK immediately for webhook delivery confirmation
- Create webhook-testing-findings.md with comprehensive test results
- Captured payloads for 5/6 webhook event types
- Critical findings: missing node.id in deletions, type mismatches
- Implementation recommendations with code examples
- Update ADR-010 with Appendix A: Manual Webhook Testing Results
- Document actual vs expected webhook behavior
- Update event mapping table with tested webhook status
- Add 6 specific implementation recommendations
- Include testing implications for future development
## Testing Results
✅ NodeCreatedEvent - fires correctly, includes node.id (integer)
✅ NodeWrittenEvent - fires correctly, includes node.id (integer)
✅ NodeDeletedEvent - fires but missing node.id field (path only)
✅ CalendarObjectCreatedEvent - fires correctly with full iCal
✅ CalendarObjectUpdatedEvent - fires correctly with full iCal
❌ CalendarObjectDeletedEvent - does not fire (potential NC bug)
## Key Findings
1. NodeDeletedEvent missing node.id field - requires path-based fallback
2. node.id returns integer not string - needs casting for consistency
3. Multiple webhooks fire per operation - needs deduplication logic
4. Calendar deletion webhooks don't fire - reported as issue #53497
5. Calendar webhooks include full iCal content - enables rich parsing
## GitHub Issues
- Created issue #56371: NodeDeletedEvent missing node.id field
- Commented on issue #53497: CalendarObjectDeletedEvent not firing
Closes#283
---
_This commit was generated with the help of AI, and reviewed by a Human_
Simplifies the OpenTelemetry tracing setup by removing the redundant
OTEL_ENABLED flag and using the presence of OTEL_EXPORTER_OTLP_ENDPOINT
to determine if tracing should be enabled. This follows the standard
OpenTelemetry environment variable conventions more closely.
Changes:
- Remove OTEL_ENABLED/tracing_enabled flag in favor of checking if
OTEL_EXPORTER_OTLP_ENDPOINT is set
- Add OTEL_EXPORTER_VERIFY_SSL configuration option for OTLP endpoints
with self-signed certificates (defaults to false for development)
- Move HTTPXClientInstrumentor initialization to module level to ensure
httpx calls are traced across all Nextcloud API requests
- Add tracing spans to vector sync operations (scan_user_documents)
- Fix authorization header logging to only warn about missing headers
in OAuth mode (BasicAuth mode doesn't use Authorization headers)
- Update observability documentation to reflect simplified configuration
- Refactor Dockerfile to use --no-editable flag for uv sync
Breaking changes:
- OTEL_ENABLED environment variable is removed
- Tracing is now automatically enabled when OTEL_EXPORTER_OTLP_ENDPOINT
is set
Migration guide:
- Remove OTEL_ENABLED=true from environment configuration
- Tracing will be enabled automatically if OTEL_EXPORTER_OTLP_ENDPOINT
is configured
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The test_attachments_category_change_handling test was failing in CI with
HTTP 412 Precondition Failed errors. This is caused by the background vector
scanner (runs every 10 seconds) modifying notes between when the test fetches
the ETag and when it attempts to update the category.
Solution: Added retry logic (up to 3 attempts) that refetches the latest ETag
and retries the update operation when encountering 412 errors. This handles
the race condition gracefully while still catching genuine errors.
Health check and metrics endpoints are frequently polled and don't
provide meaningful trace data. This change skips OpenTelemetry span
creation for:
- /health/* (liveness, readiness checks)
- /metrics (Prometheus metrics)
These endpoints still record Prometheus metrics (request count, latency,
in-flight requests) but no longer create trace spans, reducing tracing
noise and storage costs.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The Nextcloud Notes API intentionally returns all note IDs (with only 'id'
field) in the last chunk to enable deletion detection. Without using the
pruneBefore parameter, this causes duplicates - all notes appear with full
data in chunks, then again with minimal data in the last chunk.
This commit implements proper pruneBefore support:
- NotesClient.get_all_notes() now accepts prune_before timestamp parameter
- Scanner calculates max(indexed_at) from Qdrant to use as prune threshold
- Only notes modified after this timestamp are sent with full data
- Deduplication logic handles the API's deletion detection pattern
- Significantly reduces data transfer for incremental syncs
The behavior is documented in Notes API v1 spec - this is not an API bug,
but a feature we weren't utilizing correctly.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add architecture decision record for integrating Nextcloud webhooks
into the vector database synchronization system.
Key features:
- Webhook endpoint at /webhooks/nextcloud receives push notifications
- Complements existing polling (ADR-007) without replacing it
- Optional authentication via WEBHOOK_SECRET
- Simple architecture: webhooks are just another DocumentTask producer
- Administrators can reduce polling frequency when webhooks are configured
Benefits:
- Reduced latency: seconds to minutes instead of up to 1 hour
- Lower API load: ~95% reduction when polling frequency is increased
- Better scalability: only process changed documents
- No changes required to scanner or processor components
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add support for configurable document chunking parameters to Helm chart
to match docker-compose and application capabilities.
Changes:
1. values.yaml:
- Add documentChunking section with chunkSize (512) and chunkOverlap (50)
- Include comprehensive comments explaining chunking strategies
- Positioned between vectorSync and qdrant sections
2. templates/deployment.yaml:
- Add DOCUMENT_CHUNK_SIZE and DOCUMENT_CHUNK_OVERLAP env vars
- Always set (not conditional), used by vector sync processor
- Environment variables follow same pattern as config.py defaults
3. README.md:
- Add documentChunking parameter table in Vector Search section
- Document chunking strategies (small/medium/large chunks)
- Explain overlap recommendations (10-20% of chunk size)
Validation:
- helm lint: Passes
- helm template: Environment variables correctly generated
- Custom values: Work as expected (tested with chunkSize=1024)
- Always present: Not conditional on vectorSync.enabled
This maintains feature parity between Helm and docker-compose deployments,
allowing users to tune chunking for their embedding models and use cases.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Changes to make tests work without external qdrant/ollama dependencies:
1. docker-compose.yml (mcp service):
- Switch from QDRANT_URL (network mode) to QDRANT_LOCATION=":memory:"
- Comment out QDRANT_URL and QDRANT_API_KEY (not needed for in-memory)
- Keep OLLAMA_BASE_URL commented out (use SimpleEmbeddingProvider fallback)
2. nextcloud_mcp_server/vector/qdrant_client.py:
- Fix collection creation bug in in-memory mode
- Previously: All ValueError exceptions were re-raised
- Now: Only dimension mismatch ValueError is re-raised
- Allows "Collection not found" ValueError to trigger auto-creation
3. tests/integration/test_sampling.py:
- Update test to handle all sampling unsupported cases
- Check for multiple fallback search_method values
- Skip test gracefully when sampling unavailable
This configuration enables:
- CI testing without external services (qdrant, ollama)
- In-memory vector database (ephemeral but sufficient for tests)
- SimpleEmbeddingProvider for embeddings (feature hashing, 384 dims)
- Automatic collection creation on first use
Test result: test_semantic_search_answer_successful_sampling now passes
(skipped with appropriate message when sampling unsupported)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This PR enables safe switching between embedding models and multi-server
deployments by implementing auto-generated Qdrant collection names based on
deployment ID and model name.
## Problem
Previously, all deployments used a single hardcoded collection name
"nextcloud_content", which caused two critical issues:
1. **Dimension mismatches when switching models**: Changing
OLLAMA_EMBEDDING_MODEL (e.g., nomic-embed-text at 768D → all-minilm at
384D) would cause runtime errors as vectors couldn't be inserted into a
collection with incompatible dimensions.
2. **Collection collisions in multi-server setups**: Multiple MCP servers
sharing a single Qdrant instance would overwrite each other's data,
making horizontal scaling impossible.
## Solution
### Auto-Generated Collection Naming
Collections are now automatically named using the pattern:
\`{deployment-id}-{model-name}\`
**Deployment ID**: Uses \`OTEL_SERVICE_NAME\` if configured (and not default
value), otherwise falls back to \`hostname\` for simple Docker deployments.
**Model Name**: From \`OLLAMA_EMBEDDING_MODEL\` with path separators sanitized.
**Examples**:
- \`my-mcp-server-nomic-embed-text\` (with OTEL_SERVICE_NAME=my-mcp-server)
- \`mcp-container-all-minilm\` (simple Docker, hostname=mcp-container)
**Override**: Users can still set \`QDRANT_COLLECTION\` explicitly to bypass
auto-generation for backward compatibility.
### Dimension Validation
Added startup validation that checks collection dimensions match the
embedding service. If a mismatch is detected, the server fails fast with a
clear error message explaining:
- Expected vs actual dimensions
- Likely cause (model change)
- Solutions (delete collection, use different name, or revert model)
### Improved Sampling Error Handling
Enhanced MCP sampling rejection handling to treat user rejections as normal
behavior rather than errors:
- **User rejections** ("rejected", "denied") → INFO log, no traceback
- **Unsupported clients** → INFO log, no traceback
- **Other MCP errors** → WARNING log, no traceback
- **Unexpected errors** → ERROR log WITH traceback
This aligns with the MCP specification where clients SHOULD prompt users for
approval/denial of sampling requests.
## Changes
### Core Implementation
- **nextcloud_mcp_server/config.py**: Added \`get_collection_name()\` method
with deployment ID detection and model name sanitization
- **nextcloud_mcp_server/vector/qdrant_client.py**: Dimension validation on
collection open with helpful error messages
- **nextcloud_mcp_server/vector/{scanner,processor}.py**: Updated to use
\`get_collection_name()\`
- **nextcloud_mcp_server/auth/userinfo_routes.py**: Vector sync status uses
\`get_collection_name()\`
- **nextcloud_mcp_server/server/semantic.py**:
- Updated semantic search tools to use \`get_collection_name()\`
- Improved sampling rejection error handling (McpError vs Exception)
### Documentation
- **docs/semantic-search-architecture.md**: New comprehensive architecture
document (557 lines) covering background sync, semantic search flow, RAG
implementation, and deployment modes
- **docs/configuration.md**: Added detailed "Qdrant Collection Naming"
section with examples and multi-server deployment guidance
- **docker-compose.yml**: Added comments explaining collection naming behavior
- **README.md**: Updated semantic search descriptions to clarify
experimental status, Notes-only support, and infrastructure requirements
## Migration Guide
**For existing single-server deployments:**
Option 1 (Recommended): Use explicit collection name for continuity
\`\`\`bash
QDRANT_COLLECTION=nextcloud_content # Keep existing collection
\`\`\`
Option 2: Allow auto-generation and re-embed
\`\`\`bash
# Remove QDRANT_COLLECTION override
# New collection will be created based on deployment ID + model
# Requires re-embedding all documents (may take time)
\`\`\`
**For new multi-server deployments:**
Set unique OTEL service names per server:
\`\`\`bash
# Server 1
OTEL_SERVICE_NAME=mcp-prod
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
# → Collection: "mcp-prod-nomic-embed-text"
# Server 2
OTEL_SERVICE_NAME=mcp-staging
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
# → Collection: "mcp-staging-nomic-embed-text"
\`\`\`
## Benefits
✅ **Safe model switching**: Each model gets its own collection, preventing
dimension mismatch errors
✅ **Multi-server support**: Multiple MCP servers can share one Qdrant
instance without conflicts
✅ **Clear ownership**: Collection names show which deployment and model owns
the data
✅ **Better error messages**: Dimension validation provides actionable
guidance
✅ **Backward compatible**: Existing deployments can continue using
\`QDRANT_COLLECTION\` override
## Testing
Validated with:
- Single-server deployments (default hostname-based naming)
- Multi-server deployments (OTEL service name-based naming)
- Model switching scenarios (dimension validation)
- Collection override scenarios (backward compatibility)
Next steps: Testing various Ollama embedding models to investigate optimal
chunk sizes and performance characteristics.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Security fix: Move Prometheus metrics endpoint from main HTTP port to
dedicated port 9090 to prevent external exposure of metrics data.
Changes:
- Use prometheus_client.start_http_server() for dedicated metrics server
- Remove /metrics route from main application routes
- Metrics now only accessible on port 9090 (configurable via METRICS_PORT)
- Main application port no longer serves /metrics endpoint
This follows security best practice of isolating monitoring endpoints
from application traffic.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The readiness probe incorrectly tried to connect to an external Qdrant service
even when using memory or persistent mode (embedded Qdrant). This caused pods
to never become ready in Kubernetes deployments using the default configuration.
Root cause:
- In memory/persistent modes, QDRANT_URL env var is NOT set
- Readiness check used default 'http://qdrant:6333' anyway
- Tried to connect to non-existent service
- Connection failed -> 503 -> pod stuck in not-ready state
Fix:
- Only check external Qdrant health if QDRANT_URL is explicitly set (network mode)
- For embedded modes (memory/persistent), report status as 'embedded' without blocking
- Background scanner tasks don't block readiness (already non-blocking via anyio.start_soon)
This allows pods to become ready immediately when using embedded Qdrant,
while still validating external Qdrant connectivity in network mode.
Fixes: Kubernetes pods failing readiness check with default Qdrant configuration
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The vector scanner crashed when encountering notes without a 'modified' field,
causing KeyError and preventing initial sync from completing.
Changes:
- Use dict.get() with fallback value (0) instead of direct key access
- Log warnings for notes missing 'modified' field
- Apply fix to both initial sync and incremental sync code paths
This ensures the scanner continues processing all notes even if some have
missing metadata fields, preventing scanner crashes that could affect
deployment readiness.
Fixes: Notes without 'modified' field causing scanner crash and readiness check failure
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add Prometheus metrics for HTTP, MCP tools, Nextcloud API, OAuth, vector sync, and DB operations
- Add OpenTelemetry distributed tracing with OTLP export
- Add structured JSON logging with trace context correlation
- Add ObservabilityMiddleware for automatic HTTP instrumentation
- Add app_name attribute to all client classes for per-app metrics
- Add configuration for metrics, tracing, and logging via environment variables
- Add documentation in docs/observability.md
- Fix graceful degradation when tracing is disabled (default state)
- Fix uvicorn logging configuration to use observability formatters
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The Qdrant subchart was being included by default even in memory/persistent
modes. Changed the dependency condition from `qdrant.enabled` to
`qdrant.networkMode.deploySubchart` to align with the three-mode structure.
Now the Qdrant subchart is ONLY deployed when:
- qdrant.mode: "network"
- qdrant.networkMode.deploySubchart: true
Verified all three modes:
- Memory mode (:memory:): No subchart, QDRANT_LOCATION=:memory:
- Persistent mode (path): No subchart, QDRANT_LOCATION=/app/data/qdrant, PVC created
- Network mode (subchart): Qdrant subchart deployed, QDRANT_URL=http://...:6333
- Network mode (external): No subchart, QDRANT_URL=<external-url>
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The chart-releaser was failing because it couldn't resolve the
dependencies (Qdrant and Ollama subcharts) when packaging.
Changes:
- Add azure/setup-helm action to install Helm v3.16.0
- Add step to add Qdrant and Ollama Helm repositories
- Run helm dependency update before chart-releaser runs
This fixes the error:
"Error: no repository definition for https://qdrant.github.io/qdrant-helm, https://otwld.github.io/ollama-helm"
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add support for three Qdrant deployment modes in Helm chart:
1. In-memory mode (:memory:) - Default, zero-config, ephemeral storage
2. Persistent local mode (path-based) - File-based storage with PVC
3. Network mode (URL-based) - Dedicated Qdrant service or external instance
Changes:
- Restructured qdrant configuration in values.yaml with mode selector
- Added conditional environment variable logic in deployment.yaml
- Created PVC template for persistent local mode with optional existingClaim
- Added qdrantPvcName helper template in _helpers.tpl
- Updated README.md with Helm registry URL (https://cbcoutinho.github.io/nextcloud-mcp-server)
Breaking change: Default changed from requiring qdrant.enabled to using
in-memory mode (:memory:) when no Qdrant configuration is provided.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Adds flexible Qdrant deployment modes to reduce infrastructure requirements
for local development and smaller deployments:
**Configuration Changes:**
- Add QDRANT_LOCATION environment variable (mutually exclusive with QDRANT_URL)
- Three modes: network (URL), in-memory (:memory:, default), persistent (file path)
- Settings dataclass validation via __post_init__ ensures mutual exclusivity
- API key warning when set in local mode (ignored, only for network mode)
**Client Initialization:**
- Auto-detect mode: network (url + api_key) vs local (:memory: or path=)
- In-memory: AsyncQdrantClient(":memory:") - zero config default
- Persistent: AsyncQdrantClient(path="/app/data/qdrant") - file storage
- Network: AsyncQdrantClient(url, api_key) - production mode
**Docker Compose Updates:**
- Qdrant service moved to optional profile (--profile qdrant)
- MCP service uses QDRANT_LOCATION=:memory: by default
- Added mcp-data volume for persistent storage (/app/data)
- No hard dependency on qdrant service
**Documentation:**
- Comprehensive configuration guide in docs/configuration.md
- All three modes documented with pros/cons
- Docker Compose examples for each mode
- Environment variable reference table
**Tests:**
- 13 new config validation tests (mutual exclusivity, defaults, warnings)
- Persistent mode integration test (create, close, reopen, verify persistence)
- All 82 unit tests + 5 smoke tests pass
**Breaking Change:**
- Default changed from QDRANT_URL=http://qdrant:6333 to QDRANT_LOCATION=:memory:
- Simplifies local development (no external service needed)
- Production deployments: explicitly set QDRANT_URL or QDRANT_LOCATION
Related: ADR-007 background vector sync implementation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Replace asyncio.Queue with anyio.create_memory_object_stream() throughout
the vector sync system for better library consistency and improved shutdown
semantics.
## Changes Made
**scanner.py**:
- Changed parameter type from `asyncio.Queue` to `MemoryObjectSendStream[DocumentTask]`
- Replaced all `await document_queue.put()` calls with `await send_stream.send()`
- Wrapped scanner loop in `async with send_stream:` context manager for automatic cleanup
- Updated log messages: "Queued" → "Sent"
- Removed `import asyncio` (no longer needed)
**processor.py**:
- Changed parameter type from `asyncio.Queue` to `MemoryObjectReceiveStream[DocumentTask]`
- Replaced `asyncio.wait_for(document_queue.get(), timeout=1.0)` with `anyio.fail_after(1.0)` + `await receive_stream.receive()`
- Removed all `document_queue.task_done()` calls (not needed with streams)
- Added `anyio.EndOfStream` exception handling for graceful shutdown when scanner closes
- Removed `import asyncio` (no longer needed)
**app.py**:
- Removed `import asyncio` from top-level imports
- Added `from anyio.streams.memory import MemoryObjectReceiveStream, MemoryObjectSendStream`
- Updated AppContext dataclass:
- Replaced `document_queue: Optional[asyncio.Queue]` with:
- `document_send_stream: Optional[MemoryObjectSendStream]`
- `document_receive_stream: Optional[MemoryObjectReceiveStream]`
- Updated `app_lifespan_basic()`:
- Replaced `asyncio.Queue(maxsize=...)` with `anyio.create_memory_object_stream(max_buffer_size=...)`
- Pass `send_stream` to scanner_task
- Pass `receive_stream.clone()` to each processor_task (enables multiple consumers)
- Updated AppContext yield to include both streams
- Updated `starlette_lifespan()`:
- Same changes as app_lifespan_basic for streamable-http transport
- Removed `import asyncio as asyncio_module` (no longer needed)
- Updated app.state storage to use send_stream and receive_stream
**semantic.py**:
- Updated `nc_get_vector_sync_status()` tool:
- Access `document_receive_stream` instead of `document_queue` from lifespan context
- Use `stream_stats.current_buffer_used` instead of `queue.qsize()` for pending count
- More reliable metrics (qsize() was not guaranteed accurate)
## Benefits
1. **Library Consistency**: Pure anyio throughout codebase (was mixing asyncio.Queue with anyio.Event and anyio.create_task_group)
2. **Graceful Shutdown**: `async with send_stream:` automatically closes stream on exit, signaling EndOfStream to all processors
3. **Better Timeout Handling**: `anyio.fail_after()` is more idiomatic than `asyncio.wait_for()`
4. **Stream Cloning**: Easy to add multiple consumers via `receive_stream.clone()`
5. **Better Statistics**: `.statistics()` provides accurate buffer metrics (qsize() was unreliable)
6. **Type Safety**: Separate send/receive types prevent accidental misuse
7. **No task_done() tracking**: Streams handle completion automatically
## Testing
- ✅ All 69 unit tests passing
- ✅ All 5 smoke tests passing
- ✅ No regressions in functionality
- ✅ Graceful shutdown behavior improved
## References
- https://anyio.readthedocs.io/en/stable/why.html#queue-fix
- https://anyio.readthedocs.io/en/stable/streams.html#memory-object-streams🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This implements ADR-009, which documents the decision to use a generic
`semantic:read` OAuth scope instead of requiring all app-specific scopes
for semantic search functionality.
Changes:
- Created new `nextcloud_mcp_server/models/semantic.py` with semantic search models
- SemanticSearchResult (with new doc_type field for multi-app support)
- SemanticSearchResponse
- SamplingSearchResponse
- VectorSyncStatusResponse
- Created new `nextcloud_mcp_server/server/semantic.py` with semantic search tools
- nc_semantic_search (renamed from nc_notes_semantic_search)
- nc_semantic_search_answer (renamed from nc_notes_semantic_search_answer)
- nc_get_vector_sync_status (renamed from nc_notes_get_vector_sync_status)
- All tools now use @require_scopes("semantic:read") instead of "notes:read"
- Updated `nextcloud_mcp_server/server/notes.py`
- Removed semantic search tools (moved to semantic.py)
- Removed semantic search model imports
- Removed unused MCP imports (ModelHint, ModelPreferences, etc.)
- Updated `nextcloud_mcp_server/models/notes.py`
- Removed semantic search models (moved to semantic.py)
- Updated `nextcloud_mcp_server/app.py`
- Import configure_semantic_tools
- Register semantic tools when VECTOR_SYNC_ENABLED=true
- Updated `nextcloud_mcp_server/server/__init__.py`
- Export configure_semantic_tools
- Updated tests
- tests/integration/test_sampling.py: Use new tool names
- tests/unit/test_response_models.py: Import from semantic.py, add doc_type field
Architecture:
- Semantic search is now a cross-app feature, not tied to Notes
- Uses dual-phase authorization: semantic:read scope + per-document verification
- Supports future multi-app indexing (notes, calendar, deck, files, contacts)
Test results:
- All 69 unit tests passing
- All 5 smoke tests passing
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Replace static VECTOR_SYNC_ENABLED_APPS environment variable with per-user
database storage for which apps to index. This allows each user to control
their own indexing preferences (e.g., enable notes and calendar but not
deck or files).
Rationale:
- Nextcloud doesn't support granular OAuth scopes at the app level
- Per-user settings provide flexibility for multi-user deployments
- Users control app enablement via nc_enable_vector_sync MCP tool
- Aligns with OAuth architecture where users manage their own settings
Changes:
- ADR-007: Remove VECTOR_SYNC_ENABLED_APPS from configuration section
- ADR-007: Update scanner implementation to read from database
- ADR-007: Add explanation of per-user app enablement mechanism
- ADR-007: Clarify that nc_enable_vector_sync tool manages this setting
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Update ADRs to reflect that vector database and semantic search support
multiple Nextcloud apps (notes, calendar, deck, files, contacts) rather
than being notes-specific. Introduce semantic:read/write OAuth scopes
to replace app-specific scope requirements for cross-app search.
Changes:
- ADR-007: Add plugin architecture (DocumentScanner, DocumentProcessor,
DocumentVerifier) for multi-app vector sync
- ADR-008: Rename tools from nc_notes_semantic_* to nc_semantic_*, update
scope from notes:read to semantic:read
- ADR-009: NEW - Document decision to use generic semantic:read scope
with dual-phase authorization instead of requiring all app scopes
- oauth-architecture.md: Add semantic:read/write scope documentation
- README.md: Move semantic search to dedicated section separate from Notes
This is a breaking change that correctly positions semantic search as a
cross-app capability before broader adoption. Existing deployments will
need to re-authenticate with the new semantic:read scope.
Relates to user request to decouple vector database from notes-only model
and establish proper OAuth scope boundaries for multi-app semantic search.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit addresses issues with vector database synchronization that
were causing test failures:
1. **Deletion Grace Period** (scanner.py)
- Fixed premature deletion of documents due to pagination cursor
inconsistencies in Notes API
- Implemented 2-scan verification with 1.5x scan interval grace period
(15 seconds default)
- Documents must be missing for 2 consecutive scans before deletion
- Documents that reappear are removed from deletion tracking
- Prevents false deletions during concurrent note creation/indexing
2. **Vector Sync Status Tool** (server/notes.py, models/notes.py)
- Added nc_notes_get_vector_sync_status MCP tool
- Returns indexed_count, pending_count, status, and enabled fields
- Enables tests and clients to wait for vector sync completion
- Uses lifespan context to access document queue and Qdrant client
3. **Test Improvements** (test_sampling.py, conftest.py)
- Added temporary_note_factory fixture for creating multiple test notes
- Updated all sampling tests to wait for vector sync completion
- Adjusted score_threshold to 0.0 for SimpleEmbeddingProvider
(feature hashing produces low-quality embeddings)
- Fixed CallToolResult extraction (removed ["result"] key access)
- Removed invalid @pytest.mark.asyncio markers (anyio mode)
All integration tests now pass successfully.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add nc_notes_semantic_search_answer tool that combines semantic search
with MCP sampling to generate natural language answers from retrieved
Nextcloud Notes. This enables Retrieval-Augmented Generation (RAG)
patterns without requiring a server-side LLM.
Key features:
- Client-side LLM generation via ctx.session.create_message()
- Graceful fallback when sampling unavailable
- Proper source citations in generated answers
- No results optimization (skips sampling when no docs found)
- Comprehensive unit and integration tests
Implementation details:
- SamplingSearchResponse model with generated_answer and sources
- Fixed prompt template with document context and citation instructions
- Model preferences hint Claude Sonnet for balanced performance
- Falls back to returning documents without answer on sampling failure
Updates:
- Add ADR-008 documenting sampling architecture decision
- Add MCP sampling pattern guidance to CLAUDE.md
- Update README.md and docs/notes.md (7 → 9 tools)
- Add 4 unit tests and 6 integration tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add support for deploying Qdrant vector database and Ollama embedding
service as optional helm chart dependencies. Enables semantic search
capabilities for Nextcloud content with flexible deployment options.
Chart Dependencies:
- Add Qdrant v0.9.0 from qdrant/qdrant-helm (conditional)
- Add Ollama v1.33.0 from otwld/ollama-helm (conditional)
- Both dependencies only deploy when enabled
Configuration (values.yaml):
- vectorSync: Background sync settings (interval, workers, queue size)
- qdrant: Subchart config with persistence, resources, clustering
- ollama: Subchart config with model pull, persistence, resources
- Support for external Ollama via ollama.url (no subchart deployment)
- openai: Alternative embedding provider (OpenAI or compatible API)
Environment Variables (deployment.yaml):
- VECTOR_SYNC_* variables when vectorSync.enabled
- QDRANT_URL, QDRANT_COLLECTION when qdrant.enabled
- OLLAMA_BASE_URL, OLLAMA_EMBEDDING_MODEL when ollama enabled or URL set
- OPENAI_API_KEY when openai.enabled
Documentation:
- README: New "Vector Search & Semantic Capabilities" section
- README: Example 5 showing three deployment patterns
- NOTES.txt: Conditional guidance when vector features enabled
- Secret template for OpenAI API key management
All features disabled by default for backward compatibility.
Tested with helm template and helm lint.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add real-time processing status display to the browser UI at /user/page
showing indexed document count, pending queue size, and sync status.
Implements the status display described in ADR-007 lines 280-298.
Changes:
- Store document_queue and related state in app.state for route access
- Add _get_processing_status() helper to query Qdrant and check queue
- Display status section in user_info_html() with indexed/pending counts
- Show color-coded status badge (green "Idle" or orange "Syncing")
- Only displays when VECTOR_SYNC_ENABLED=true
Status appears in both BasicAuth and OAuth modes, positioned after
session info but before logout buttons. Numbers are formatted with
commas for readability.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Replace deprecated qdrant_client.search() with query_points() API
- Update semantic search implementation in notes.py
- Update all integration tests to use query_points()
- Fix Keycloak login in test_keycloak_dcr.py to use form.submit() instead of button click
- Remove unnecessary popup handler code
- Simplify consent screen logging
The urllib3<2.0 constraint was added unnecessarily during troubleshooting.
urllib3 2.x works perfectly fine with qdrant-client. The import path for
urllib3.util.Url and parse_url remains the same across 1.x and 2.x versions.
Changes:
- Remove urllib3<2.0 constraint from pyproject.toml
- Upgrade to urllib3 2.5.0 (latest)
- All integration tests pass with urllib3 2.x
Verified:
- from urllib3.util import Url, parse_url works in 2.5.0
- All 6 semantic search integration tests pass
- qdrant-client 1.15.1 works correctly with urllib3 2.5.0
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Adds comprehensive integration tests for vector database semantic search that
work without external dependencies (Ollama), making them suitable for CI/CD.
Changes:
- Add SimpleEmbeddingProvider: in-process TF-IDF-like embeddings using feature hashing
- Make Ollama optional: embedding service now falls back to SimpleEmbeddingProvider
- Add 6 integration tests covering semantic search, filtering, and batch operations
- Downgrade urllib3 to 1.26.x for qdrant-client compatibility
- Update docker-compose.yml to comment out Ollama configuration (optional)
The SimpleEmbeddingProvider generates deterministic, normalized embeddings
suitable for testing semantic similarity without requiring external services.
Tests validate that similar texts have higher cosine similarity and that
semantic search correctly ranks results by relevance.
Test coverage:
- Deterministic embedding generation
- Semantic similarity between texts
- Full search flow with Qdrant (in-memory)
- Category filtering
- Empty result handling
- Batch embedding generation
All tests pass and can run in GitHub CI without Ollama infrastructure.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Completes the ADR-007 implementation by adding user-facing semantic search
functionality. Previous phases implemented scanner and processor for background
indexing; this adds the query interface.
Changes:
- Add nc_notes_semantic_search MCP tool for natural language queries
- Fix Qdrant point IDs to use UUIDs instead of strings (was causing 400 errors)
- Reduce scan interval default from 1 hour to 5 minutes for faster updates
- Add SemanticSearchResult and SemanticSearchNotesResponse models
- Implement dual-phase authorization (Qdrant filter + Nextcloud API verification)
The semantic search enables finding notes by meaning rather than exact keywords,
using vector embeddings to understand query intent. Point ID fix resolves
critical bug where all document indexing failed with "invalid point ID" errors.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Chris Coutinho