Commit Graph
7 Commits
Author SHA1 Message Date
Chris CoutinhoandClaude 4ea5ed72d4 feat: Add Grafana dashboard and vector sync metric instrumentation
Implement comprehensive observability for vector database synchronization
with Grafana dashboard and Prometheus metrics.

## Part 1: Grafana Dashboard

Created all-in-one operations dashboard with 7 rows and 34 panels:

### Dashboard Structure:
- **Overview Row**: Request rate, error rate, P95 latency, active requests
- **HTTP Metrics (RED)**: Request/error rates by endpoint, latency percentiles
- **MCP Tools**: Call volume, error rates, execution duration by tool
- **Nextcloud API**: API calls/latency by app, retry patterns
- **OAuth & Authentication**: Token validations, exchanges, cache hit rate
- **Dependencies & Health**: Status for Nextcloud/Qdrant/Keycloak/Unstructured
- **Vector Sync**: Processing throughput, queue depth, Qdrant operations

### Helm Chart Integration:
- Added dashboard-configmap.yaml template for automatic provisioning
- Configured Grafana sidecar auto-discovery (label: grafana_dashboard="1")
- Added dashboards configuration section in values.yaml (opt-in)
- Updated Chart.yaml with dashboard annotations
- Enhanced NOTES.txt with dashboard deployment instructions
- Comprehensive documentation in dashboards/README.md

Dashboard supports dynamic filtering via variables:
- datasource: Prometheus data source selection
- namespace: Filter by Kubernetes namespace
- pod: Multi-select pod filtering
- interval: Query interval (1m/5m/10m/30m/1h)

## Part 2: Vector Sync Metric Instrumentation

Implemented metric recording throughout vector sync pipeline:

### metrics.py:
Added convenience functions:
- record_vector_sync_scan() - Track documents per scan
- record_vector_sync_processing() - Track processing duration/status
- record_qdrant_operation() - Track database operations
- update_vector_sync_queue_size() - Track queue depth

### scanner.py:
- Record number of documents found in each scan
- Enables monitoring of scan throughput

### processor.py:
- Record processing duration for each document
- Track success/failure status with timing
- Record Qdrant upsert/delete operations
- Handle all code paths (success, deletion, error)

### semantic.py:
- Wrap Qdrant query_points with try/except
- Record search operation success/failure

## Metrics Exposed:

- mcp_vector_sync_documents_scanned_total
- mcp_vector_sync_documents_processed_total{status}
- mcp_vector_sync_processing_duration_seconds (histogram)
- mcp_vector_sync_queue_size (gauge)
- mcp_qdrant_operations_total{operation,status}

This enables monitoring of:
- Scan and processing throughput
- Processing latency (P50/P95/P99)
- Error rates for processing and Qdrant operations
- Queue depth trends
- Complete observability of vector sync pipeline

## Testing:

Verified locally that metrics are recorded correctly:
- 36 documents scanned
- 3 documents processed (avg 7.5s each)
- 3 successful Qdrant upsert operations
- Search operations tracked

## Deployment:

Enable dashboard provisioning in Helm values:
```yaml
dashboards:
  enabled: true
  grafanaFolder: "Nextcloud MCP"
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-13 11:49:20 +01:00
Chris CoutinhoandClaude d80e54ff97 feat(helm): Add document chunking configuration
Add support for configurable document chunking parameters to Helm chart
to match docker-compose and application capabilities.

Changes:
1. values.yaml:
   - Add documentChunking section with chunkSize (512) and chunkOverlap (50)
   - Include comprehensive comments explaining chunking strategies
   - Positioned between vectorSync and qdrant sections

2. templates/deployment.yaml:
   - Add DOCUMENT_CHUNK_SIZE and DOCUMENT_CHUNK_OVERLAP env vars
   - Always set (not conditional), used by vector sync processor
   - Environment variables follow same pattern as config.py defaults

3. README.md:
   - Add documentChunking parameter table in Vector Search section
   - Document chunking strategies (small/medium/large chunks)
   - Explain overlap recommendations (10-20% of chunk size)

Validation:
- helm lint: Passes
- helm template: Environment variables correctly generated
- Custom values: Work as expected (tested with chunkSize=1024)
- Always present: Not conditional on vectorSync.enabled

This maintains feature parity between Helm and docker-compose deployments,
allowing users to tune chunking for their embedding models and use cases.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 03:34:16 +01:00
Chris CoutinhoandClaude 167e49788e feat(helm): add Qdrant local mode support with three deployment options [skip ci]
Add support for three Qdrant deployment modes in Helm chart:
1. In-memory mode (:memory:) - Default, zero-config, ephemeral storage
2. Persistent local mode (path-based) - File-based storage with PVC
3. Network mode (URL-based) - Dedicated Qdrant service or external instance

Changes:
- Restructured qdrant configuration in values.yaml with mode selector
- Added conditional environment variable logic in deployment.yaml
- Created PVC template for persistent local mode with optional existingClaim
- Added qdrantPvcName helper template in _helpers.tpl
- Updated README.md with Helm registry URL (https://cbcoutinho.github.io/nextcloud-mcp-server)

Breaking change: Default changed from requiring qdrant.enabled to using
in-memory mode (:memory:) when no Qdrant configuration is provided.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-09 07:14:19 +01:00
Chris CoutinhoandClaude e32c8f4aec feat: add optional vector database and semantic search to helm chart
Add support for deploying Qdrant vector database and Ollama embedding
service as optional helm chart dependencies. Enables semantic search
capabilities for Nextcloud content with flexible deployment options.

Chart Dependencies:
- Add Qdrant v0.9.0 from qdrant/qdrant-helm (conditional)
- Add Ollama v1.33.0 from otwld/ollama-helm (conditional)
- Both dependencies only deploy when enabled

Configuration (values.yaml):
- vectorSync: Background sync settings (interval, workers, queue size)
- qdrant: Subchart config with persistence, resources, clustering
- ollama: Subchart config with model pull, persistence, resources
  - Support for external Ollama via ollama.url (no subchart deployment)
- openai: Alternative embedding provider (OpenAI or compatible API)

Environment Variables (deployment.yaml):
- VECTOR_SYNC_* variables when vectorSync.enabled
- QDRANT_URL, QDRANT_COLLECTION when qdrant.enabled
- OLLAMA_BASE_URL, OLLAMA_EMBEDDING_MODEL when ollama enabled or URL set
- OPENAI_API_KEY when openai.enabled

Documentation:
- README: New "Vector Search & Semantic Capabilities" section
- README: Example 5 showing three deployment patterns
- NOTES.txt: Conditional guidance when vector features enabled
- Secret template for OpenAI API key management

All features disabled by default for backward compatibility.
Tested with helm template and helm lint.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-09 00:03:51 +01:00
Chris Coutinho e03a3c2e83 fix(helm): Remove image tag overide 2025-10-29 12:18:12 +01:00
Chris Coutinho 3b12e585ca fix(helm): Update helm chart with extraArgs 2025-10-29 11:57:13 +01:00
Chris Coutinho b11c3ddfb6 build: Rename /helm -> /charts 2025-10-29 10:30:48 +01:00