feat: Add Grafana dashboard and vector sync metric instrumentation

Implement comprehensive observability for vector database synchronization
with Grafana dashboard and Prometheus metrics.

## Part 1: Grafana Dashboard

Created all-in-one operations dashboard with 7 rows and 34 panels:

### Dashboard Structure:
- **Overview Row**: Request rate, error rate, P95 latency, active requests
- **HTTP Metrics (RED)**: Request/error rates by endpoint, latency percentiles
- **MCP Tools**: Call volume, error rates, execution duration by tool
- **Nextcloud API**: API calls/latency by app, retry patterns
- **OAuth & Authentication**: Token validations, exchanges, cache hit rate
- **Dependencies & Health**: Status for Nextcloud/Qdrant/Keycloak/Unstructured
- **Vector Sync**: Processing throughput, queue depth, Qdrant operations

### Helm Chart Integration:
- Added dashboard-configmap.yaml template for automatic provisioning
- Configured Grafana sidecar auto-discovery (label: grafana_dashboard="1")
- Added dashboards configuration section in values.yaml (opt-in)
- Updated Chart.yaml with dashboard annotations
- Enhanced NOTES.txt with dashboard deployment instructions
- Comprehensive documentation in dashboards/README.md

Dashboard supports dynamic filtering via variables:
- datasource: Prometheus data source selection
- namespace: Filter by Kubernetes namespace
- pod: Multi-select pod filtering
- interval: Query interval (1m/5m/10m/30m/1h)

## Part 2: Vector Sync Metric Instrumentation

Implemented metric recording throughout vector sync pipeline:

### metrics.py:
Added convenience functions:
- record_vector_sync_scan() - Track documents per scan
- record_vector_sync_processing() - Track processing duration/status
- record_qdrant_operation() - Track database operations
- update_vector_sync_queue_size() - Track queue depth

### scanner.py:
- Record number of documents found in each scan
- Enables monitoring of scan throughput

### processor.py:
- Record processing duration for each document
- Track success/failure status with timing
- Record Qdrant upsert/delete operations
- Handle all code paths (success, deletion, error)

### semantic.py:
- Wrap Qdrant query_points with try/except
- Record search operation success/failure

## Metrics Exposed:

- mcp_vector_sync_documents_scanned_total
- mcp_vector_sync_documents_processed_total{status}
- mcp_vector_sync_processing_duration_seconds (histogram)
- mcp_vector_sync_queue_size (gauge)
- mcp_qdrant_operations_total{operation,status}

This enables monitoring of:
- Scan and processing throughput
- Processing latency (P50/P95/P99)
- Error rates for processing and Qdrant operations
- Queue depth trends
- Complete observability of vector sync pipeline

## Testing:

Verified locally that metrics are recorded correctly:
- 36 documents scanned
- 3 documents processed (avg 7.5s each)
- 3 successful Qdrant upsert operations
- Search operations tracked

## Deployment:

Enable dashboard provisioning in Helm values:
```yaml
dashboards:
  enabled: true
  grafanaFolder: "Nextcloud MCP"
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2025-11-13 11:49:20 +01:00
co-authored by Claude
parent 619ba5684d
commit 4ea5ed72d4
11 changed files with 1976 additions and 562 deletions
+66
View File
@@ -280,6 +280,72 @@ Use OpenAI or any OpenAI-compatible API instead of Ollama.
| `openai.secretKey` | Key in secret containing API key | `api-key` |
| `openai.baseUrl` | Custom API endpoint (optional) | `""` |
#### Observability & Monitoring
The chart includes comprehensive observability features including Prometheus metrics, OpenTelemetry tracing, and Grafana dashboards.
**Metrics Configuration:**
| Parameter | Description | Default |
|-----------|-------------|---------|
| `observability.metrics.enabled` | Enable Prometheus metrics | `true` |
| `observability.metrics.port` | Metrics port | `9090` |
| `observability.metrics.path` | Metrics endpoint path | `/metrics` |
**Tracing Configuration:**
| Parameter | Description | Default |
|-----------|-------------|---------|
| `observability.tracing.enabled` | Enable OpenTelemetry tracing | `false` |
| `observability.tracing.endpoint` | OTLP collector endpoint | `""` |
| `observability.tracing.serviceName` | Service name in traces | `nextcloud-mcp-server` |
| `observability.tracing.samplingRate` | Trace sampling rate (0.0-1.0) | `1.0` |
**Logging Configuration:**
| Parameter | Description | Default |
|-----------|-------------|---------|
| `observability.logging.format` | Log format (json or text) | `json` |
| `observability.logging.level` | Log level | `INFO` |
| `observability.logging.includeTraceContext` | Include trace IDs in logs | `true` |
**ServiceMonitor (Prometheus Operator):**
| Parameter | Description | Default |
|-----------|-------------|---------|
| `serviceMonitor.enabled` | Create ServiceMonitor resource | `false` |
| `serviceMonitor.interval` | Scrape interval | `30s` |
| `serviceMonitor.scrapeTimeout` | Scrape timeout | `10s` |
| `serviceMonitor.labels` | Additional labels for ServiceMonitor | `{}` |
**PrometheusRule (Prometheus Operator):**
| Parameter | Description | Default |
|-----------|-------------|---------|
| `prometheusRule.enabled` | Create PrometheusRule with alert rules | `false` |
| `prometheusRule.labels` | Additional labels for PrometheusRule | `{}` |
**Grafana Dashboards:**
| Parameter | Description | Default |
|-----------|-------------|---------|
| `dashboards.enabled` | Enable automatic dashboard provisioning | `false` |
| `dashboards.grafanaFolder` | Grafana folder name for dashboards | `Nextcloud MCP` |
| `dashboards.labels` | Additional labels for dashboard ConfigMap | `{}` |
| `dashboards.annotations` | Additional annotations for dashboard ConfigMap | `{}` |
When `dashboards.enabled` is `true`, a ConfigMap with the Grafana dashboard is created with the `grafana_dashboard: "1"` label. This enables automatic discovery by Grafana sidecar containers (commonly used with kube-prometheus-stack).
The dashboard provides comprehensive monitoring including:
- HTTP request metrics (RED pattern: Rate, Errors, Duration)
- MCP tool performance and errors
- Nextcloud API performance by app (notes, calendar, contacts, etc.)
- OAuth token operations and cache hit rates
- External dependency health (Nextcloud, Qdrant, Keycloak, Unstructured API)
- Vector sync processing pipeline (when enabled)
For manual import or more details, see `charts/nextcloud-mcp-server/dashboards/README.md`.
## Examples
### Example 1: Basic Auth with Ingress