Implement comprehensive observability for vector database synchronization
with Grafana dashboard and Prometheus metrics.
## Part 1: Grafana Dashboard
Created all-in-one operations dashboard with 7 rows and 34 panels:
### Dashboard Structure:
- **Overview Row**: Request rate, error rate, P95 latency, active requests
- **HTTP Metrics (RED)**: Request/error rates by endpoint, latency percentiles
- **MCP Tools**: Call volume, error rates, execution duration by tool
- **Nextcloud API**: API calls/latency by app, retry patterns
- **OAuth & Authentication**: Token validations, exchanges, cache hit rate
- **Dependencies & Health**: Status for Nextcloud/Qdrant/Keycloak/Unstructured
- **Vector Sync**: Processing throughput, queue depth, Qdrant operations
### Helm Chart Integration:
- Added dashboard-configmap.yaml template for automatic provisioning
- Configured Grafana sidecar auto-discovery (label: grafana_dashboard="1")
- Added dashboards configuration section in values.yaml (opt-in)
- Updated Chart.yaml with dashboard annotations
- Enhanced NOTES.txt with dashboard deployment instructions
- Comprehensive documentation in dashboards/README.md
Dashboard supports dynamic filtering via variables:
- datasource: Prometheus data source selection
- namespace: Filter by Kubernetes namespace
- pod: Multi-select pod filtering
- interval: Query interval (1m/5m/10m/30m/1h)
## Part 2: Vector Sync Metric Instrumentation
Implemented metric recording throughout vector sync pipeline:
### metrics.py:
Added convenience functions:
- record_vector_sync_scan() - Track documents per scan
- record_vector_sync_processing() - Track processing duration/status
- record_qdrant_operation() - Track database operations
- update_vector_sync_queue_size() - Track queue depth
### scanner.py:
- Record number of documents found in each scan
- Enables monitoring of scan throughput
### processor.py:
- Record processing duration for each document
- Track success/failure status with timing
- Record Qdrant upsert/delete operations
- Handle all code paths (success, deletion, error)
### semantic.py:
- Wrap Qdrant query_points with try/except
- Record search operation success/failure
## Metrics Exposed:
- mcp_vector_sync_documents_scanned_total
- mcp_vector_sync_documents_processed_total{status}
- mcp_vector_sync_processing_duration_seconds (histogram)
- mcp_vector_sync_queue_size (gauge)
- mcp_qdrant_operations_total{operation,status}
This enables monitoring of:
- Scan and processing throughput
- Processing latency (P50/P95/P99)
- Error rates for processing and Qdrant operations
- Queue depth trends
- Complete observability of vector sync pipeline
## Testing:
Verified locally that metrics are recorded correctly:
- 36 documents scanned
- 3 documents processed (avg 7.5s each)
- 3 successful Qdrant upsert operations
- Search operations tracked
## Deployment:
Enable dashboard provisioning in Helm values:
```yaml
dashboards:
enabled: true
grafanaFolder: "Nextcloud MCP"
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
132 lines
6.2 KiB
Plaintext
132 lines
6.2 KiB
Plaintext
Thank you for installing {{ .Chart.Name }}!
|
|
|
|
Your Nextcloud MCP Server has been deployed in {{ .Values.auth.mode }} authentication mode.
|
|
|
|
1. Get the application URL by running these commands:
|
|
{{- if .Values.ingress.enabled }}
|
|
{{- range $host := .Values.ingress.hosts }}
|
|
{{- range .paths }}
|
|
http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host.host }}{{ .path }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- else if contains "NodePort" .Values.service.type }}
|
|
export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "nextcloud-mcp-server.fullname" . }})
|
|
export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
|
|
echo http://$NODE_IP:$NODE_PORT
|
|
{{- else if contains "LoadBalancer" .Values.service.type }}
|
|
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
|
|
You can watch the status of by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "nextcloud-mcp-server.fullname" . }}'
|
|
export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "nextcloud-mcp-server.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}")
|
|
echo http://$SERVICE_IP:{{ .Values.service.port }}
|
|
{{- else if contains "ClusterIP" .Values.service.type }}
|
|
export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "nextcloud-mcp-server.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
|
|
export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
|
|
echo "Visit http://127.0.0.1:8080 to use your MCP server"
|
|
kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:$CONTAINER_PORT
|
|
{{- end }}
|
|
|
|
2. Check the deployment status:
|
|
kubectl --namespace {{ .Release.Namespace }} get pods -l "app.kubernetes.io/name={{ include "nextcloud-mcp-server.name" . }},app.kubernetes.io/instance={{ .Release.Name }}"
|
|
|
|
{{- if eq .Values.auth.mode "basic" }}
|
|
|
|
3. Basic Authentication Mode:
|
|
{{- if .Values.auth.basic.existingSecret }}
|
|
- Credentials: (using existing secret {{ .Values.auth.basic.existingSecret }})
|
|
{{- else }}
|
|
- Username: {{ .Values.auth.basic.username }}
|
|
- Password: (stored in secret {{ include "nextcloud-mcp-server.basicAuthSecretName" . }})
|
|
{{- end }}
|
|
- Connected to: {{ .Values.nextcloud.host }}
|
|
{{- else if eq .Values.auth.mode "oauth" }}
|
|
|
|
3. OAuth Authentication Mode:
|
|
- Server URL: {{ include "nextcloud-mcp-server.mcpServerUrl" . }}
|
|
- Issuer URL: {{ include "nextcloud-mcp-server.publicIssuerUrl" . }}
|
|
- Connected to: {{ .Values.nextcloud.host }}
|
|
{{- if .Values.auth.oauth.existingSecret }}
|
|
- Using existing OAuth client secret: {{ .Values.auth.oauth.existingSecret }}
|
|
{{- else if and .Values.auth.oauth.clientId .Values.auth.oauth.clientSecret }}
|
|
- Using pre-registered OAuth client
|
|
{{- else }}
|
|
- Using Dynamic Client Registration (DCR)
|
|
{{- end }}
|
|
{{- if .Values.auth.oauth.persistence.enabled }}
|
|
- OAuth client credentials are persisted in PVC: {{ include "nextcloud-mcp-server.oauthPvcName" . }}
|
|
{{- end }}
|
|
|
|
IMPORTANT: OAuth mode is experimental and requires patches to the user_oidc app.
|
|
See: https://github.com/cbcoutinho/nextcloud-mcp-server#authentication
|
|
{{- end }}
|
|
|
|
{{- if .Values.documentProcessing.enabled }}
|
|
|
|
4. Document Processing:
|
|
- Enabled: {{ .Values.documentProcessing.enabled }}
|
|
- Default processor: {{ .Values.documentProcessing.defaultProcessor }}
|
|
{{- if .Values.documentProcessing.unstructured.enabled }}
|
|
- Unstructured API: {{ .Values.documentProcessing.unstructured.apiUrl }}
|
|
{{- end }}
|
|
{{- end }}
|
|
|
|
{{- if .Values.vectorSync.enabled }}
|
|
|
|
5. Vector Search & Semantic Capabilities:
|
|
- Vector Sync: Enabled
|
|
- Scan Interval: {{ .Values.vectorSync.scanInterval }}s
|
|
- Processor Workers: {{ .Values.vectorSync.processorWorkers }}
|
|
{{- if .Values.qdrant.enabled }}
|
|
- Qdrant: Deployed as subchart ({{ .Release.Name }}-qdrant:6333)
|
|
{{- else }}
|
|
- Qdrant: Not deployed (configure external instance)
|
|
{{- end }}
|
|
{{- if .Values.ollama.enabled }}
|
|
- Ollama: Deployed as subchart ({{ .Release.Name }}-ollama:11434)
|
|
- Embedding Model: {{ .Values.ollama.embeddingModel }}
|
|
{{- else if .Values.ollama.url }}
|
|
- Ollama: Using external instance at {{ .Values.ollama.url }}
|
|
- Embedding Model: {{ .Values.ollama.embeddingModel }}
|
|
{{- else if .Values.openai.enabled }}
|
|
- OpenAI: Enabled for embeddings
|
|
{{- else }}
|
|
- WARNING: No embedding provider configured (Ollama or OpenAI required)
|
|
{{- end }}
|
|
|
|
Check vector sync status:
|
|
kubectl --namespace {{ .Release.Namespace }} exec -it deploy/{{ include "nextcloud-mcp-server.fullname" . }} -- curl -s http://localhost:{{ include "nextcloud-mcp-server.port" . }}/user/page | grep "Vector Sync"
|
|
{{- end }}
|
|
|
|
{{- if .Values.dashboards.enabled }}
|
|
|
|
6. Grafana Dashboards:
|
|
- Dashboard provisioning: Enabled
|
|
- ConfigMap: {{ include "nextcloud-mcp-server.fullname" . }}-dashboard
|
|
- Grafana Folder: {{ .Values.dashboards.grafanaFolder }}
|
|
|
|
The dashboard will be automatically imported by Grafana if the sidecar is configured
|
|
to watch for ConfigMaps with label "grafana_dashboard: 1".
|
|
|
|
To manually import the dashboard:
|
|
kubectl --namespace {{ .Release.Namespace }} get configmap {{ include "nextcloud-mcp-server.fullname" . }}-dashboard -o jsonpath='{.data.nextcloud-mcp-server\.json}' | jq . > dashboard.json
|
|
|
|
Then import dashboard.json via Grafana UI (Dashboards → Import).
|
|
{{- else }}
|
|
|
|
6. Grafana Dashboards:
|
|
- Dashboard provisioning: Disabled
|
|
- To enable automatic dashboard provisioning, set: dashboards.enabled=true
|
|
|
|
Manual import option:
|
|
The dashboard JSON is available in the chart at charts/nextcloud-mcp-server/dashboards/nextcloud-mcp-server.json
|
|
{{- end }}
|
|
|
|
For more information and documentation:
|
|
- GitHub: https://github.com/cbcoutinho/nextcloud-mcp-server
|
|
- Documentation: https://github.com/cbcoutinho/nextcloud-mcp-server#readme
|
|
|
|
To upgrade this deployment:
|
|
helm upgrade {{ .Release.Name }} nextcloud-mcp-server
|
|
|
|
To uninstall:
|
|
helm uninstall {{ .Release.Name }}
|