Files
mcp-nextcloud/docker-compose.yml
T
Chris CoutinhoandClaude Opus 4.7 292cbb3292 feat(storage): pluggable database backend via DATABASE_URL (ADR-026)
Adds a `DATABASE_URL` setting that lets `RefreshTokenStorage` run against
any SQLAlchemy async backend, primarily `postgresql+asyncpg://...` for
HA k8s deployments. Default behavior is unchanged: when `DATABASE_URL` is
unset the server falls back to the existing `TOKEN_STORAGE_DB` path /
ephemeral SQLite tempfile.

Why
---
Today every MCP pod needs its own PVC to hold the SQLite file, which
pins the Deployment to one replica and blocks horizontal scaling. With
this change, operators can point all replicas at a shared Postgres
(CNPG, RDS, etc.) and the pods become stateless. Encryption stays in
Python (Fernet); the database only sees ciphertext.

What changed
------------
- `config.get_database_url()` resolves DATABASE_URL → TOKEN_STORAGE_DB →
  ephemeral tempfile in that priority order.
- `RefreshTokenStorage` builds a process-shared `AsyncEngine` in
  `initialize()`. SQLite gets NullPool; Postgres gets pool_size=10,
  max_overflow=20, pool_pre_ping=True. 30 aiosqlite call sites adapted
  via a thin `_DBConn` / `_Cursor` / `_Row` / `_ExecuteCtx` shim so
  existing method bodies need no churn beyond the connection
  context-manager swap.
- 7 `INSERT OR REPLACE` statements rewritten as portable
  `INSERT ... ON CONFLICT (...) DO UPDATE` (SQLite ≥ 3.24, Postgres ≥ 9.5).
- `sqlite_master` legacy-detection lookup replaced with SQLAlchemy
  inspector so the path works against either backend.
- File-permission hardening + parent-dir creation gated on
  `is_sqlite_url(...)` — centralized backends manage their own filesystem.
- Alembic migrations 001/002/003/005 converted from raw `op.execute(SQL)`
  to portable `op.create_table()` / `op.create_index()` with SQLAlchemy
  types. All timestamp columns are `sa.BigInteger` so Postgres allocates
  BIGINT (unix epochs don't fit in INT4). SQLite treats BIGINT as
  INTEGER, so existing deployments at revision 006 see no schema drift.
- `migrations.py` + CLI take URLs; `db {upgrade,downgrade,current,history}`
  gain `--database-url / -u` alongside the legacy `--database-path / -d`.
  `get_current_revision()` uses SQLAlchemy inspector instead of raw
  sqlite3, so the CLI works against Postgres too.
- `docker-compose.yml` adds a `postgres-test` service under the
  `postgres` profile (pinned `postgres:16-alpine` digest) for
  integration testing.
- Unit storage tests parametrized over backends via shared
  `tests/fixtures/storage_backend.py` — every test in
  `test_app_password_storage.py` and `test_webhook_storage.py` runs
  once per backend that is available. Postgres is opted in by
  `TEST_DATABASE_URL`.
- New `tests/integration/test_storage_postgres.py` (5 tests, marked
  `postgres` + `integration`) covers refresh-token, app-password,
  OAuth-session, webhook, and audit-log paths end-to-end on Postgres.
- New `docs/ADR-026-pluggable-database-backend.md` records the decision;
  `docs/configuration.md` documents `DATABASE_URL` with examples.

Out of scope
------------
- No SQLite → Postgres data migration tool (clean cutover; tokens reissue
  on next login, webhooks re-register on next sync tick).
- This repo does not provision Postgres. The matching helm chart change
  lives in cbcoutinho/helm-charts (database.url / existingSecret values).

Verification
------------
- `uv run pytest tests/unit/` — 1012 passed, SQLite path unchanged.
- `docker compose --profile postgres up -d postgres-test`
- `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres -v`
  — 5 passed.
- `TEST_DATABASE_URL=... uv run pytest tests/unit/test_app_password_storage.py
  tests/unit/test_webhook_storage.py` — 50 passed (25 per backend).
- `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean.

Tracked on Astrolabe Cloud POC board, card #99.

---

_This PR was generated with the help of AI, and reviewed by a Human_

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 18:06:42 +02:00

389 lines
15 KiB
YAML

services:
# Note: MariaDB is external service. You can find more information about the configuration here:
# https://hub.docker.com/_/mariadb
db:
# Note: Check the recommend version here: https://docs.nextcloud.com/server/latest/admin_manual/installation/system_requirements.html#server
image: docker.io/library/mariadb:11.8.6-noble@sha256:31c1aa20ba2d78c4cbc45b2675e40ffd6a3cd59897a9767ce88fc284309f55ab
restart: always
command: --transaction-isolation=READ-COMMITTED
volumes:
- db:/var/lib/mysql
ports:
- 127.0.0.1:3306:3306
environment:
- MYSQL_ROOT_PASSWORD=password
- MYSQL_PASSWORD=password
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nextcloud
# Note: Redis is an external service. You can find more information about the configuration here:
# https://hub.docker.com/_/redis
redis:
image: docker.io/library/redis:8.6.2-alpine@sha256:81b6f81d6a6c5b9019231a2e8eb10085e3a139a34f833dcc965a8a959b040b72
restart: always
app:
image: ${NEXTCLOUD_IMAGE:-docker.io/library/nextcloud:32.0.9@sha256:ad1cf2d4ced89a8dfeeaa796a2c4dec0b5c3d4e132e9df1c8d15b30897813bcd}
restart: always
ports:
- 127.0.0.1:8080:80
depends_on:
- redis
- db
volumes:
- nextcloud:/var/www/html
- ./app-hooks:/docker-entrypoint-hooks.d:ro
# Mount OIDC development directory outside /var/www/html to avoid rsync conflicts
# The post-installation hook will register /opt/apps as an additional app directory
#- ./third_party:/opt/apps:ro
#- ./third_party/astrolabe:/opt/apps/astrolabe:ro
#- ./third_party/oidc:/opt/apps/oidc:ro
environment:
- NEXTCLOUD_TRUSTED_DOMAINS=app
- NEXTCLOUD_ADMIN_USER=admin
- NEXTCLOUD_ADMIN_PASSWORD=admin
- MYSQL_PASSWORD=password
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nextcloud
- MYSQL_HOST=db
- REDIS_HOST=redis
# Set MCP_SERVER_URL to enable Astrolabe OAuth client auto-creation
# (before-starting hook creates OIDC client + stores credentials in config.php)
# Example: MCP_SERVER_URL=http://mcp-login-flow:8004 docker compose --profile login-flow up -d
- MCP_SERVER_URL=${MCP_SERVER_URL:-}
healthcheck:
test: ["CMD-SHELL", "curl -Ss http://localhost/status.php | grep '\"installed\":true' || exit 1"]
interval: 10s
timeout: 30s
retries: 30
recipes:
image: docker.io/library/nginx:1.29.7-alpine@sha256:e7257f1ef28ba17cf7c248cb8ccf6f0c6e0228ab9c315c152f9c203cd34cf6d1
restart: always
volumes:
- ./tests/fixtures/test_recipe.html:/usr/share/nginx/html/test_recipe.html:ro
- ./tests/fixtures/nginx.conf:/etc/nginx/nginx.conf:ro
unstructured:
image: downloads.unstructured.io/unstructured-io/unstructured-api:0.1.2@sha256:986bc7e5d6c24916c22d45d925db366f55bf4d2ebb0079a7b5d38b48fbc6fb1d
restart: always
ports:
- 127.0.0.1:8005:8000
# Unstructured API runs on port 8000 internally
# We expose it on 8005 externally to avoid conflict
profiles:
- unstructured
mcp:
build: .
restart: always
command: ["--transport", "streamable-http"]
depends_on:
app:
condition: service_healthy
ports:
- 127.0.0.1:8000:8000
- 127.0.0.1:9090:9090
volumes:
- mcp-data:/app/data
- ./settings.toml.example:/app/settings.toml:ro
environment:
- NEXTCLOUD_HOST=http://app:80
- NEXTCLOUD_USERNAME=admin
- NEXTCLOUD_PASSWORD=admin
- NEXTCLOUD_PUBLIC_ISSUER_URL=http://localhost:8080
# Pin the token DB inside the mcp-data volume so the runtime default
# (ephemeral tempfile) doesn't silently apply inside containers.
- TOKEN_STORAGE_DB=/app/data/tokens.db
# Semantic search configuration (ADR-007, ADR-021)
- ENABLE_SEMANTIC_SEARCH=true
- VECTOR_SYNC_SCAN_INTERVAL=5
- VECTOR_SYNC_PROCESSOR_WORKERS=2
#- LOG_FORMAT=json
# Qdrant configuration (three modes):
# 1. Network mode: Set QDRANT_URL=http://qdrant:6333 (requires qdrant service)
# 2. In-memory mode: Set QDRANT_LOCATION=:memory: (default if nothing set)
# 3. Persistent local: Set QDRANT_LOCATION=/app/data/qdrant (stored in mcp-data volume)
- QDRANT_LOCATION=":memory:"
#- QDRANT_URL=http://qdrant:6333 # Uncomment for network mode
#- QDRANT_API_KEY=${QDRANT_API_KEY:-my_secret_api_key} # Only for network mode
# Observability
#- OTEL_SERVICE_NAME=nextcloud-mcp-docker-compose
#- OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
# Collection naming: Auto-generated as {deployment-id}-{model-name}
# - Deployment ID: OTEL_SERVICE_NAME (if set) or hostname (fallback)
# - Model name: OLLAMA_EMBEDDING_MODEL
# - Example: "nextcloud-mcp-server-nomic-embed-text"
# - Changing models creates new collection (requires re-embedding)
# - Set QDRANT_COLLECTION to override auto-generation:
#- QDRANT_COLLECTION=nextcloud_content
# Ollama configuration (optional - uses SimpleEmbeddingProvider if not set)
# - OLLAMA_BASE_URL=http://ollama:11434
# - OLLAMA_EMBEDDING_MODEL=nomic-embed-text # Changing this creates new collection
# - OLLAMA_VERIFY_SSL=false
# Document chunking configuration (for vector embeddings)
# Tune these based on your embedding model and content type
# - DOCUMENT_CHUNK_SIZE=512 # Words per chunk (default: 512)
# - DOCUMENT_CHUNK_OVERLAP=50 # Overlapping words (default: 50, recommended: 10-20% of chunk size)
profiles:
- single-user
mcp-multi-user-basic:
build: .
restart: always
command: ["--transport", "streamable-http"]
depends_on:
app:
condition: service_healthy
ports:
- 127.0.0.1:8003:8000
environment:
# Multi-user BasicAuth pass-through mode (ADR-020, ADR-022)
- NEXTCLOUD_HOST=http://app:80
- NEXTCLOUD_MCP_SERVER_URL=http://localhost:8003
- NEXTCLOUD_PUBLIC_ISSUER_URL=http://localhost:8080
- MCP_DEPLOYMENT_MODE=multi_user_basic
- ENABLE_BACKGROUND_OPERATIONS=true
# Token storage (required for middleware initialization).
# Source the key from .env — see env.sample. To generate a fresh key:
# python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
- TOKEN_ENCRYPTION_KEY=${TOKEN_ENCRYPTION_KEY:?TOKEN_ENCRYPTION_KEY must be set in .env (see env.sample)}
- TOKEN_STORAGE_DB=/app/data/tokens.db
- ENABLE_SEMANTIC_SEARCH=true
- VECTOR_SYNC_SCAN_INTERVAL=60
- VECTOR_SYNC_PROCESSOR_WORKERS=1
# OAuth credentials for background sync (optional - uses DCR if not provided)
# Uncomment to avoid DCR:
# - NEXTCLOUD_OIDC_CLIENT_ID=your_client_id
# - NEXTCLOUD_OIDC_CLIENT_SECRET=your_client_secret
# Management API allowlist (ADR-018): only Astrolabe-issued tokens may
# call /api/management/*. Default test client_id used by the
# configure_astrolabe_for_mcp_server fixture.
- ALLOWED_MGMT_CLIENT=nextcloudMcpServerUIPublicClient
# NO admin credentials - credentials come from client Authorization header
volumes:
- multi-user-basic-data:/app/data
- ./settings.toml.example:/app/settings.toml:ro
profiles:
- multi-user-basic
keycloak:
image: quay.io/keycloak/keycloak:26.5.4@sha256:ae8efb0d218d8921334b03a2dbee7069a0b868240691c50a3ffc9f42fabba8b4
command:
- "start-dev"
- "--import-realm"
- "--hostname=http://localhost:8888"
- "--hostname-strict=false"
- "--hostname-backchannel-dynamic=true"
- "--features=preview" # Enable Legacy V1 token exchange (supports both Standard V2 and Legacy V1)
ports:
- 127.0.0.1:8888:8080
environment:
- KC_BOOTSTRAP_ADMIN_USERNAME=admin
- KC_BOOTSTRAP_ADMIN_PASSWORD=admin
volumes:
- ./keycloak/realm-export.json:/opt/keycloak/data/import/realm.json:ro
healthcheck:
test: ["CMD-SHELL", "exec 3<>/dev/tcp/localhost/8080 && echo -e 'GET /realms/nextcloud-mcp HTTP/1.1\\r\\nHost: localhost\\r\\nConnection: close\\r\\n\\r\\n' >&3 && cat <&3 | grep -q 'HTTP/1.1 200'"]
interval: 10s
timeout: 5s
retries: 30
profiles:
- keycloak
mcp-keycloak:
build: .
command: ["--transport", "streamable-http", "--oauth", "--port", "8002"]
restart: always
depends_on:
keycloak:
condition: service_healthy
app:
condition: service_started
ports:
- 127.0.0.1:8002:8002
environment:
# Generic OIDC configuration (external IdP mode - Keycloak)
# Provider auto-detected from OIDC_DISCOVERY_URL issuer
# Using internal Docker hostname for discovery to get consistent issuer
- OIDC_DISCOVERY_URL=http://keycloak:8080/realms/nextcloud-mcp/.well-known/openid-configuration
- NEXTCLOUD_OIDC_CLIENT_ID=nextcloud-mcp-server
- NEXTCLOUD_OIDC_CLIENT_SECRET=mcp-secret-change-in-production
- OIDC_JWKS_URI=http://keycloak:8080/realms/nextcloud-mcp/protocol/openid-connect/certs
# Nextcloud API endpoint (for accessing APIs with validated token)
- NEXTCLOUD_HOST=http://app:80
- NEXTCLOUD_MCP_SERVER_URL=http://localhost:8002
- NEXTCLOUD_RESOURCE_URI=nextcloud # ADR-005: Keycloak uses client IDs as audiences, not URLs
- NEXTCLOUD_PUBLIC_ISSUER_URL=http://localhost:8888/realms/nextcloud-mcp
# Refresh token storage (ADR-002 Tier 1 & 2). Source from .env.
- ENABLE_BACKGROUND_OPERATIONS=true
- TOKEN_ENCRYPTION_KEY=${TOKEN_ENCRYPTION_KEY:?TOKEN_ENCRYPTION_KEY must be set in .env (see env.sample)}
- TOKEN_STORAGE_DB=/app/data/tokens.db
# ADR-005: Token exchange mode (RFC 8693)
# Exchange MCP tokens (aud: nextcloud-mcp-server) for Nextcloud tokens (aud: http://localhost:8080)
# Provides strict audience separation between MCP session and Nextcloud API access
- ENABLE_TOKEN_EXCHANGE=true
- TOKEN_EXCHANGE_CACHE_TTL=300 # Cache exchanged tokens for 5 minutes (default)
# Login Flow v2 (ADR-022) with external IdP — derived from the
# auto-detected LOGIN_FLOW deployment mode; no separate flag needed.
- ENABLE_DCR=true
# OAuth scopes (optional - uses defaults if not specified)
- NEXTCLOUD_OIDC_SCOPES=openid profile email offline_access notes.read notes.write calendar.read calendar.write contacts.read contacts.write cookbook.read cookbook.write deck.read deck.write tables.read tables.write files.read files.write sharing.read sharing.write todo.read todo.write
# NO admin credentials - using external IdP OAuth only!
volumes:
- keycloak-tokens:/app/data
- keycloak-oauth-storage:/app/.oauth
- ./settings.toml.example:/app/settings.toml:ro
profiles:
- keycloak
# Login Flow v2 mode (ADR-022)
# Test with: docker compose --profile login-flow up --build -d
mcp-login-flow:
build: .
restart: always
# --oauth enables the OAuth/OIDC identity layer that Login Flow v2 builds on
# (user identity via OAuth session, Nextcloud access via app passwords)
command: ["--transport", "streamable-http", "--oauth", "--port", "8004"]
depends_on:
app:
condition: service_healthy
ports:
- 127.0.0.1:8004:8004
environment:
- NEXTCLOUD_HOST=http://app:80
- NEXTCLOUD_MCP_SERVER_URL=http://localhost:8004
#- NEXTCLOUD_MCP_SERVER_URL=https://nextcloud-mcp-dev.tail148d5.ts.net
- NEXTCLOUD_PUBLIC_ISSUER_URL=http://localhost:8080
# Login Flow v2 (ADR-022) — the deployment mode is the single switch;
# the browser-based app-password layer is derived automatically.
- MCP_DEPLOYMENT_MODE=login_flow
# Token storage (required for app password + session persistence).
# Source the key from .env — see env.sample. To generate a fresh key:
# python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
- TOKEN_ENCRYPTION_KEY=${TOKEN_ENCRYPTION_KEY:?TOKEN_ENCRYPTION_KEY must be set in .env (see env.sample)}
- TOKEN_STORAGE_DB=/app/data/tokens.db
# Semantic search
- ENABLE_SEMANTIC_SEARCH=true
- VECTOR_SYNC_SCAN_INTERVAL=60
- VECTOR_SYNC_PROCESSOR_WORKERS=1
# Management API allowlist (ADR-018) — matches mcp-multi-user-basic so
# the same configure_astrolabe_for_mcp_server fixture (which creates the
# static `nextcloudMcpServerUIPublicClient` OIDC client) works for tests
# against this profile too.
- ALLOWED_MGMT_CLIENT=astrolabeMcpClientOAuth00000000000
volumes:
- login-flow-data:/app/data
- login-flow-oauth-storage:/app/.oauth
- ./settings.toml.example:/app/settings.toml:ro
profiles:
- login-flow
# Tailscale Funnel for Claude AI connector testing
# Usage: docker compose --profile login-flow --profile claude-funnel up --build -d
# Requires: TS_AUTHKEY in .env file (see env.sample.claude-funnel)
tailscale-mcp:
image: docker.io/tailscale/tailscale:v1.94.2@sha256:95e528798bebe75f39b10e74e7051cf51188ee615934f232ba7ad06a3390ffa1
hostname: nextcloud-mcp-dev
restart: always
environment:
- TS_AUTHKEY=${TS_AUTHKEY}
- TS_STATE_DIR=/var/lib/tailscale
- TS_SERVE_CONFIG=/config/serve-config.json
- TS_HOSTNAME=nextcloud-mcp-dev
- TS_EXTRA_ARGS=--advertise-tags=tag:container
volumes:
- tailscale-state:/var/lib/tailscale
- ./claude-funnel/serve-config.json:/config/serve-config.json:ro
cap_add:
- NET_ADMIN
- SYS_MODULE
profiles:
- claude-funnel
nginx-claude-filter:
image: docker.io/library/nginx:1.29.7-alpine@sha256:e7257f1ef28ba17cf7c248cb8ccf6f0c6e0228ab9c315c152f9c203cd34cf6d1
restart: always
depends_on:
- mcp-login-flow
volumes:
- ./claude-funnel/nginx.conf:/etc/nginx/nginx.conf:ro
profiles:
- claude-funnel
qdrant:
image: docker.io/qdrant/qdrant:v1.18.0@sha256:1cf3e07c9f269030c7cfed0d5c0beea7bb081848a88e2ae13b35a613d4dd5019
restart: always
ports:
- 127.0.0.1:6333:6333 # REST API
- 127.0.0.1:6334:6334 # gRPC (optional)
volumes:
- qdrant-data:/qdrant/storage
environment:
- QDRANT__SERVICE__API_KEY=${QDRANT_API_KEY:-my_secret_api_key}
healthcheck:
test: ["CMD-SHELL", "test -f /qdrant/.qdrant-initialized"]
interval: 10s
timeout: 5s
retries: 10
profiles:
- qdrant
# Centralized Postgres backend for token storage. Used by:
# 1. Integration tests gated on @pytest.mark.postgres — they read
# TEST_DATABASE_URL=postgresql+asyncpg://mcp:mcp@localhost:5433/mcp.
# 2. Manual smoke testing of the HA (multi-replica) deployment story.
# Bring up with: docker compose --profile postgres up -d postgres-test
# See ADR-026 for the pluggable-database-backend design.
postgres-test:
image: docker.io/library/postgres:16-alpine@sha256:16bc17c64a573ef34162af9298258d1aec548232985b33ed7b1eac33ba35c229
restart: unless-stopped
environment:
POSTGRES_USER: mcp
POSTGRES_PASSWORD: mcp
POSTGRES_DB: mcp
ports:
- 127.0.0.1:5433:5432
healthcheck:
test: ["CMD-SHELL", "pg_isready -U mcp -d mcp"]
interval: 5s
timeout: 3s
retries: 10
profiles:
- postgres
volumes:
nextcloud:
db:
keycloak-tokens:
keycloak-oauth-storage:
login-flow-data:
login-flow-oauth-storage:
qdrant-data:
mcp-data:
multi-user-basic-data:
tailscale-state: