mcp-nextcloud/docs/ADR-026-pluggable-database-backend.md

ADR-026: Pluggable database backend (DATABASE_URL)

Status

Accepted — 2026-05-16

Context

RefreshTokenStorage (in nextcloud_mcp_server/auth/storage.py) holds all of the MCP server's persistent state: refresh tokens, OAuth client credentials, OAuth sessions, browser sessions, app passwords, login-flow sessions, audit logs, and webhook registrations. Until this ADR it was backed by a single SQLite file, with the path configured by TOKEN_STORAGE_DB.

This works well for single-user deployments but blocks horizontal scaling in Kubernetes:

• Every pod needs its own PVC (ReadWriteOnce) or a ReadWriteMany volume.
• Tokens stored on pod A are invisible to pod B, so a Service can only route traffic to one pod at a time.
• Restart / re-deploy cycles either drop the volume (token loss) or require coordinated PVC handling.
• Backup, encryption-at-rest, and multi-region replication become per-pod concerns rather than centrally managed DB concerns.

We needed a way for pods to be stateless and share a centralized store without giving up the zero-config SQLite path that single-user installs and local development rely on.

Decision

Introduce a DATABASE_URL setting that accepts any SQLAlchemy async URL, with sqlite+aiosqlite:///... remaining the default. The runtime keeps a single linear migration history and a single RefreshTokenStorage class — the backend is selected purely by the URL.

Resolution order

get_database_url() (in nextcloud_mcp_server/config.py) returns:

1. DATABASE_URL if set — wins over everything.
2. Otherwise sqlite+aiosqlite:///{get_token_db_path()}, so the legacy TOKEN_STORAGE_DB env var and the process-local ephemeral tempfile fallback both keep working unchanged.

Why SQLAlchemy Core + async engine, not an ABC with parallel drivers

Two alternatives were considered:

Option	Why rejected
Define a Storage ABC with SQLiteStorage (aiosqlite) and PostgresStorage (asyncpg) implementations	Doubles the surface area — every schema change has to land in two backends, with two sets of migrations, two SQL dialects, two upsert idioms. Diverges over time.
Switch to a full SQLAlchemy ORM (declarative models)	Larger refactor; the existing explicit-SQL style is intentional and well-understood by reviewers.
Keep RefreshTokenStorage and put SQLAlchemy Core under it (chosen)	One method body per operation, one migration history (Alembic is already SQLAlchemy-based). The URL drives dialect, pool, and DDL.

A thin compatibility shim (_DBConn / _Cursor / _Row in storage.py) adapts the async with aiosqlite.connect(...) as db: async with db.execute(...) as cursor: ... idiom to SQLAlchemy AsyncEngine / AsyncConnection. Existing method bodies needed only their connection context-manager swapped; ? placeholders are rewritten to named binds on the fly. The seven INSERT OR REPLACE statements were rewritten as portable INSERT ... ON CONFLICT (...) DO UPDATE (SQLite ≥ 3.24, Postgres ≥ 9.5; we already require SQLite ≥ 3.35 elsewhere).

Distribution: asyncpg is an optional extra, bundled in Docker

asyncpg carries a compiled C extension (~5 MB plus a build toolchain on source installs) — too heavy a default for the pip install nextcloud-mcp-server audience, the majority of whom run the SQLite path. It is shipped as a PyPI optional dependency::

pip install 'nextcloud-mcp-server[postgres]'

The published Docker image runs uv sync --extra postgres so the container always has the driver, matching the HA-deployment audience that exercises the Postgres backend. When DATABASE_URL=postgresql+asyncpg://... is set on a venv without the extra installed, RefreshTokenStorage raises a clear actionable error before the engine is built — operators see "install with [postgres] extra" rather than a generic ModuleNotFoundError: No module named 'asyncpg'.

Alembic env.py runs the async engine inside a worker thread

nextcloud_mcp_server/alembic/env.py uses async_engine_from_config(...) + anyio.run(run_async_migrations), and the runtime invokes it from RefreshTokenStorage.initialize() via anyio.to_thread.run_sync(upgrade_database, ...). This is intentional:

• Alembic wants a synchronous entry point (upgrade_database()), but async_engine_from_config returns an async engine.
• Running anyio.run() directly inside an already-running event loop would deadlock; we have to be on a different thread.
• to_thread.run_sync puts the call on a worker thread, which has no running event loop — anyio.run() is then free to spin up its own.

The pattern is non-obvious; this note exists so a future maintainer doesn't try to "simplify" it back into the main loop.

TLS for the Postgres backend

Two settings mirror the existing NEXTCLOUD_VERIFY_SSL / NEXTCLOUD_CA_BUNDLE pattern: DATABASE_VERIFY_SSL and DATABASE_CA_BUNDLE. get_database_ssl() (in nextcloud_mcp_server/config.py) returns the value to pass to asyncpg via SQLAlchemy's connect_args={"ssl": ...}.

The default is deliberately less strict than the Nextcloud HTTPS default: DATABASE_VERIFY_SSL defaults to None rather than True. When both env vars are unset we omit the ssl kwarg entirely and asyncpg's default (prefer) applies — TLS if the server offers it, no certificate validation. The reasoning:

• Cluster-internal Postgres (CNPG via a Service, RDS over a private VPC, PgBouncer sidecar) is the common HA pattern and frequently runs without TLS or with cert hostnames asyncpg wouldn't match anyway.
• The HTTPS analogy doesn't carry over: the Nextcloud client talks to external hostnames over public networks where verify-full is the right default. The database client talks to a controlled peer.
• Just-shipped PR #798 had no TLS knobs and worked against cluster-local Postgres-test; flipping the default to True here would break that flow on upgrade.

Operators in production with a managed Postgres opt in with DATABASE_VERIFY_SSL=true. Homelab operators with a private CA set DATABASE_CA_BUNDLE=/path/to/ca.pem (which implies verify=true). DATABASE_VERIFY_SSL=false is the escape hatch for incident response — it wins over DATABASE_CA_BUNDLE so an operator can quickly silence cert errors without editing the secret store.

Encryption stays in Python (Fernet), not the DB

The DB only ever sees ciphertext for sensitive columns (encrypted_token, encrypted_client_secret, encrypted_password, encrypted_poll_token). The Fernet key remains a TOKEN_ENCRYPTION_KEY env var, applied in Python before INSERT and after SELECT. This means:

• Switching backends does not invalidate or re-key existing data.
• Postgres-level features like pgcrypto are not required.
• Operators rotating the encryption key still go through the existing Python path.

DDL portability

All Alembic migrations were rewritten from raw op.execute("CREATE TABLE ...") strings to op.create_table() / op.create_index() calls with SQLAlchemy types. Notable choices:

• All *_at / expiration / timestamp columns use sa.BigInteger — Postgres INTEGER is 32-bit and unix epochs are already past that range. SQLite treats BIGINT and INTEGER identically (dynamic typing) so this is backwards compatible.
• BLOB → sa.LargeBinary (becomes BYTEA on Postgres).
• BOOLEAN DEFAULT FALSE → sa.Boolean, server_default=sa.false().
• Existing SQLite deployments are at revision 006 and skip the rewritten migrations entirely — content rewrites are safe.

No data migration, no shipped Postgres

Two scope decisions worth recording:

1. Clean cutover, no SQLite → Postgres data migration tool. Tokens are reissued on the next login; webhooks re-register on the next sync tick. Acceptable because the ephemeral-default already implies this, and the data being preserved (audit logs, OAuth sessions) is either short-lived or reconstructible.
2. Bring-your-own database. The MCP server consumes a DATABASE_URL; it does not provision Postgres itself. Operators use CNPG, RDS, the project's existing Helm chart with a sub-chart, etc. The postgres-test service in docker-compose.yml exists only for integration tests and manual HA smoke testing — it is gated on the postgres profile and is not the recommended production pattern.

CLI changes

The nextcloud-mcp-server db {upgrade,downgrade,current,history} commands gain a --database-url / -u flag (env DATABASE_URL) alongside the existing --database-path / -d (env TOKEN_STORAGE_DB). -u wins over -d; both fall back to get_database_url().

Consequences

Positive

• MCP server pods become stateless. A Kubernetes Deployment can run with replicas: 3 behind a Service, with all pods pointed at the same Postgres URL — tokens written by pod A are immediately visible to pod B.
• Centralized DB operations (backup, restore, replication, encryption at rest, monitoring) are handled by the operator's existing Postgres infrastructure rather than duplicated per-pod.
• No regression for single-user / local-development / docker-compose installs — the SQLite tempfile path is unchanged and remains the default when DATABASE_URL is unset.
• Test coverage doubles automatically: every test that uses the temp_storage fixture now runs against both SQLite and Postgres when TEST_DATABASE_URL is exported.

Negative

• One more thing operators have to think about for HA deployments (Postgres connection string, credentials secret, network policies).
• Adds SQLAlchemy + asyncpg to the runtime dependency set. SQLAlchemy was already transitively present via Alembic; asyncpg is genuinely new.
• The compatibility shim in storage.py is a small piece of bespoke code that future contributors need to understand. The alternative — rewriting every method body to SQLAlchemy idioms — was rejected as too risky for this PR but might be revisited.

Neutral

• The Alembic migration history was content-rewritten but its revision graph is unchanged (still 001 → 006), so existing SQLite deployments do not re-run anything.
• TOKEN_STORAGE_DB still works exactly as before; deployments that already set it require no changes.

Related

• ADR-022 Login Flow v2 — defines the per-user app password storage that this ADR centralizes.
• ADR-002 Vector sync authentication — explains the offline-access tokens that benefit most from HA storage.

Verification

1. uv run pytest tests/unit/ — SQLite path unchanged (1012 tests).
2. docker compose --profile postgres up -d postgres-test then TEST_DATABASE_URL=postgresql+asyncpg://mcp:mcp@localhost:5433/mcp uv run pytest tests/unit/test_app_password_storage.py tests/unit/test_webhook_storage.py — every test runs once per backend.
3. Manual end-to-end smoke against mcp-login-flow with a Postgres URL (commands in /home/chris/.claude/plans/spicy-enchanting-flurry.md → Verification).
4. k8s HA validation (after merge in homelab-argocd): replicas: 3, confirm session continuity through the Service.