feat(storage): pluggable database backend via DATABASE_URL (ADR-026)
Adds a `DATABASE_URL` setting that lets `RefreshTokenStorage` run against
any SQLAlchemy async backend, primarily `postgresql+asyncpg://...` for
HA k8s deployments. Default behavior is unchanged: when `DATABASE_URL` is
unset the server falls back to the existing `TOKEN_STORAGE_DB` path /
ephemeral SQLite tempfile.
Why
---
Today every MCP pod needs its own PVC to hold the SQLite file, which
pins the Deployment to one replica and blocks horizontal scaling. With
this change, operators can point all replicas at a shared Postgres
(CNPG, RDS, etc.) and the pods become stateless. Encryption stays in
Python (Fernet); the database only sees ciphertext.
What changed
------------
- `config.get_database_url()` resolves DATABASE_URL → TOKEN_STORAGE_DB →
ephemeral tempfile in that priority order.
- `RefreshTokenStorage` builds a process-shared `AsyncEngine` in
`initialize()`. SQLite gets NullPool; Postgres gets pool_size=10,
max_overflow=20, pool_pre_ping=True. 30 aiosqlite call sites adapted
via a thin `_DBConn` / `_Cursor` / `_Row` / `_ExecuteCtx` shim so
existing method bodies need no churn beyond the connection
context-manager swap.
- 7 `INSERT OR REPLACE` statements rewritten as portable
`INSERT ... ON CONFLICT (...) DO UPDATE` (SQLite ≥ 3.24, Postgres ≥ 9.5).
- `sqlite_master` legacy-detection lookup replaced with SQLAlchemy
inspector so the path works against either backend.
- File-permission hardening + parent-dir creation gated on
`is_sqlite_url(...)` — centralized backends manage their own filesystem.
- Alembic migrations 001/002/003/005 converted from raw `op.execute(SQL)`
to portable `op.create_table()` / `op.create_index()` with SQLAlchemy
types. All timestamp columns are `sa.BigInteger` so Postgres allocates
BIGINT (unix epochs don't fit in INT4). SQLite treats BIGINT as
INTEGER, so existing deployments at revision 006 see no schema drift.
- `migrations.py` + CLI take URLs; `db {upgrade,downgrade,current,history}`
gain `--database-url / -u` alongside the legacy `--database-path / -d`.
`get_current_revision()` uses SQLAlchemy inspector instead of raw
sqlite3, so the CLI works against Postgres too.
- `docker-compose.yml` adds a `postgres-test` service under the
`postgres` profile (pinned `postgres:16-alpine` digest) for
integration testing.
- Unit storage tests parametrized over backends via shared
`tests/fixtures/storage_backend.py` — every test in
`test_app_password_storage.py` and `test_webhook_storage.py` runs
once per backend that is available. Postgres is opted in by
`TEST_DATABASE_URL`.
- New `tests/integration/test_storage_postgres.py` (5 tests, marked
`postgres` + `integration`) covers refresh-token, app-password,
OAuth-session, webhook, and audit-log paths end-to-end on Postgres.
- New `docs/ADR-026-pluggable-database-backend.md` records the decision;
`docs/configuration.md` documents `DATABASE_URL` with examples.
Out of scope
------------
- No SQLite → Postgres data migration tool (clean cutover; tokens reissue
on next login, webhooks re-register on next sync tick).
- This repo does not provision Postgres. The matching helm chart change
lives in cbcoutinho/helm-charts (database.url / existingSecret values).
Verification
------------
- `uv run pytest tests/unit/` — 1012 passed, SQLite path unchanged.
- `docker compose --profile postgres up -d postgres-test`
- `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres -v`
— 5 passed.
- `TEST_DATABASE_URL=... uv run pytest tests/unit/test_app_password_storage.py
tests/unit/test_webhook_storage.py` — 50 passed (25 per backend).
- `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean.
Tracked on Astrolabe Cloud POC board, card #99.
---
_This PR was generated with the help of AI, and reviewed by a Human_
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
15a7e680a6
commit
292cbb3292
@@ -0,0 +1,169 @@
|
||||
# ADR-026: Pluggable database backend (DATABASE_URL)
|
||||
|
||||
## Status
|
||||
|
||||
Accepted — 2026-05-16
|
||||
|
||||
## Context
|
||||
|
||||
`RefreshTokenStorage` (in `nextcloud_mcp_server/auth/storage.py`) holds all
|
||||
of the MCP server's persistent state: refresh tokens, OAuth client
|
||||
credentials, OAuth sessions, browser sessions, app passwords, login-flow
|
||||
sessions, audit logs, and webhook registrations. Until this ADR it was
|
||||
backed by a single SQLite file, with the path configured by
|
||||
`TOKEN_STORAGE_DB`.
|
||||
|
||||
This works well for single-user deployments but blocks horizontal scaling
|
||||
in Kubernetes:
|
||||
|
||||
- Every pod needs its own PVC (ReadWriteOnce) or a ReadWriteMany volume.
|
||||
- Tokens stored on pod A are invisible to pod B, so a Service can only
|
||||
route traffic to one pod at a time.
|
||||
- Restart / re-deploy cycles either drop the volume (token loss) or
|
||||
require coordinated PVC handling.
|
||||
- Backup, encryption-at-rest, and multi-region replication become
|
||||
per-pod concerns rather than centrally managed DB concerns.
|
||||
|
||||
We needed a way for pods to be stateless and share a centralized store
|
||||
without giving up the zero-config SQLite path that single-user installs and
|
||||
local development rely on.
|
||||
|
||||
## Decision
|
||||
|
||||
Introduce a `DATABASE_URL` setting that accepts any SQLAlchemy async URL,
|
||||
with `sqlite+aiosqlite:///...` remaining the default. The runtime keeps a
|
||||
single linear migration history and a single `RefreshTokenStorage` class —
|
||||
the backend is selected purely by the URL.
|
||||
|
||||
### Resolution order
|
||||
|
||||
`get_database_url()` (in `nextcloud_mcp_server/config.py`) returns:
|
||||
|
||||
1. `DATABASE_URL` if set — wins over everything.
|
||||
2. Otherwise `sqlite+aiosqlite:///{get_token_db_path()}`, so the legacy
|
||||
`TOKEN_STORAGE_DB` env var and the process-local ephemeral tempfile
|
||||
fallback both keep working unchanged.
|
||||
|
||||
### Why SQLAlchemy Core + async engine, not an ABC with parallel drivers
|
||||
|
||||
Two alternatives were considered:
|
||||
|
||||
| Option | Why rejected |
|
||||
|---|---|
|
||||
| Define a `Storage` ABC with `SQLiteStorage` (aiosqlite) and `PostgresStorage` (asyncpg) implementations | Doubles the surface area — every schema change has to land in two backends, with two sets of migrations, two SQL dialects, two upsert idioms. Diverges over time. |
|
||||
| Switch to a full SQLAlchemy ORM (declarative models) | Larger refactor; the existing explicit-SQL style is intentional and well-understood by reviewers. |
|
||||
| **Keep `RefreshTokenStorage` and put SQLAlchemy Core under it** *(chosen)* | One method body per operation, one migration history (Alembic is already SQLAlchemy-based). The URL drives dialect, pool, and DDL. |
|
||||
|
||||
A thin compatibility shim (`_DBConn` / `_Cursor` / `_Row` in `storage.py`)
|
||||
adapts the `async with aiosqlite.connect(...) as db: async with
|
||||
db.execute(...) as cursor: ...` idiom to SQLAlchemy `AsyncEngine` /
|
||||
`AsyncConnection`. Existing method bodies needed only their connection
|
||||
context-manager swapped; `?` placeholders are rewritten to named binds on
|
||||
the fly. The seven `INSERT OR REPLACE` statements were rewritten as
|
||||
portable `INSERT ... ON CONFLICT (...) DO UPDATE` (SQLite ≥ 3.24, Postgres
|
||||
≥ 9.5; we already require SQLite ≥ 3.35 elsewhere).
|
||||
|
||||
### Encryption stays in Python (Fernet), not the DB
|
||||
|
||||
The DB only ever sees ciphertext for sensitive columns
|
||||
(`encrypted_token`, `encrypted_client_secret`, `encrypted_password`,
|
||||
`encrypted_poll_token`). The Fernet key remains a `TOKEN_ENCRYPTION_KEY`
|
||||
env var, applied in Python before INSERT and after SELECT. This means:
|
||||
|
||||
- Switching backends does not invalidate or re-key existing data.
|
||||
- Postgres-level features like `pgcrypto` are not required.
|
||||
- Operators rotating the encryption key still go through the existing
|
||||
Python path.
|
||||
|
||||
### DDL portability
|
||||
|
||||
All Alembic migrations were rewritten from raw `op.execute("CREATE TABLE
|
||||
...")` strings to `op.create_table()` / `op.create_index()` calls with
|
||||
SQLAlchemy types. Notable choices:
|
||||
|
||||
- All `*_at` / expiration / timestamp columns use `sa.BigInteger` —
|
||||
Postgres `INTEGER` is 32-bit and unix epochs are already past that range.
|
||||
SQLite treats `BIGINT` and `INTEGER` identically (dynamic typing) so
|
||||
this is backwards compatible.
|
||||
- `BLOB` → `sa.LargeBinary` (becomes `BYTEA` on Postgres).
|
||||
- `BOOLEAN DEFAULT FALSE` → `sa.Boolean, server_default=sa.false()`.
|
||||
- Existing SQLite deployments are at revision `006` and skip the
|
||||
rewritten migrations entirely — content rewrites are safe.
|
||||
|
||||
### No data migration, no shipped Postgres
|
||||
|
||||
Two scope decisions worth recording:
|
||||
|
||||
1. **Clean cutover, no SQLite → Postgres data migration tool.** Tokens
|
||||
are reissued on the next login; webhooks re-register on the next sync
|
||||
tick. Acceptable because the ephemeral-default already implies this,
|
||||
and the data being preserved (audit logs, OAuth sessions) is either
|
||||
short-lived or reconstructible.
|
||||
2. **Bring-your-own database.** The MCP server consumes a
|
||||
`DATABASE_URL`; it does not provision Postgres itself. Operators use
|
||||
CNPG, RDS, the project's existing Helm chart with a sub-chart, etc.
|
||||
The `postgres-test` service in `docker-compose.yml` exists only for
|
||||
integration tests and manual HA smoke testing — it is gated on the
|
||||
`postgres` profile and is not the recommended production pattern.
|
||||
|
||||
### CLI changes
|
||||
|
||||
The `nextcloud-mcp-server db {upgrade,downgrade,current,history}` commands
|
||||
gain a `--database-url / -u` flag (env `DATABASE_URL`) alongside the
|
||||
existing `--database-path / -d` (env `TOKEN_STORAGE_DB`). `-u` wins over
|
||||
`-d`; both fall back to `get_database_url()`.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- MCP server pods become stateless. A Kubernetes Deployment can run with
|
||||
`replicas: 3` behind a Service, with all pods pointed at the same
|
||||
Postgres URL — tokens written by pod A are immediately visible to pod B.
|
||||
- Centralized DB operations (backup, restore, replication, encryption at
|
||||
rest, monitoring) are handled by the operator's existing Postgres
|
||||
infrastructure rather than duplicated per-pod.
|
||||
- No regression for single-user / local-development / docker-compose
|
||||
installs — the SQLite tempfile path is unchanged and remains the default
|
||||
when `DATABASE_URL` is unset.
|
||||
- Test coverage doubles automatically: every test that uses the
|
||||
`temp_storage` fixture now runs against both SQLite and Postgres when
|
||||
`TEST_DATABASE_URL` is exported.
|
||||
|
||||
### Negative
|
||||
|
||||
- One more thing operators have to think about for HA deployments
|
||||
(Postgres connection string, credentials secret, network policies).
|
||||
- Adds SQLAlchemy + asyncpg to the runtime dependency set. SQLAlchemy was
|
||||
already transitively present via Alembic; asyncpg is genuinely new.
|
||||
- The compatibility shim in `storage.py` is a small piece of bespoke code
|
||||
that future contributors need to understand. The alternative — rewriting
|
||||
every method body to SQLAlchemy idioms — was rejected as too risky for
|
||||
this PR but might be revisited.
|
||||
|
||||
### Neutral
|
||||
|
||||
- The Alembic migration history was content-rewritten but its revision
|
||||
graph is unchanged (still `001 → 006`), so existing SQLite deployments
|
||||
do not re-run anything.
|
||||
- `TOKEN_STORAGE_DB` still works exactly as before; deployments that
|
||||
already set it require no changes.
|
||||
|
||||
## Related
|
||||
|
||||
- [ADR-022 Login Flow v2](ADR-022-deployment-mode-consolidation.md) —
|
||||
defines the per-user app password storage that this ADR centralizes.
|
||||
- [ADR-002 Vector sync authentication](ADR-002-vector-sync-authentication.md)
|
||||
— explains the offline-access tokens that benefit most from HA storage.
|
||||
|
||||
## Verification
|
||||
|
||||
1. `uv run pytest tests/unit/` — SQLite path unchanged (1012 tests).
|
||||
2. `docker compose --profile postgres up -d postgres-test` then
|
||||
`TEST_DATABASE_URL=postgresql+asyncpg://mcp:mcp@localhost:5433/mcp uv run pytest tests/unit/test_app_password_storage.py tests/unit/test_webhook_storage.py`
|
||||
— every test runs once per backend.
|
||||
3. Manual end-to-end smoke against `mcp-login-flow` with a Postgres URL
|
||||
(commands in `/home/chris/.claude/plans/spicy-enchanting-flurry.md` →
|
||||
Verification).
|
||||
4. k8s HA validation (after merge in `homelab-argocd`): `replicas: 3`,
|
||||
confirm session continuity through the Service.
|
||||
@@ -118,6 +118,48 @@ See [Login Flow v2](login-flow-v2.md) for full setup, scope reference, and troub
|
||||
|
||||
---
|
||||
|
||||
## Centralized Token Storage (DATABASE_URL, Optional)
|
||||
|
||||
By default the MCP server stores tokens / sessions / app passwords in a
|
||||
local SQLite file (`TOKEN_STORAGE_DB`, falling back to a per-process
|
||||
tempfile). For HA Kubernetes deployments where you need multiple
|
||||
stateless pods to share state, point the server at a centralized
|
||||
database via `DATABASE_URL`.
|
||||
|
||||
```env
|
||||
# Centralized Postgres backend (HA k8s deployments)
|
||||
DATABASE_URL=postgresql+asyncpg://mcp:secret@postgres.svc.cluster.local:5432/mcp
|
||||
TOKEN_ENCRYPTION_KEY=<fernet-key>
|
||||
```
|
||||
|
||||
| Variable | Required | Description |
|
||||
|----------|----------|-------------|
|
||||
| `DATABASE_URL` | Optional | SQLAlchemy async URL for any supported backend. When set, wins over `TOKEN_STORAGE_DB`. Primary supported targets: `postgresql+asyncpg://...` (recommended for HA) and `sqlite+aiosqlite:///...` (development). |
|
||||
| `TOKEN_STORAGE_DB` | Optional | Legacy SQLite-only path. Used when `DATABASE_URL` is unset. Falls back to a per-process ephemeral tempfile when both are unset. |
|
||||
|
||||
Notes:
|
||||
|
||||
- **Bring-your-own DB.** The MCP server doesn't provision the database;
|
||||
it just consumes the URL. Use CNPG, RDS, your existing Helm chart's
|
||||
Postgres sub-chart, etc.
|
||||
- **Encryption stays in the app.** `TOKEN_ENCRYPTION_KEY` (Fernet) is
|
||||
applied in Python; the database only ever sees ciphertext for
|
||||
sensitive columns. You don't need `pgcrypto`.
|
||||
- **Schema is managed automatically.** On startup the server runs
|
||||
Alembic migrations against the configured backend. Existing SQLite
|
||||
deployments are stamped at the current revision and skip re-execution.
|
||||
- **No data migration tool.** Moving from SQLite to Postgres is a clean
|
||||
cutover — tokens are reissued on the next login, webhooks
|
||||
re-register on the next sync tick.
|
||||
- **Testing a Postgres backend locally:** `docker compose --profile
|
||||
postgres up -d postgres-test` then export
|
||||
`DATABASE_URL=postgresql+asyncpg://mcp:mcp@localhost:5433/mcp`.
|
||||
|
||||
See [ADR-026 Pluggable database backend](ADR-026-pluggable-database-backend.md)
|
||||
for the architecture rationale.
|
||||
|
||||
---
|
||||
|
||||
## SSL/TLS Configuration (Optional)
|
||||
|
||||
If your Nextcloud instance uses a self-signed certificate or a private CA (common with reverse proxies like Traefik or Caddy), the MCP server will reject the connection by default. Use these settings to configure certificate verification.
|
||||
|
||||
Reference in New Issue
Block a user