feat(storage): pluggable database backend via DATABASE_URL (ADR-026)
Adds a `DATABASE_URL` setting that lets `RefreshTokenStorage` run against
any SQLAlchemy async backend, primarily `postgresql+asyncpg://...` for
HA k8s deployments. Default behavior is unchanged: when `DATABASE_URL` is
unset the server falls back to the existing `TOKEN_STORAGE_DB` path /
ephemeral SQLite tempfile.
Why
---
Today every MCP pod needs its own PVC to hold the SQLite file, which
pins the Deployment to one replica and blocks horizontal scaling. With
this change, operators can point all replicas at a shared Postgres
(CNPG, RDS, etc.) and the pods become stateless. Encryption stays in
Python (Fernet); the database only sees ciphertext.
What changed
------------
- `config.get_database_url()` resolves DATABASE_URL → TOKEN_STORAGE_DB →
ephemeral tempfile in that priority order.
- `RefreshTokenStorage` builds a process-shared `AsyncEngine` in
`initialize()`. SQLite gets NullPool; Postgres gets pool_size=10,
max_overflow=20, pool_pre_ping=True. 30 aiosqlite call sites adapted
via a thin `_DBConn` / `_Cursor` / `_Row` / `_ExecuteCtx` shim so
existing method bodies need no churn beyond the connection
context-manager swap.
- 7 `INSERT OR REPLACE` statements rewritten as portable
`INSERT ... ON CONFLICT (...) DO UPDATE` (SQLite ≥ 3.24, Postgres ≥ 9.5).
- `sqlite_master` legacy-detection lookup replaced with SQLAlchemy
inspector so the path works against either backend.
- File-permission hardening + parent-dir creation gated on
`is_sqlite_url(...)` — centralized backends manage their own filesystem.
- Alembic migrations 001/002/003/005 converted from raw `op.execute(SQL)`
to portable `op.create_table()` / `op.create_index()` with SQLAlchemy
types. All timestamp columns are `sa.BigInteger` so Postgres allocates
BIGINT (unix epochs don't fit in INT4). SQLite treats BIGINT as
INTEGER, so existing deployments at revision 006 see no schema drift.
- `migrations.py` + CLI take URLs; `db {upgrade,downgrade,current,history}`
gain `--database-url / -u` alongside the legacy `--database-path / -d`.
`get_current_revision()` uses SQLAlchemy inspector instead of raw
sqlite3, so the CLI works against Postgres too.
- `docker-compose.yml` adds a `postgres-test` service under the
`postgres` profile (pinned `postgres:16-alpine` digest) for
integration testing.
- Unit storage tests parametrized over backends via shared
`tests/fixtures/storage_backend.py` — every test in
`test_app_password_storage.py` and `test_webhook_storage.py` runs
once per backend that is available. Postgres is opted in by
`TEST_DATABASE_URL`.
- New `tests/integration/test_storage_postgres.py` (5 tests, marked
`postgres` + `integration`) covers refresh-token, app-password,
OAuth-session, webhook, and audit-log paths end-to-end on Postgres.
- New `docs/ADR-026-pluggable-database-backend.md` records the decision;
`docs/configuration.md` documents `DATABASE_URL` with examples.
Out of scope
------------
- No SQLite → Postgres data migration tool (clean cutover; tokens reissue
on next login, webhooks re-register on next sync tick).
- This repo does not provision Postgres. The matching helm chart change
lives in cbcoutinho/helm-charts (database.url / existingSecret values).
Verification
------------
- `uv run pytest tests/unit/` — 1012 passed, SQLite path unchanged.
- `docker compose --profile postgres up -d postgres-test`
- `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres -v`
— 5 passed.
- `TEST_DATABASE_URL=... uv run pytest tests/unit/test_app_password_storage.py
tests/unit/test_webhook_storage.py` — 50 passed (25 per backend).
- `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean.
Tracked on Astrolabe Cloud POC board, card #99.
---
_This PR was generated with the help of AI, and reviewed by a Human_
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
15a7e680a6
commit
292cbb3292
@@ -2,6 +2,11 @@
|
||||
|
||||
import pytest
|
||||
|
||||
# Re-export the parametrized storage backend fixture so it's auto-discovered
|
||||
# by every unit test that names it as a parameter, without each test module
|
||||
# having to import it explicitly.
|
||||
from tests.fixtures.storage_backend import storage_backend # noqa: F401
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reload_dynaconf_after_test():
|
||||
|
||||
@@ -3,6 +3,12 @@ Unit tests for App Password Storage functionality.
|
||||
|
||||
Tests the app password methods in RefreshTokenStorage for multi-user
|
||||
BasicAuth mode background sync.
|
||||
|
||||
These tests are parametrized over both supported backends so the storage
|
||||
layer is exercised against SQLite (default, always runs) and Postgres (gated
|
||||
on ``TEST_DATABASE_URL``; bring up ``docker compose --profile postgres up
|
||||
-d postgres-test`` and export ``TEST_DATABASE_URL=postgresql+asyncpg://mcp:mcp@localhost:5433/mcp``
|
||||
to opt in).
|
||||
"""
|
||||
|
||||
import tempfile
|
||||
@@ -23,15 +29,32 @@ def encryption_key():
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def temp_storage(encryption_key):
|
||||
"""Create temporary storage instance with encryption for testing."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
db_path = Path(tmpdir) / "test_app_passwords.db"
|
||||
async def temp_storage(encryption_key, storage_backend):
|
||||
"""Create a storage instance backed by either SQLite or Postgres.
|
||||
|
||||
The ``storage_backend`` fixture is parametrized by pytest, so every test
|
||||
that uses ``temp_storage`` runs once per backend that is available in
|
||||
the current environment.
|
||||
"""
|
||||
if storage_backend["kind"] == "sqlite":
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
db_path = Path(tmpdir) / "test_app_passwords.db"
|
||||
storage = RefreshTokenStorage(
|
||||
db_path=str(db_path), encryption_key=encryption_key
|
||||
)
|
||||
await storage.initialize()
|
||||
yield storage
|
||||
else:
|
||||
storage = RefreshTokenStorage(
|
||||
db_path=str(db_path), encryption_key=encryption_key
|
||||
database_url=storage_backend["url"], encryption_key=encryption_key
|
||||
)
|
||||
await storage.initialize()
|
||||
yield storage
|
||||
try:
|
||||
yield storage
|
||||
finally:
|
||||
# Each test gets an isolated schema; tear it down so the next
|
||||
# parametrized run starts clean.
|
||||
await storage_backend["reset"]()
|
||||
|
||||
|
||||
async def test_store_app_password(temp_storage):
|
||||
|
||||
@@ -3,6 +3,9 @@ Unit tests for Webhook Storage functionality.
|
||||
|
||||
Tests the webhook tracking methods in RefreshTokenStorage without
|
||||
requiring real database connections or network calls.
|
||||
|
||||
Runs against both SQLite and Postgres backends — see the docstring on
|
||||
``tests.fixtures.storage_backend`` for opt-in instructions.
|
||||
"""
|
||||
|
||||
import tempfile
|
||||
@@ -17,14 +20,23 @@ pytestmark = pytest.mark.unit
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def temp_storage():
|
||||
"""Create temporary storage instance for testing."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
db_path = Path(tmpdir) / "test_webhooks.db"
|
||||
# No encryption key needed for webhook tracking
|
||||
storage = RefreshTokenStorage(db_path=str(db_path), encryption_key=None)
|
||||
async def temp_storage(storage_backend):
|
||||
"""Create a storage instance backed by either SQLite or Postgres."""
|
||||
if storage_backend["kind"] == "sqlite":
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
db_path = Path(tmpdir) / "test_webhooks.db"
|
||||
storage = RefreshTokenStorage(db_path=str(db_path), encryption_key=None)
|
||||
await storage.initialize()
|
||||
yield storage
|
||||
else:
|
||||
storage = RefreshTokenStorage(
|
||||
database_url=storage_backend["url"], encryption_key=None
|
||||
)
|
||||
await storage.initialize()
|
||||
yield storage
|
||||
try:
|
||||
yield storage
|
||||
finally:
|
||||
await storage_backend["reset"]()
|
||||
|
||||
|
||||
async def test_store_webhook(temp_storage):
|
||||
|
||||
Reference in New Issue
Block a user