Chris CoutinhoandClaude Opus 4.7
292cbb3292
Adds a `DATABASE_URL` setting that lets `RefreshTokenStorage` run against
any SQLAlchemy async backend, primarily `postgresql+asyncpg://...` for
HA k8s deployments. Default behavior is unchanged: when `DATABASE_URL` is
unset the server falls back to the existing `TOKEN_STORAGE_DB` path /
ephemeral SQLite tempfile.
Why
---
Today every MCP pod needs its own PVC to hold the SQLite file, which
pins the Deployment to one replica and blocks horizontal scaling. With
this change, operators can point all replicas at a shared Postgres
(CNPG, RDS, etc.) and the pods become stateless. Encryption stays in
Python (Fernet); the database only sees ciphertext.
What changed
------------
- `config.get_database_url()` resolves DATABASE_URL → TOKEN_STORAGE_DB →
ephemeral tempfile in that priority order.
- `RefreshTokenStorage` builds a process-shared `AsyncEngine` in
`initialize()`. SQLite gets NullPool; Postgres gets pool_size=10,
max_overflow=20, pool_pre_ping=True. 30 aiosqlite call sites adapted
via a thin `_DBConn` / `_Cursor` / `_Row` / `_ExecuteCtx` shim so
existing method bodies need no churn beyond the connection
context-manager swap.
- 7 `INSERT OR REPLACE` statements rewritten as portable
`INSERT ... ON CONFLICT (...) DO UPDATE` (SQLite ≥ 3.24, Postgres ≥ 9.5).
- `sqlite_master` legacy-detection lookup replaced with SQLAlchemy
inspector so the path works against either backend.
- File-permission hardening + parent-dir creation gated on
`is_sqlite_url(...)` — centralized backends manage their own filesystem.
- Alembic migrations 001/002/003/005 converted from raw `op.execute(SQL)`
to portable `op.create_table()` / `op.create_index()` with SQLAlchemy
types. All timestamp columns are `sa.BigInteger` so Postgres allocates
BIGINT (unix epochs don't fit in INT4). SQLite treats BIGINT as
INTEGER, so existing deployments at revision 006 see no schema drift.
- `migrations.py` + CLI take URLs; `db {upgrade,downgrade,current,history}`
gain `--database-url / -u` alongside the legacy `--database-path / -d`.
`get_current_revision()` uses SQLAlchemy inspector instead of raw
sqlite3, so the CLI works against Postgres too.
- `docker-compose.yml` adds a `postgres-test` service under the
`postgres` profile (pinned `postgres:16-alpine` digest) for
integration testing.
- Unit storage tests parametrized over backends via shared
`tests/fixtures/storage_backend.py` — every test in
`test_app_password_storage.py` and `test_webhook_storage.py` runs
once per backend that is available. Postgres is opted in by
`TEST_DATABASE_URL`.
- New `tests/integration/test_storage_postgres.py` (5 tests, marked
`postgres` + `integration`) covers refresh-token, app-password,
OAuth-session, webhook, and audit-log paths end-to-end on Postgres.
- New `docs/ADR-026-pluggable-database-backend.md` records the decision;
`docs/configuration.md` documents `DATABASE_URL` with examples.
Out of scope
------------
- No SQLite → Postgres data migration tool (clean cutover; tokens reissue
on next login, webhooks re-register on next sync tick).
- This repo does not provision Postgres. The matching helm chart change
lives in cbcoutinho/helm-charts (database.url / existingSecret values).
Verification
------------
- `uv run pytest tests/unit/` — 1012 passed, SQLite path unchanged.
- `docker compose --profile postgres up -d postgres-test`
- `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres -v`
— 5 passed.
- `TEST_DATABASE_URL=... uv run pytest tests/unit/test_app_password_storage.py
tests/unit/test_webhook_storage.py` — 50 passed (25 per backend).
- `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean.
Tracked on Astrolabe Cloud POC board, card #99.
---
_This PR was generated with the help of AI, and reviewed by a Human_
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
208 lines
7.4 KiB
Python
208 lines
7.4 KiB
Python
"""
|
|
Unit tests for Webhook Storage functionality.
|
|
|
|
Tests the webhook tracking methods in RefreshTokenStorage without
|
|
requiring real database connections or network calls.
|
|
|
|
Runs against both SQLite and Postgres backends — see the docstring on
|
|
``tests.fixtures.storage_backend`` for opt-in instructions.
|
|
"""
|
|
|
|
import tempfile
|
|
import time
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
|
|
|
|
pytestmark = pytest.mark.unit
|
|
|
|
|
|
@pytest.fixture
|
|
async def temp_storage(storage_backend):
|
|
"""Create a storage instance backed by either SQLite or Postgres."""
|
|
if storage_backend["kind"] == "sqlite":
|
|
with tempfile.TemporaryDirectory() as tmpdir:
|
|
db_path = Path(tmpdir) / "test_webhooks.db"
|
|
storage = RefreshTokenStorage(db_path=str(db_path), encryption_key=None)
|
|
await storage.initialize()
|
|
yield storage
|
|
else:
|
|
storage = RefreshTokenStorage(
|
|
database_url=storage_backend["url"], encryption_key=None
|
|
)
|
|
await storage.initialize()
|
|
try:
|
|
yield storage
|
|
finally:
|
|
await storage_backend["reset"]()
|
|
|
|
|
|
async def test_store_webhook(temp_storage):
|
|
"""Test storing a webhook."""
|
|
await temp_storage.store_webhook(webhook_id=123, preset_id="notes_sync")
|
|
|
|
webhooks = await temp_storage.list_all_webhooks()
|
|
assert len(webhooks) == 1
|
|
assert webhooks[0]["webhook_id"] == 123
|
|
assert webhooks[0]["preset_id"] == "notes_sync"
|
|
assert "created_at" in webhooks[0]
|
|
|
|
|
|
async def test_store_webhook_duplicate(temp_storage):
|
|
"""Test storing duplicate webhook replaces existing."""
|
|
await temp_storage.store_webhook(webhook_id=123, preset_id="notes_sync")
|
|
await temp_storage.store_webhook(webhook_id=123, preset_id="calendar_sync")
|
|
|
|
webhooks = await temp_storage.list_all_webhooks()
|
|
# Should only have one entry due to UNIQUE constraint
|
|
assert len(webhooks) == 1
|
|
assert webhooks[0]["preset_id"] == "calendar_sync"
|
|
|
|
|
|
async def test_get_webhooks_by_preset(temp_storage):
|
|
"""Test retrieving webhooks by preset."""
|
|
await temp_storage.store_webhook(webhook_id=123, preset_id="notes_sync")
|
|
await temp_storage.store_webhook(webhook_id=456, preset_id="notes_sync")
|
|
await temp_storage.store_webhook(webhook_id=789, preset_id="calendar_sync")
|
|
|
|
notes_webhooks = await temp_storage.get_webhooks_by_preset("notes_sync")
|
|
assert len(notes_webhooks) == 2
|
|
assert 123 in notes_webhooks
|
|
assert 456 in notes_webhooks
|
|
|
|
calendar_webhooks = await temp_storage.get_webhooks_by_preset("calendar_sync")
|
|
assert len(calendar_webhooks) == 1
|
|
assert 789 in calendar_webhooks
|
|
|
|
|
|
async def test_get_webhooks_by_preset_empty(temp_storage):
|
|
"""Test retrieving webhooks for non-existent preset."""
|
|
webhooks = await temp_storage.get_webhooks_by_preset("nonexistent")
|
|
assert len(webhooks) == 0
|
|
|
|
|
|
async def test_delete_webhook(temp_storage):
|
|
"""Test deleting a webhook."""
|
|
await temp_storage.store_webhook(webhook_id=123, preset_id="notes_sync")
|
|
await temp_storage.store_webhook(webhook_id=456, preset_id="notes_sync")
|
|
|
|
deleted = await temp_storage.delete_webhook(webhook_id=123)
|
|
assert deleted is True
|
|
|
|
webhooks = await temp_storage.get_webhooks_by_preset("notes_sync")
|
|
assert len(webhooks) == 1
|
|
assert 456 in webhooks
|
|
|
|
|
|
async def test_delete_webhook_nonexistent(temp_storage):
|
|
"""Test deleting non-existent webhook."""
|
|
deleted = await temp_storage.delete_webhook(webhook_id=999)
|
|
assert deleted is False
|
|
|
|
|
|
async def test_list_all_webhooks(temp_storage):
|
|
"""Test listing all webhooks."""
|
|
await temp_storage.store_webhook(webhook_id=123, preset_id="notes_sync")
|
|
await temp_storage.store_webhook(webhook_id=456, preset_id="calendar_sync")
|
|
await temp_storage.store_webhook(webhook_id=789, preset_id="notes_sync")
|
|
|
|
webhooks = await temp_storage.list_all_webhooks()
|
|
assert len(webhooks) == 3
|
|
|
|
# Verify all expected fields present
|
|
for webhook in webhooks:
|
|
assert "webhook_id" in webhook
|
|
assert "preset_id" in webhook
|
|
assert "created_at" in webhook
|
|
|
|
# Verify webhook IDs
|
|
webhook_ids = [w["webhook_id"] for w in webhooks]
|
|
assert 123 in webhook_ids
|
|
assert 456 in webhook_ids
|
|
assert 789 in webhook_ids
|
|
|
|
|
|
async def test_list_all_webhooks_empty(temp_storage):
|
|
"""Test listing webhooks when none exist."""
|
|
webhooks = await temp_storage.list_all_webhooks()
|
|
assert len(webhooks) == 0
|
|
|
|
|
|
async def test_clear_preset_webhooks(temp_storage):
|
|
"""Test clearing all webhooks for a preset."""
|
|
await temp_storage.store_webhook(webhook_id=123, preset_id="notes_sync")
|
|
await temp_storage.store_webhook(webhook_id=456, preset_id="notes_sync")
|
|
await temp_storage.store_webhook(webhook_id=789, preset_id="calendar_sync")
|
|
|
|
deleted_count = await temp_storage.clear_preset_webhooks("notes_sync")
|
|
assert deleted_count == 2
|
|
|
|
# Verify notes_sync webhooks are gone
|
|
notes_webhooks = await temp_storage.get_webhooks_by_preset("notes_sync")
|
|
assert len(notes_webhooks) == 0
|
|
|
|
# Verify calendar_sync webhook still exists
|
|
calendar_webhooks = await temp_storage.get_webhooks_by_preset("calendar_sync")
|
|
assert len(calendar_webhooks) == 1
|
|
assert 789 in calendar_webhooks
|
|
|
|
|
|
async def test_clear_preset_webhooks_nonexistent(temp_storage):
|
|
"""Test clearing webhooks for non-existent preset."""
|
|
deleted_count = await temp_storage.clear_preset_webhooks("nonexistent")
|
|
assert deleted_count == 0
|
|
|
|
|
|
async def test_webhook_timestamps(temp_storage):
|
|
"""Test that webhook timestamps are properly stored."""
|
|
start_time = time.time()
|
|
await temp_storage.store_webhook(webhook_id=123, preset_id="notes_sync")
|
|
end_time = time.time()
|
|
|
|
webhooks = await temp_storage.list_all_webhooks()
|
|
assert len(webhooks) == 1
|
|
|
|
created_at = webhooks[0]["created_at"]
|
|
assert start_time <= created_at <= end_time
|
|
|
|
|
|
async def test_storage_without_encryption_key():
|
|
"""Test that storage can be initialized without encryption key."""
|
|
with tempfile.TemporaryDirectory() as tmpdir:
|
|
db_path = Path(tmpdir) / "test_no_encryption.db"
|
|
storage = RefreshTokenStorage(db_path=str(db_path), encryption_key=None)
|
|
await storage.initialize()
|
|
|
|
# Webhook operations should work without encryption key
|
|
await storage.store_webhook(webhook_id=123, preset_id="notes_sync")
|
|
webhooks = await storage.get_webhooks_by_preset("notes_sync")
|
|
assert len(webhooks) == 1
|
|
assert 123 in webhooks
|
|
|
|
|
|
async def test_multiple_presets_independence(temp_storage):
|
|
"""Test that different presets maintain independent webhook lists."""
|
|
presets = ["notes_sync", "calendar_sync", "deck_sync", "files_sync"]
|
|
|
|
# Store webhooks for each preset
|
|
for i, preset in enumerate(presets):
|
|
webhook_id = 100 + i
|
|
await temp_storage.store_webhook(webhook_id=webhook_id, preset_id=preset)
|
|
|
|
# Verify each preset has exactly one webhook
|
|
for i, preset in enumerate(presets):
|
|
webhooks = await temp_storage.get_webhooks_by_preset(preset)
|
|
assert len(webhooks) == 1
|
|
assert (100 + i) in webhooks
|
|
|
|
# Clear one preset
|
|
deleted = await temp_storage.clear_preset_webhooks("notes_sync")
|
|
assert deleted == 1
|
|
|
|
# Verify other presets unchanged
|
|
for preset in ["calendar_sync", "deck_sync", "files_sync"]:
|
|
webhooks = await temp_storage.get_webhooks_by_preset(preset)
|
|
assert len(webhooks) == 1
|