Files
mcp-nextcloud/tests/integration/test_storage_postgres.py
T
Chris CoutinhoandClaude Opus 4.7 f2b7bf132f fix(storage): address PR #798 review feedback (credentials, asyncpg extra, TLS, pool)
Round-2 fixes after the bot review on PR #798 plus two user follow-ups
(self-signed Postgres support; asyncpg should be a PyPI extra). Folded
into the same PR rather than a follow-up since the work is still
unmerged.

Security
--------
- Mask database credentials in all 5 log call sites (storage.py × 4,
  migrations.py × 1) via a new `mask_db_password()` helper in config.py.
  Uses SQLAlchemy's `make_url(...).render_as_string(hide_password=True)`
  with a regex fallback so the masking path never raises.
- New `tests/unit/test_storage_logging.py` asserts a sentinel password
  never appears in `caplog` during `RefreshTokenStorage.initialize()`.

Distribution
------------
- `asyncpg` moved to `[project.optional-dependencies] postgres` so a
  vanilla `pip install nextcloud-mcp-server` no longer pulls in the
  ~5 MB C extension. The Docker image runs `uv sync --extra postgres`,
  so containerized deployments are unchanged.
- When `DATABASE_URL=postgresql+asyncpg://...` is set on a venv missing
  the extra, `RefreshTokenStorage.initialize()` raises a friendly
  RuntimeError pointing at `[postgres]` rather than the generic
  ModuleNotFoundError.

TLS for the Postgres backend
----------------------------
- New `DATABASE_VERIFY_SSL` + `DATABASE_CA_BUNDLE` env vars mirror the
  existing `NEXTCLOUD_VERIFY_SSL` / `NEXTCLOUD_CA_BUNDLE` pattern
  (validators in Settings.__post_init__, `get_database_ssl()` helper
  alongside `get_nextcloud_ssl_verify()`). `DATABASE_VERIFY_SSL=false`
  wins over `DATABASE_CA_BUNDLE` for incident-response convenience.
- Default is **None** rather than True — keeps PR #798's behavior
  intact for cluster-internal Postgres that runs without TLS. Operators
  opt into verify-full or supply a private CA. ADR-026 records the
  reasoning vs the Nextcloud HTTPS default.
- Engine factory in `storage.py` passes `ssl` via `connect_args` only
  when `get_database_ssl()` returns non-None; otherwise asyncpg's
  default (`prefer`) applies.
- Storage logs which TLS mode is active at INFO (no secret material).

Configurable connection pool
----------------------------
- `DATABASE_POOL_SIZE` (default 10) and `DATABASE_MAX_OVERFLOW`
  (default 20) replace the hardcoded engine values. With many replicas
  this can blow past managed-Postgres `max_connections=100`; tune down
  for large fleets.
- gte-1 / gte-0 validators in __post_init__ reject 0/negative pool
  sizes at startup with the offending value in the error.

Consistency polish
------------------
- Migration 006: convert raw `op.execute("ALTER TABLE ... ADD COLUMN")`
  to `op.batch_alter_table(...).add_column(sa.Column("nonce", sa.Text))`
  for stylistic consistency with the rewritten 001-005. Downgrade now
  drops the column instead of being a no-op.
- `registered_webhooks.created_at` standardized from `sa.Float` to
  `sa.BigInteger` (all other `*_at` columns); `store_webhook()` casts
  `time.time()` → `int`.
- `is_sqlite_url()` made case-insensitive.

Testing
-------
- New `tests/integration/test_storage_postgres.py::test_cleanup_expired_roundtrip`
  exercises `cleanup_expired_tokens`, `cleanup_expired_sessions`, and
  `cleanup_expired_browser_sessions` — relies on DELETE rowcount,
  historically dialect-tricky.
- `tests/unit/test_ssl_config.py` extended with `TestDatabaseSSLSettings`
  + `TestGetDatabaseSSL` classes (9 new tests) mirroring the existing
  Nextcloud SSL tests one-for-one.

Docs
----
- `docs/configuration.md` Centralized-Storage section grew the four new
  env vars + a homelab example with a private CA.
- `docs/ADR-026` grew Distribution, TLS, and `alembic/env.py` async-pattern
  subsections explaining the non-obvious design choices.

Helm chart counterpart in cbcoutinho/helm-charts PR #34 (separate
commit on `feat/nextcloud-mcp-server-database-url`).

Verification
------------
- `uv run pytest tests/unit/` — 1025 passed.
- `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 6 passed (including new cleanup test).
- `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean.

Tracked on Astrolabe Cloud POC board, card #99.

---

_This PR was generated with the help of AI, and reviewed by a Human_

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 18:53:45 +02:00

221 lines
7.9 KiB
Python

"""End-to-end Postgres backend smoke for RefreshTokenStorage (ADR-026).
Exercises every storage method touched by the SQLAlchemy / asyncpg port
against a fresh Postgres schema. The test is opt-in: it requires the
``postgres-test`` docker-compose service to be running and
``TEST_DATABASE_URL`` to be exported.
Bring up the dependency once::
docker compose --profile postgres up -d postgres-test
export TEST_DATABASE_URL=postgresql+asyncpg://mcp:mcp@localhost:5433/mcp
Then run::
uv run pytest tests/integration/test_storage_postgres.py -v -m postgres
When ``TEST_DATABASE_URL`` is unset (or the service is unreachable) the
test is skipped so the full suite still passes locally without Docker.
"""
from __future__ import annotations
import os
import socket
from urllib.parse import urlparse
import pytest
from cryptography.fernet import Fernet
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
pytestmark = [pytest.mark.integration, pytest.mark.postgres]
def _postgres_url() -> str | None:
return os.environ.get("TEST_DATABASE_URL") or None
def _reachable(url: str) -> bool:
parsed = urlparse(url)
try:
with socket.create_connection(
(parsed.hostname or "localhost", parsed.port or 5432), timeout=1.0
):
return True
except OSError:
return False
@pytest.fixture
def postgres_url() -> str:
url = _postgres_url()
if not url:
pytest.skip(
"TEST_DATABASE_URL not set — run "
"`docker compose --profile postgres up -d postgres-test` and export "
"TEST_DATABASE_URL=postgresql+asyncpg://mcp:mcp@localhost:5433/mcp"
)
if not _reachable(url):
pytest.skip(f"Postgres at {url} is not reachable")
return url
@pytest.fixture
async def reset_schema(postgres_url: str):
"""Drop+recreate the public schema before and after each test."""
from sqlalchemy import text
from sqlalchemy.ext.asyncio import create_async_engine
async def _reset() -> None:
engine = create_async_engine(postgres_url, future=True)
try:
async with engine.begin() as conn:
await conn.execute(text("DROP SCHEMA public CASCADE"))
await conn.execute(text("CREATE SCHEMA public"))
finally:
await engine.dispose()
await _reset()
yield
await _reset()
@pytest.fixture
async def storage(postgres_url: str, reset_schema):
key = Fernet.generate_key()
s = RefreshTokenStorage(database_url=postgres_url, encryption_key=key)
await s.initialize()
yield s
async def test_refresh_token_roundtrip(storage: RefreshTokenStorage):
"""Store + retrieve + upsert + delete a refresh token end-to-end."""
await storage.store_refresh_token(
user_id="alice", refresh_token="rt-1", expires_at=9_999_999_999
)
tok = await storage.get_refresh_token("alice")
assert tok is not None
assert tok["refresh_token"] == "rt-1"
assert tok["expires_at"] == 9_999_999_999
# Upsert preserves user_id, swaps token contents.
await storage.store_refresh_token(
user_id="alice", refresh_token="rt-2", expires_at=9_999_999_999
)
tok = await storage.get_refresh_token("alice")
assert tok is not None and tok["refresh_token"] == "rt-2"
assert await storage.delete_refresh_token("alice") is True
assert await storage.get_refresh_token("alice") is None
async def test_app_password_roundtrip(storage: RefreshTokenStorage):
"""Store + retrieve + replace + delete a scoped app password."""
await storage.store_app_password(user_id="bob", app_password="pw-1")
assert await storage.get_app_password("bob") == "pw-1"
# Replace path exercises the ON CONFLICT DO UPDATE on the singleton row.
await storage.store_app_password(user_id="bob", app_password="pw-2")
assert await storage.get_app_password("bob") == "pw-2"
assert await storage.delete_app_password("bob") is True
assert await storage.get_app_password("bob") is None
async def test_oauth_session_lifecycle(storage: RefreshTokenStorage):
"""Cover the ADR-004 progressive-consent session table."""
await storage.store_oauth_session(
session_id="sess-1",
client_redirect_uri="http://localhost:12345/callback",
mcp_authorization_code="mcp-code-abc",
flow_type="hybrid",
ttl_seconds=600,
)
fetched = await storage.get_oauth_session("sess-1")
assert fetched is not None
assert fetched["mcp_authorization_code"] == "mcp-code-abc"
by_code = await storage.get_oauth_session_by_mcp_code("mcp-code-abc")
assert by_code is not None and by_code["session_id"] == "sess-1"
async def test_webhook_tracking(storage: RefreshTokenStorage):
"""Tracks webhook ↔ preset mappings via ON CONFLICT upserts."""
await storage.store_webhook(webhook_id=101, preset_id="notes_sync")
await storage.store_webhook(webhook_id=202, preset_id="notes_sync")
await storage.store_webhook(webhook_id=303, preset_id="calendar_sync")
assert sorted(await storage.get_webhooks_by_preset("notes_sync")) == [101, 202]
assert await storage.get_webhooks_by_preset("calendar_sync") == [303]
# Re-storing the same webhook_id is a no-op upsert.
await storage.store_webhook(webhook_id=101, preset_id="notes_sync")
assert sorted(await storage.get_webhooks_by_preset("notes_sync")) == [101, 202]
assert await storage.delete_webhook(webhook_id=101) is True
assert await storage.get_webhooks_by_preset("notes_sync") == [202]
async def test_audit_log_capture(storage: RefreshTokenStorage):
"""Audit events from upstream methods land in audit_logs."""
await storage.store_app_password(user_id="carol", app_password="x")
logs = await storage.get_audit_logs(user_id="carol", limit=10)
assert any(entry["event"] == "store_app_password" for entry in logs)
async def test_cleanup_expired_roundtrip(storage: RefreshTokenStorage):
"""``cleanup_expired_*`` paths rely on DELETE rowcount across dialects.
Regression guard for the bot review on PR #798 — the original
integration tests didn't exercise these methods, which historically
have been a source of dialect-portability bugs.
"""
# Insert one fresh + one expired refresh token.
await storage.store_refresh_token(
user_id="fresh-user", refresh_token="fresh", expires_at=9_999_999_999
)
await storage.store_refresh_token(
user_id="expired-user", refresh_token="stale", expires_at=1
)
# Insert one fresh + one expired OAuth session.
await storage.store_oauth_session(
session_id="sess-fresh",
client_redirect_uri="http://localhost/cb",
mcp_authorization_code="code-fresh",
ttl_seconds=600,
)
await storage.store_oauth_session(
session_id="sess-stale",
client_redirect_uri="http://localhost/cb",
mcp_authorization_code="code-stale",
ttl_seconds=-3600, # expires_at = now - 1h
)
# Insert one fresh + one expired browser session.
await storage.create_browser_session(
session_id="bs-fresh", user_id="alice", ttl_seconds=600
)
await storage.create_browser_session(
session_id="bs-stale", user_id="alice", ttl_seconds=-3600
)
tokens_deleted = await storage.cleanup_expired_tokens()
sessions_deleted = await storage.cleanup_expired_sessions()
browser_deleted = await storage.cleanup_expired_browser_sessions()
assert tokens_deleted == 1, f"expected 1 expired token, got {tokens_deleted}"
assert sessions_deleted == 1, (
f"expected 1 expired oauth session, got {sessions_deleted}"
)
assert browser_deleted == 1, (
f"expected 1 expired browser session, got {browser_deleted}"
)
# Fresh rows survived.
assert await storage.get_refresh_token("fresh-user") is not None
assert await storage.get_refresh_token("expired-user") is None
assert await storage.get_oauth_session("sess-fresh") is not None
assert await storage.get_oauth_session("sess-stale") is None