Round-2 fixes after the bot review on PR #798 plus two user follow-ups (self-signed Postgres support; asyncpg should be a PyPI extra). Folded into the same PR rather than a follow-up since the work is still unmerged. Security -------- - Mask database credentials in all 5 log call sites (storage.py × 4, migrations.py × 1) via a new `mask_db_password()` helper in config.py. Uses SQLAlchemy's `make_url(...).render_as_string(hide_password=True)` with a regex fallback so the masking path never raises. - New `tests/unit/test_storage_logging.py` asserts a sentinel password never appears in `caplog` during `RefreshTokenStorage.initialize()`. Distribution ------------ - `asyncpg` moved to `[project.optional-dependencies] postgres` so a vanilla `pip install nextcloud-mcp-server` no longer pulls in the ~5 MB C extension. The Docker image runs `uv sync --extra postgres`, so containerized deployments are unchanged. - When `DATABASE_URL=postgresql+asyncpg://...` is set on a venv missing the extra, `RefreshTokenStorage.initialize()` raises a friendly RuntimeError pointing at `[postgres]` rather than the generic ModuleNotFoundError. TLS for the Postgres backend ---------------------------- - New `DATABASE_VERIFY_SSL` + `DATABASE_CA_BUNDLE` env vars mirror the existing `NEXTCLOUD_VERIFY_SSL` / `NEXTCLOUD_CA_BUNDLE` pattern (validators in Settings.__post_init__, `get_database_ssl()` helper alongside `get_nextcloud_ssl_verify()`). `DATABASE_VERIFY_SSL=false` wins over `DATABASE_CA_BUNDLE` for incident-response convenience. - Default is **None** rather than True — keeps PR #798's behavior intact for cluster-internal Postgres that runs without TLS. Operators opt into verify-full or supply a private CA. ADR-026 records the reasoning vs the Nextcloud HTTPS default. - Engine factory in `storage.py` passes `ssl` via `connect_args` only when `get_database_ssl()` returns non-None; otherwise asyncpg's default (`prefer`) applies. - Storage logs which TLS mode is active at INFO (no secret material). Configurable connection pool ---------------------------- - `DATABASE_POOL_SIZE` (default 10) and `DATABASE_MAX_OVERFLOW` (default 20) replace the hardcoded engine values. With many replicas this can blow past managed-Postgres `max_connections=100`; tune down for large fleets. - gte-1 / gte-0 validators in __post_init__ reject 0/negative pool sizes at startup with the offending value in the error. Consistency polish ------------------ - Migration 006: convert raw `op.execute("ALTER TABLE ... ADD COLUMN")` to `op.batch_alter_table(...).add_column(sa.Column("nonce", sa.Text))` for stylistic consistency with the rewritten 001-005. Downgrade now drops the column instead of being a no-op. - `registered_webhooks.created_at` standardized from `sa.Float` to `sa.BigInteger` (all other `*_at` columns); `store_webhook()` casts `time.time()` → `int`. - `is_sqlite_url()` made case-insensitive. Testing ------- - New `tests/integration/test_storage_postgres.py::test_cleanup_expired_roundtrip` exercises `cleanup_expired_tokens`, `cleanup_expired_sessions`, and `cleanup_expired_browser_sessions` — relies on DELETE rowcount, historically dialect-tricky. - `tests/unit/test_ssl_config.py` extended with `TestDatabaseSSLSettings` + `TestGetDatabaseSSL` classes (9 new tests) mirroring the existing Nextcloud SSL tests one-for-one. Docs ---- - `docs/configuration.md` Centralized-Storage section grew the four new env vars + a homelab example with a private CA. - `docs/ADR-026` grew Distribution, TLS, and `alembic/env.py` async-pattern subsections explaining the non-obvious design choices. Helm chart counterpart in cbcoutinho/helm-charts PR #34 (separate commit on `feat/nextcloud-mcp-server-database-url`). Verification ------------ - `uv run pytest tests/unit/` — 1025 passed. - `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 6 passed (including new cleanup test). - `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean. Tracked on Astrolabe Cloud POC board, card #99. --- _This PR was generated with the help of AI, and reviewed by a Human_ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
221 lines
7.9 KiB
Python
221 lines
7.9 KiB
Python
"""End-to-end Postgres backend smoke for RefreshTokenStorage (ADR-026).
|
|
|
|
Exercises every storage method touched by the SQLAlchemy / asyncpg port
|
|
against a fresh Postgres schema. The test is opt-in: it requires the
|
|
``postgres-test`` docker-compose service to be running and
|
|
``TEST_DATABASE_URL`` to be exported.
|
|
|
|
Bring up the dependency once::
|
|
|
|
docker compose --profile postgres up -d postgres-test
|
|
export TEST_DATABASE_URL=postgresql+asyncpg://mcp:mcp@localhost:5433/mcp
|
|
|
|
Then run::
|
|
|
|
uv run pytest tests/integration/test_storage_postgres.py -v -m postgres
|
|
|
|
When ``TEST_DATABASE_URL`` is unset (or the service is unreachable) the
|
|
test is skipped so the full suite still passes locally without Docker.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import socket
|
|
from urllib.parse import urlparse
|
|
|
|
import pytest
|
|
from cryptography.fernet import Fernet
|
|
|
|
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
|
|
|
|
pytestmark = [pytest.mark.integration, pytest.mark.postgres]
|
|
|
|
|
|
def _postgres_url() -> str | None:
|
|
return os.environ.get("TEST_DATABASE_URL") or None
|
|
|
|
|
|
def _reachable(url: str) -> bool:
|
|
parsed = urlparse(url)
|
|
try:
|
|
with socket.create_connection(
|
|
(parsed.hostname or "localhost", parsed.port or 5432), timeout=1.0
|
|
):
|
|
return True
|
|
except OSError:
|
|
return False
|
|
|
|
|
|
@pytest.fixture
|
|
def postgres_url() -> str:
|
|
url = _postgres_url()
|
|
if not url:
|
|
pytest.skip(
|
|
"TEST_DATABASE_URL not set — run "
|
|
"`docker compose --profile postgres up -d postgres-test` and export "
|
|
"TEST_DATABASE_URL=postgresql+asyncpg://mcp:mcp@localhost:5433/mcp"
|
|
)
|
|
if not _reachable(url):
|
|
pytest.skip(f"Postgres at {url} is not reachable")
|
|
return url
|
|
|
|
|
|
@pytest.fixture
|
|
async def reset_schema(postgres_url: str):
|
|
"""Drop+recreate the public schema before and after each test."""
|
|
from sqlalchemy import text
|
|
from sqlalchemy.ext.asyncio import create_async_engine
|
|
|
|
async def _reset() -> None:
|
|
engine = create_async_engine(postgres_url, future=True)
|
|
try:
|
|
async with engine.begin() as conn:
|
|
await conn.execute(text("DROP SCHEMA public CASCADE"))
|
|
await conn.execute(text("CREATE SCHEMA public"))
|
|
finally:
|
|
await engine.dispose()
|
|
|
|
await _reset()
|
|
yield
|
|
await _reset()
|
|
|
|
|
|
@pytest.fixture
|
|
async def storage(postgres_url: str, reset_schema):
|
|
key = Fernet.generate_key()
|
|
s = RefreshTokenStorage(database_url=postgres_url, encryption_key=key)
|
|
await s.initialize()
|
|
yield s
|
|
|
|
|
|
async def test_refresh_token_roundtrip(storage: RefreshTokenStorage):
|
|
"""Store + retrieve + upsert + delete a refresh token end-to-end."""
|
|
await storage.store_refresh_token(
|
|
user_id="alice", refresh_token="rt-1", expires_at=9_999_999_999
|
|
)
|
|
tok = await storage.get_refresh_token("alice")
|
|
assert tok is not None
|
|
assert tok["refresh_token"] == "rt-1"
|
|
assert tok["expires_at"] == 9_999_999_999
|
|
|
|
# Upsert preserves user_id, swaps token contents.
|
|
await storage.store_refresh_token(
|
|
user_id="alice", refresh_token="rt-2", expires_at=9_999_999_999
|
|
)
|
|
tok = await storage.get_refresh_token("alice")
|
|
assert tok is not None and tok["refresh_token"] == "rt-2"
|
|
|
|
assert await storage.delete_refresh_token("alice") is True
|
|
assert await storage.get_refresh_token("alice") is None
|
|
|
|
|
|
async def test_app_password_roundtrip(storage: RefreshTokenStorage):
|
|
"""Store + retrieve + replace + delete a scoped app password."""
|
|
await storage.store_app_password(user_id="bob", app_password="pw-1")
|
|
assert await storage.get_app_password("bob") == "pw-1"
|
|
|
|
# Replace path exercises the ON CONFLICT DO UPDATE on the singleton row.
|
|
await storage.store_app_password(user_id="bob", app_password="pw-2")
|
|
assert await storage.get_app_password("bob") == "pw-2"
|
|
|
|
assert await storage.delete_app_password("bob") is True
|
|
assert await storage.get_app_password("bob") is None
|
|
|
|
|
|
async def test_oauth_session_lifecycle(storage: RefreshTokenStorage):
|
|
"""Cover the ADR-004 progressive-consent session table."""
|
|
await storage.store_oauth_session(
|
|
session_id="sess-1",
|
|
client_redirect_uri="http://localhost:12345/callback",
|
|
mcp_authorization_code="mcp-code-abc",
|
|
flow_type="hybrid",
|
|
ttl_seconds=600,
|
|
)
|
|
fetched = await storage.get_oauth_session("sess-1")
|
|
assert fetched is not None
|
|
assert fetched["mcp_authorization_code"] == "mcp-code-abc"
|
|
|
|
by_code = await storage.get_oauth_session_by_mcp_code("mcp-code-abc")
|
|
assert by_code is not None and by_code["session_id"] == "sess-1"
|
|
|
|
|
|
async def test_webhook_tracking(storage: RefreshTokenStorage):
|
|
"""Tracks webhook ↔ preset mappings via ON CONFLICT upserts."""
|
|
await storage.store_webhook(webhook_id=101, preset_id="notes_sync")
|
|
await storage.store_webhook(webhook_id=202, preset_id="notes_sync")
|
|
await storage.store_webhook(webhook_id=303, preset_id="calendar_sync")
|
|
|
|
assert sorted(await storage.get_webhooks_by_preset("notes_sync")) == [101, 202]
|
|
assert await storage.get_webhooks_by_preset("calendar_sync") == [303]
|
|
|
|
# Re-storing the same webhook_id is a no-op upsert.
|
|
await storage.store_webhook(webhook_id=101, preset_id="notes_sync")
|
|
assert sorted(await storage.get_webhooks_by_preset("notes_sync")) == [101, 202]
|
|
|
|
assert await storage.delete_webhook(webhook_id=101) is True
|
|
assert await storage.get_webhooks_by_preset("notes_sync") == [202]
|
|
|
|
|
|
async def test_audit_log_capture(storage: RefreshTokenStorage):
|
|
"""Audit events from upstream methods land in audit_logs."""
|
|
await storage.store_app_password(user_id="carol", app_password="x")
|
|
logs = await storage.get_audit_logs(user_id="carol", limit=10)
|
|
assert any(entry["event"] == "store_app_password" for entry in logs)
|
|
|
|
|
|
async def test_cleanup_expired_roundtrip(storage: RefreshTokenStorage):
|
|
"""``cleanup_expired_*`` paths rely on DELETE rowcount across dialects.
|
|
|
|
Regression guard for the bot review on PR #798 — the original
|
|
integration tests didn't exercise these methods, which historically
|
|
have been a source of dialect-portability bugs.
|
|
"""
|
|
# Insert one fresh + one expired refresh token.
|
|
await storage.store_refresh_token(
|
|
user_id="fresh-user", refresh_token="fresh", expires_at=9_999_999_999
|
|
)
|
|
await storage.store_refresh_token(
|
|
user_id="expired-user", refresh_token="stale", expires_at=1
|
|
)
|
|
|
|
# Insert one fresh + one expired OAuth session.
|
|
await storage.store_oauth_session(
|
|
session_id="sess-fresh",
|
|
client_redirect_uri="http://localhost/cb",
|
|
mcp_authorization_code="code-fresh",
|
|
ttl_seconds=600,
|
|
)
|
|
await storage.store_oauth_session(
|
|
session_id="sess-stale",
|
|
client_redirect_uri="http://localhost/cb",
|
|
mcp_authorization_code="code-stale",
|
|
ttl_seconds=-3600, # expires_at = now - 1h
|
|
)
|
|
|
|
# Insert one fresh + one expired browser session.
|
|
await storage.create_browser_session(
|
|
session_id="bs-fresh", user_id="alice", ttl_seconds=600
|
|
)
|
|
await storage.create_browser_session(
|
|
session_id="bs-stale", user_id="alice", ttl_seconds=-3600
|
|
)
|
|
|
|
tokens_deleted = await storage.cleanup_expired_tokens()
|
|
sessions_deleted = await storage.cleanup_expired_sessions()
|
|
browser_deleted = await storage.cleanup_expired_browser_sessions()
|
|
|
|
assert tokens_deleted == 1, f"expected 1 expired token, got {tokens_deleted}"
|
|
assert sessions_deleted == 1, (
|
|
f"expected 1 expired oauth session, got {sessions_deleted}"
|
|
)
|
|
assert browser_deleted == 1, (
|
|
f"expected 1 expired browser session, got {browser_deleted}"
|
|
)
|
|
|
|
# Fresh rows survived.
|
|
assert await storage.get_refresh_token("fresh-user") is not None
|
|
assert await storage.get_refresh_token("expired-user") is None
|
|
assert await storage.get_oauth_session("sess-fresh") is not None
|
|
assert await storage.get_oauth_session("sess-stale") is None
|