Files
mcp-nextcloud/nextcloud_mcp_server/migrations.py
T
Chris CoutinhoandClaude Opus 4.7 f2b7bf132f fix(storage): address PR #798 review feedback (credentials, asyncpg extra, TLS, pool)
Round-2 fixes after the bot review on PR #798 plus two user follow-ups
(self-signed Postgres support; asyncpg should be a PyPI extra). Folded
into the same PR rather than a follow-up since the work is still
unmerged.

Security
--------
- Mask database credentials in all 5 log call sites (storage.py × 4,
  migrations.py × 1) via a new `mask_db_password()` helper in config.py.
  Uses SQLAlchemy's `make_url(...).render_as_string(hide_password=True)`
  with a regex fallback so the masking path never raises.
- New `tests/unit/test_storage_logging.py` asserts a sentinel password
  never appears in `caplog` during `RefreshTokenStorage.initialize()`.

Distribution
------------
- `asyncpg` moved to `[project.optional-dependencies] postgres` so a
  vanilla `pip install nextcloud-mcp-server` no longer pulls in the
  ~5 MB C extension. The Docker image runs `uv sync --extra postgres`,
  so containerized deployments are unchanged.
- When `DATABASE_URL=postgresql+asyncpg://...` is set on a venv missing
  the extra, `RefreshTokenStorage.initialize()` raises a friendly
  RuntimeError pointing at `[postgres]` rather than the generic
  ModuleNotFoundError.

TLS for the Postgres backend
----------------------------
- New `DATABASE_VERIFY_SSL` + `DATABASE_CA_BUNDLE` env vars mirror the
  existing `NEXTCLOUD_VERIFY_SSL` / `NEXTCLOUD_CA_BUNDLE` pattern
  (validators in Settings.__post_init__, `get_database_ssl()` helper
  alongside `get_nextcloud_ssl_verify()`). `DATABASE_VERIFY_SSL=false`
  wins over `DATABASE_CA_BUNDLE` for incident-response convenience.
- Default is **None** rather than True — keeps PR #798's behavior
  intact for cluster-internal Postgres that runs without TLS. Operators
  opt into verify-full or supply a private CA. ADR-026 records the
  reasoning vs the Nextcloud HTTPS default.
- Engine factory in `storage.py` passes `ssl` via `connect_args` only
  when `get_database_ssl()` returns non-None; otherwise asyncpg's
  default (`prefer`) applies.
- Storage logs which TLS mode is active at INFO (no secret material).

Configurable connection pool
----------------------------
- `DATABASE_POOL_SIZE` (default 10) and `DATABASE_MAX_OVERFLOW`
  (default 20) replace the hardcoded engine values. With many replicas
  this can blow past managed-Postgres `max_connections=100`; tune down
  for large fleets.
- gte-1 / gte-0 validators in __post_init__ reject 0/negative pool
  sizes at startup with the offending value in the error.

Consistency polish
------------------
- Migration 006: convert raw `op.execute("ALTER TABLE ... ADD COLUMN")`
  to `op.batch_alter_table(...).add_column(sa.Column("nonce", sa.Text))`
  for stylistic consistency with the rewritten 001-005. Downgrade now
  drops the column instead of being a no-op.
- `registered_webhooks.created_at` standardized from `sa.Float` to
  `sa.BigInteger` (all other `*_at` columns); `store_webhook()` casts
  `time.time()` → `int`.
- `is_sqlite_url()` made case-insensitive.

Testing
-------
- New `tests/integration/test_storage_postgres.py::test_cleanup_expired_roundtrip`
  exercises `cleanup_expired_tokens`, `cleanup_expired_sessions`, and
  `cleanup_expired_browser_sessions` — relies on DELETE rowcount,
  historically dialect-tricky.
- `tests/unit/test_ssl_config.py` extended with `TestDatabaseSSLSettings`
  + `TestGetDatabaseSSL` classes (9 new tests) mirroring the existing
  Nextcloud SSL tests one-for-one.

Docs
----
- `docs/configuration.md` Centralized-Storage section grew the four new
  env vars + a homelab example with a private CA.
- `docs/ADR-026` grew Distribution, TLS, and `alembic/env.py` async-pattern
  subsections explaining the non-obvious design choices.

Helm chart counterpart in cbcoutinho/helm-charts PR #34 (separate
commit on `feat/nextcloud-mcp-server-database-url`).

Verification
------------
- `uv run pytest tests/unit/` — 1025 passed.
- `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 6 passed (including new cleanup test).
- `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean.

Tracked on Astrolabe Cloud POC board, card #99.

---

_This PR was generated with the help of AI, and reviewed by a Human_

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 18:53:45 +02:00

180 lines
6.3 KiB
Python

"""Database migration utilities for nextcloud-mcp-server.
This module provides helper functions for managing Alembic database migrations
programmatically. It enables automatic migration on application startup and
provides CLI integration.
All helpers accept a SQLAlchemy URL (``sqlite+aiosqlite:///...`` or
``postgresql+asyncpg://...``). When called without an explicit URL they fall
back to :func:`nextcloud_mcp_server.config.get_database_url`.
"""
import logging
from pathlib import Path
from alembic.config import Config
from sqlalchemy import create_engine, inspect, text
import nextcloud_mcp_server.alembic as alembic_package
from alembic import command
from nextcloud_mcp_server.config import get_database_url, mask_db_password
logger = logging.getLogger(__name__)
def _coerce_url(database_url: str | Path | None) -> str:
"""Accept either a URL string, a Path (legacy SQLite path), or None.
A bare ``Path`` is interpreted as a SQLite database file for backward
compatibility with the prior path-based API.
"""
if database_url is None:
return get_database_url()
if isinstance(database_url, Path):
return f"sqlite+aiosqlite:///{database_url.resolve()}"
return database_url
def _to_sync_url(database_url: str) -> str:
"""Map an async driver URL to its sync equivalent for blocking inspection.
SQLAlchemy's :func:`inspect` and :func:`create_engine` used below are
synchronous APIs. The runtime uses async drivers (``aiosqlite``,
``asyncpg``) but Alembic and these utility queries don't need them.
"""
return database_url.replace("+aiosqlite", "").replace("+asyncpg", "")
def get_alembic_config(database_url: str | Path | None = None) -> Config:
"""
Get Alembic configuration for programmatic use.
Works in both development and installed (Docker) modes by using
package location instead of alembic.ini file.
Args:
database_url: SQLAlchemy URL. If None, resolves via
:func:`get_database_url` (DATABASE_URL env var, falling back
to the ephemeral SQLite tempfile under ``TOKEN_STORAGE_DB``).
For backward compatibility a ``Path`` is treated as a SQLite
file path.
Returns:
Alembic Config object configured for the resolved URL.
"""
if alembic_package.__file__ is None:
raise RuntimeError("alembic package __file__ is None")
script_location = Path(alembic_package.__file__).parent
config = Config()
config.set_main_option("script_location", str(script_location))
config.set_main_option("path_separator", "os")
url = _coerce_url(database_url)
config.set_main_option("sqlalchemy.url", url)
logger.debug("Alembic script location: %s", script_location)
logger.debug("Database URL: %s", mask_db_password(url))
return config
def upgrade_database(
database_url: str | Path | None = None, revision: str = "head"
) -> None:
"""Upgrade database to a specific revision (default: latest)."""
config = get_alembic_config(database_url)
logger.info("Upgrading database to revision: %s", revision)
command.upgrade(config, revision)
logger.info("Database upgrade completed successfully")
def downgrade_database(
database_url: str | Path | None = None, revision: str = "-1"
) -> None:
"""Downgrade database to a specific revision (default: previous)."""
config = get_alembic_config(database_url)
logger.warning("Downgrading database to revision: %s", revision)
command.downgrade(config, revision)
logger.info("Database downgrade completed successfully")
def get_current_revision(database_url: str | Path | None = None) -> str | None:
"""
Get the current database revision by reading the ``alembic_version`` table.
Returns ``None`` when the database does not exist or has no
``alembic_version`` table (i.e. has never been migrated).
"""
url = _to_sync_url(_coerce_url(database_url))
if url.startswith("sqlite:///"):
path = url[len("sqlite:///") :]
if path and not Path(path).exists():
logger.debug("Database does not exist: %s", path)
return None
try:
engine = create_engine(url, future=True)
try:
inspector = inspect(engine)
if not inspector.has_table("alembic_version"):
return None
with engine.connect() as conn:
row = conn.execute(
text("SELECT version_num FROM alembic_version")
).fetchone()
return row[0] if row else None
finally:
engine.dispose()
except Exception as e:
logger.error("Failed to get current revision: %s", e)
return None
def stamp_database(
database_url: str | Path | None = None, revision: str = "head"
) -> None:
"""
Stamp database with a specific revision without running migrations.
Useful for marking pre-Alembic databases as already at a known revision.
"""
config = get_alembic_config(database_url)
logger.info("Stamping database with revision: %s", revision)
command.stamp(config, revision)
logger.info("Database stamped successfully")
def show_migration_history(database_url: str | Path | None = None) -> None:
"""Display migration history."""
config = get_alembic_config(database_url)
command.history(config, verbose=True)
def create_migration(message: str, autogenerate: bool = False) -> None:
"""
Create a new migration script.
Args:
message: Description of the migration
autogenerate: Whether to attempt auto-generation (requires SQLAlchemy models)
Note:
Since we don't use SQLAlchemy models, autogenerate will be disabled
and migrations must be written manually using portable Alembic
operations (``op.create_table``, ``op.add_column`` …) rather than
raw SQL so they work on both SQLite and Postgres.
"""
config = get_alembic_config()
logger.info("Creating new migration: %s", message)
if autogenerate:
logger.warning(
"Auto-generation is not supported (no SQLAlchemy models). "
"Migration will be created with empty upgrade/downgrade functions."
)
command.revision(config, message=message, autogenerate=False)
logger.info("Migration created successfully. Edit the file to add SQL statements.")