fix(storage): address PR #798 review feedback (credentials, asyncpg extra, TLS, pool)

Round-2 fixes after the bot review on PR #798 plus two user follow-ups (self-signed Postgres support; asyncpg should be a PyPI extra). Folded into the same PR rather than a follow-up since the work is still unmerged. Security -------- - Mask database credentials in all 5 log call sites (storage.py × 4, migrations.py × 1) via a new `mask_db_password()` helper in config.py. Uses SQLAlchemy's `make_url(...).render_as_string(hide_password=True)` with a regex fallback so the masking path never raises. - New `tests/unit/test_storage_logging.py` asserts a sentinel password never appears in `caplog` during `RefreshTokenStorage.initialize()`. Distribution ------------ - `asyncpg` moved to `[project.optional-dependencies] postgres` so a vanilla `pip install nextcloud-mcp-server` no longer pulls in the ~5 MB C extension. The Docker image runs `uv sync --extra postgres`, so containerized deployments are unchanged. - When `DATABASE_URL=postgresql+asyncpg://...` is set on a venv missing the extra, `RefreshTokenStorage.initialize()` raises a friendly RuntimeError pointing at `[postgres]` rather than the generic ModuleNotFoundError. TLS for the Postgres backend ---------------------------- - New `DATABASE_VERIFY_SSL` + `DATABASE_CA_BUNDLE` env vars mirror the existing `NEXTCLOUD_VERIFY_SSL` / `NEXTCLOUD_CA_BUNDLE` pattern (validators in Settings.__post_init__, `get_database_ssl()` helper alongside `get_nextcloud_ssl_verify()`). `DATABASE_VERIFY_SSL=false` wins over `DATABASE_CA_BUNDLE` for incident-response convenience. - Default is **None** rather than True — keeps PR #798's behavior intact for cluster-internal Postgres that runs without TLS. Operators opt into verify-full or supply a private CA. ADR-026 records the reasoning vs the Nextcloud HTTPS default. - Engine factory in `storage.py` passes `ssl` via `connect_args` only when `get_database_ssl()` returns non-None; otherwise asyncpg's default (`prefer`) applies. - Storage logs which TLS mode is active at INFO (no secret material). Configurable connection pool ---------------------------- - `DATABASE_POOL_SIZE` (default 10) and `DATABASE_MAX_OVERFLOW` (default 20) replace the hardcoded engine values. With many replicas this can blow past managed-Postgres `max_connections=100`; tune down for large fleets. - gte-1 / gte-0 validators in __post_init__ reject 0/negative pool sizes at startup with the offending value in the error. Consistency polish ------------------ - Migration 006: convert raw `op.execute("ALTER TABLE ... ADD COLUMN")` to `op.batch_alter_table(...).add_column(sa.Column("nonce", sa.Text))` for stylistic consistency with the rewritten 001-005. Downgrade now drops the column instead of being a no-op. - `registered_webhooks.created_at` standardized from `sa.Float` to `sa.BigInteger` (all other `*_at` columns); `store_webhook()` casts `time.time()` → `int`. - `is_sqlite_url()` made case-insensitive. Testing ------- - New `tests/integration/test_storage_postgres.py::test_cleanup_expired_roundtrip` exercises `cleanup_expired_tokens`, `cleanup_expired_sessions`, and `cleanup_expired_browser_sessions` — relies on DELETE rowcount, historically dialect-tricky. - `tests/unit/test_ssl_config.py` extended with `TestDatabaseSSLSettings` + `TestGetDatabaseSSL` classes (9 new tests) mirroring the existing Nextcloud SSL tests one-for-one. Docs ---- - `docs/configuration.md` Centralized-Storage section grew the four new env vars + a homelab example with a private CA. - `docs/ADR-026` grew Distribution, TLS, and `alembic/env.py` async-pattern subsections explaining the non-obvious design choices. Helm chart counterpart in cbcoutinho/helm-charts PR #34 (separate commit on `feat/nextcloud-mcp-server-database-url`). Verification ------------ - `uv run pytest tests/unit/` — 1025 passed. - `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 6 passed (including new cleanup test). - `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean. Tracked on Astrolabe Cloud POC board, card #99. --- _This PR was generated with the help of AI, and reviewed by a Human_ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 18:53:45 +02:00
parent 292cbb3292
commit f2b7bf132f
14 changed files with 510 additions and 22 deletions
@@ -125,7 +125,10 @@ def upgrade() -> None:
        sa.Column("id", sa.Integer, primary_key=True, autoincrement=True),
        sa.Column("webhook_id", sa.Integer, nullable=False, unique=True),
        sa.Column("preset_id", sa.Text, nullable=False),
-        sa.Column("created_at", sa.Float, nullable=False),
+        # BigInteger for consistency with every other *_at column (PR #798
+        # review): subsecond precision wasn't load-bearing for webhook
+        # bookkeeping. ``store_webhook()`` casts to ``int(time.time())``.
+        sa.Column("created_at", sa.BigInteger, nullable=False),
    )
    op.create_index("idx_webhooks_preset", "registered_webhooks", ["preset_id"])
    op.create_index("idx_webhooks_created", "registered_webhooks", ["created_at"])
@@ -12,6 +12,8 @@ Revises: 005
 Create Date: 2026-05-02 16:00:00.000000
 """

+import sqlalchemy as sa
+
 from alembic import op

 revision = "006"
@@ -21,9 +23,14 @@ depends_on = None


 def upgrade() -> None:
-    op.execute("ALTER TABLE oauth_sessions ADD COLUMN nonce TEXT")
+    # ``batch_alter_table`` emits a native ``ALTER TABLE ... ADD COLUMN``
+    # on Postgres and works around SQLite's pre-3.35 limitations by
+    # recreating the table when needed. Matches the portable-DDL style of
+    # the rewritten migrations 001-005 (PR #798 review nit).
+    with op.batch_alter_table("oauth_sessions") as batch_op:
+        batch_op.add_column(sa.Column("nonce", sa.Text))


 def downgrade() -> None:
-    # SQLite < 3.35 cannot DROP COLUMN; leave the column on downgrade.
-    pass
+    with op.batch_alter_table("oauth_sessions") as batch_op:
+        batch_op.drop_column("nonce")
@@ -25,6 +25,7 @@ Token storage requires TOKEN_ENCRYPTION_KEY, but webhook tracking does not.
 Sensitive data (tokens, secrets) is encrypted at rest using Fernet symmetric encryption.
 """

+import importlib.util
 import json
 import logging
 import os
@@ -44,9 +45,12 @@ from sqlalchemy.ext.asyncio import AsyncConnection, AsyncEngine, create_async_en
 from sqlalchemy.pool import NullPool

 from nextcloud_mcp_server.config import (
+    get_database_ssl,
    get_database_url,
+    get_settings,
    is_ephemeral_token_db,
    is_sqlite_url,
+    mask_db_password,
 )
 from nextcloud_mcp_server.migrations import stamp_database, upgrade_database
 from nextcloud_mcp_server.observability.metrics import record_db_operation
@@ -345,7 +349,9 @@ class RefreshTokenStorage:
                    sqlite_path,
                )
        else:
-            logger.info("Using centralized token storage at %s", database_url)
+            logger.info(
+                "Using centralized token storage at %s", mask_db_password(database_url)
+            )
        encryption_key_b64 = os.getenv("TOKEN_ENCRYPTION_KEY")

        encryption_key = None
@@ -423,11 +429,43 @@ class RefreshTokenStorage:
                future=True,
            )
        else:
+            # Postgres ships as an optional PyPI extra (`[postgres]`) so the
+            # default `pip install nextcloud-mcp-server` audience doesn't
+            # pull in asyncpg's C extension. The Docker image bundles it.
+            # Surface a clear actionable error when the driver is missing
+            # rather than the generic ModuleNotFoundError SQLAlchemy emits.
+            if "+asyncpg" in self.database_url.lower() and (
+                importlib.util.find_spec("asyncpg") is None
+            ):
+                raise RuntimeError(
+                    "DATABASE_URL points at Postgres via asyncpg but the "
+                    "'asyncpg' driver is not installed. Install with "
+                    "`pip install nextcloud-mcp-server[postgres]` or use "
+                    "the Docker image, which bundles it. See ADR-026."
+                )
+            # Conditionally pass TLS config through to asyncpg. When
+            # get_database_ssl() returns None we omit ``ssl`` entirely so
+            # asyncpg's default (``prefer``) applies — keeps cluster-local
+            # Postgres without TLS working out of the box. See ADR-026.
+            connect_args: dict[str, object] = {}
+            ssl_arg = get_database_ssl()
+            if ssl_arg is not None:
+                connect_args["ssl"] = ssl_arg
+                logger.info(
+                    "Postgres backend TLS: %s",
+                    "disabled"
+                    if ssl_arg is False
+                    else "custom CA bundle"
+                    if not isinstance(ssl_arg, bool)
+                    else "verify-full (system CAs)",
+                )
+            settings = get_settings()
            self.engine = create_async_engine(
                self.database_url,
-                pool_size=10,
-                max_overflow=20,
+                pool_size=settings.database_pool_size,
+                max_overflow=settings.database_max_overflow,
                pool_pre_ping=True,
+                connect_args=connect_args,
                future=True,
            )
        self._dialect = self.engine.dialect.name
@@ -446,7 +484,7 @@ class RefreshTokenStorage:
            if has_schema:
                logger.info(
                    "Detected pre-Alembic database at %s, stamping with initial revision",
-                    self.database_url,
+                    mask_db_password(self.database_url),
                )
                await to_thread.run_sync(stamp_database, self.database_url, "001")
                logger.info(
@@ -456,7 +494,7 @@ class RefreshTokenStorage:
            else:
                logger.info(
                    "Initializing new database at %s with migrations",
-                    self.database_url,
+                    mask_db_password(self.database_url),
                )
                await to_thread.run_sync(upgrade_database, self.database_url, "head")
                logger.info("Database initialized with migrations")
@@ -468,7 +506,10 @@ class RefreshTokenStorage:
            os.chmod(self.db_path, 0o600)

        self._initialized = True
-        logger.info("Initialized refresh token storage at %s", self.database_url)
+        logger.info(
+            "Initialized refresh token storage at %s",
+            mask_db_password(self.database_url),
+        )

    @asynccontextmanager
    async def _db(self):
@@ -1638,7 +1679,7 @@ class RefreshTokenStorage:
                    preset_id = EXCLUDED.preset_id,
                    created_at = EXCLUDED.created_at
                """,
-                (webhook_id, preset_id, time.time()),
+                (webhook_id, preset_id, int(time.time())),
            )
            await db.commit()

@@ -60,6 +60,19 @@ _DEFAULTS: dict[str, Any] = {
    # when set. Use postgresql+asyncpg://user:pw@host/db for HA k8s
    # deployments so pods can be stateless. See ADR-026.
    "database_url": None,
+    # TLS for the Postgres backend (mirror NEXTCLOUD_VERIFY_SSL pattern).
+    # Default is None — preserve asyncpg's `prefer` mode so cluster-local
+    # Postgres without TLS works out of the box. Set to True for full
+    # verification or False to silence cert errors against self-signed
+    # homelab servers. DATABASE_CA_BUNDLE points at a private-CA PEM.
+    "database_verify_ssl": None,
+    "database_ca_bundle": None,
+    # Postgres connection pool sizing (ADR-026, reviewer feedback on #798).
+    # Per-pod pool defaults to 10 + 20 overflow = 30 max connections.
+    # With many replicas this can blow past managed-Postgres
+    # `max_connections=100`; tune down via env when needed.
+    "database_pool_size": 10,
+    "database_max_overflow": 20,
    # Webhook delivery authentication (ADR-010): when set, registrations
    # tell NC to add `Authorization: Bearer <secret>` to webhook deliveries
    # and the receiver rejects unauthenticated requests.
@@ -304,7 +317,30 @@ def is_sqlite_url(url: str) -> bool:
    """Return True for SQLite SQLAlchemy URLs (used to gate sqlite-only logic
    like file-permission hardening and ``sqlite_master`` legacy lookups).
    """
-    return url.startswith("sqlite")
+    return url.lower().startswith("sqlite")
+
+
+def mask_db_password(url: str) -> str:
+    """Return a logger-safe rendering of a SQLAlchemy URL.
+
+    DATABASE_URL routinely carries a password (e.g.
+    ``postgresql+asyncpg://mcp:secret@db/mcp``); logging it raw leaks the
+    secret to stdout/stderr and any aggregator. SQLAlchemy's
+    :func:`make_url` + ``render_as_string(hide_password=True)`` substitutes
+    a fixed ``***`` placeholder while keeping the rest of the URL intact
+    so operators can still see which host / driver they're hitting.
+    """
+    try:
+        from sqlalchemy.engine.url import make_url  # noqa: PLC0415
+
+        return make_url(url).render_as_string(hide_password=True)
+    except Exception:
+        # If parsing fails (e.g. an explicit ssl-disable test URL with an
+        # exotic shape), fall back to a regex that scrubs any
+        # ``://user:password@`` pattern. Never raise from a logging path.
+        import re  # noqa: PLC0415
+
+        return re.sub(r"(://[^:/]+):[^@]*@", r"\1:***@", url)


 LOGGING_CONFIG = {
@@ -470,6 +506,19 @@ class Settings:
    nextcloud_verify_ssl: bool = True
    nextcloud_ca_bundle: str | None = None

+    # Postgres backend TLS settings (ADR-026). Default verify_ssl is None,
+    # not True: when DATABASE_URL is unset there's nothing to verify, and
+    # when it is set we don't want to break cluster-internal Postgres that
+    # commonly runs without TLS. Operators opt in to verify-full with True
+    # or supply a private-CA bundle.
+    database_verify_ssl: bool | None = None
+    database_ca_bundle: str | None = None
+    # Postgres connection pool sizing (ADR-026). The asyncpg engine maps
+    # these to its underlying QueuePool. Per-pod max = pool_size +
+    # max_overflow. Validate >= 1 in __post_init__.
+    database_pool_size: int = 10
+    database_max_overflow: int = 20
+
    # ADR-005: Token Audience Validation (required for OAuth mode)
    nextcloud_mcp_server_url: str | None = None  # MCP server URL (used as audience)
    nextcloud_resource_uri: str | None = None  # Nextcloud resource identifier
@@ -603,6 +652,35 @@ class Settings:
                )
            logger.info("Using custom CA bundle: %s", self.nextcloud_ca_bundle)

+        # Validate Postgres backend TLS configuration (ADR-026)
+        if self.database_verify_ssl is False:
+            logger.warning(
+                "DATABASE_VERIFY_SSL is disabled. "
+                "TLS certificate verification is turned off for the Postgres "
+                "backend. Only acceptable for homelab / self-signed setups; "
+                "prefer DATABASE_CA_BUNDLE for production."
+            )
+        if self.database_ca_bundle:
+            if not os.path.isfile(self.database_ca_bundle):
+                raise ValueError(
+                    f"DATABASE_CA_BUNDLE path does not exist: {self.database_ca_bundle}"
+                )
+            logger.info(
+                "Using custom CA bundle for Postgres backend: %s",
+                self.database_ca_bundle,
+            )
+
+        # Pool sizing must be sensible — guard against operators accidentally
+        # setting 0 / negative via env (would deadlock at first request).
+        if self.database_pool_size < 1:
+            raise ValueError(
+                f"DATABASE_POOL_SIZE must be >= 1; got {self.database_pool_size}"
+            )
+        if self.database_max_overflow < 0:
+            raise ValueError(
+                f"DATABASE_MAX_OVERFLOW must be >= 0; got {self.database_max_overflow}"
+            )
+
        # Ensure mutual exclusivity
        if self.qdrant_url and self.qdrant_location:
            raise ValueError(
@@ -958,6 +1036,12 @@ def get_settings() -> Settings:
        # Nextcloud SSL/TLS settings
        "nextcloud_verify_ssl": "NEXTCLOUD_VERIFY_SSL",
        "nextcloud_ca_bundle": "NEXTCLOUD_CA_BUNDLE",
+        # Postgres backend TLS (ADR-026)
+        "database_verify_ssl": "DATABASE_VERIFY_SSL",
+        "database_ca_bundle": "DATABASE_CA_BUNDLE",
+        # Postgres backend pool sizing (ADR-026)
+        "database_pool_size": "DATABASE_POOL_SIZE",
+        "database_max_overflow": "DATABASE_MAX_OVERFLOW",
        # ADR-005: Token Audience Validation
        "nextcloud_mcp_server_url": "NEXTCLOUD_MCP_SERVER_URL",
        "nextcloud_resource_uri": "NEXTCLOUD_RESOURCE_URI",
@@ -1056,3 +1140,33 @@ def get_nextcloud_ssl_verify() -> bool | ssl.SSLContext:
        ctx = ssl.create_default_context(cafile=settings.nextcloud_ca_bundle)
        return ctx
    return True
+
+
+def get_database_ssl() -> bool | ssl.SSLContext | None:
+    """Return the asyncpg ``ssl`` arg for the Postgres backend (ADR-026).
+
+    Returns:
+        - ``None`` when both DATABASE_VERIFY_SSL and DATABASE_CA_BUNDLE are
+          unset — caller skips passing ``ssl`` so asyncpg keeps its default
+          (``prefer``). Preserves PR #798 behavior for cluster-local
+          Postgres without TLS.
+        - ``False`` if DATABASE_VERIFY_SSL=false (silence cert errors).
+        - ``ssl.SSLContext`` if DATABASE_CA_BUNDLE is set (custom private
+          CA, implies verify-full).
+        - ``True`` if DATABASE_VERIFY_SSL=true and no bundle (verify-full
+          against system trust store).
+
+    DATABASE_VERIFY_SSL=false wins over DATABASE_CA_BUNDLE so an operator
+    can quickly silence cert errors during incident response without
+    having to delete the bundle path from their secret store. Matches the
+    Nextcloud-pattern precedence for symmetry with
+    :func:`get_nextcloud_ssl_verify`.
+    """
+    settings = get_settings()
+    if settings.database_verify_ssl is False:
+        return False
+    if settings.database_ca_bundle:
+        return ssl.create_default_context(cafile=settings.database_ca_bundle)
+    if settings.database_verify_ssl is True:
+        return True
+    return None
@@ -17,7 +17,7 @@ from sqlalchemy import create_engine, inspect, text

 import nextcloud_mcp_server.alembic as alembic_package
 from alembic import command
-from nextcloud_mcp_server.config import get_database_url
+from nextcloud_mcp_server.config import get_database_url, mask_db_password

 logger = logging.getLogger(__name__)

@@ -74,7 +74,7 @@ def get_alembic_config(database_url: str | Path | None = None) -> Config:
    config.set_main_option("sqlalchemy.url", url)

    logger.debug("Alembic script location: %s", script_location)
-    logger.debug("Database URL: %s", url)
+    logger.debug("Database URL: %s", mask_db_password(url))

    return config