fix(storage): address PR #798 review feedback (credentials, asyncpg extra, TLS, pool)
Round-2 fixes after the bot review on PR #798 plus two user follow-ups (self-signed Postgres support; asyncpg should be a PyPI extra). Folded into the same PR rather than a follow-up since the work is still unmerged. Security -------- - Mask database credentials in all 5 log call sites (storage.py × 4, migrations.py × 1) via a new `mask_db_password()` helper in config.py. Uses SQLAlchemy's `make_url(...).render_as_string(hide_password=True)` with a regex fallback so the masking path never raises. - New `tests/unit/test_storage_logging.py` asserts a sentinel password never appears in `caplog` during `RefreshTokenStorage.initialize()`. Distribution ------------ - `asyncpg` moved to `[project.optional-dependencies] postgres` so a vanilla `pip install nextcloud-mcp-server` no longer pulls in the ~5 MB C extension. The Docker image runs `uv sync --extra postgres`, so containerized deployments are unchanged. - When `DATABASE_URL=postgresql+asyncpg://...` is set on a venv missing the extra, `RefreshTokenStorage.initialize()` raises a friendly RuntimeError pointing at `[postgres]` rather than the generic ModuleNotFoundError. TLS for the Postgres backend ---------------------------- - New `DATABASE_VERIFY_SSL` + `DATABASE_CA_BUNDLE` env vars mirror the existing `NEXTCLOUD_VERIFY_SSL` / `NEXTCLOUD_CA_BUNDLE` pattern (validators in Settings.__post_init__, `get_database_ssl()` helper alongside `get_nextcloud_ssl_verify()`). `DATABASE_VERIFY_SSL=false` wins over `DATABASE_CA_BUNDLE` for incident-response convenience. - Default is **None** rather than True — keeps PR #798's behavior intact for cluster-internal Postgres that runs without TLS. Operators opt into verify-full or supply a private CA. ADR-026 records the reasoning vs the Nextcloud HTTPS default. - Engine factory in `storage.py` passes `ssl` via `connect_args` only when `get_database_ssl()` returns non-None; otherwise asyncpg's default (`prefer`) applies. - Storage logs which TLS mode is active at INFO (no secret material). Configurable connection pool ---------------------------- - `DATABASE_POOL_SIZE` (default 10) and `DATABASE_MAX_OVERFLOW` (default 20) replace the hardcoded engine values. With many replicas this can blow past managed-Postgres `max_connections=100`; tune down for large fleets. - gte-1 / gte-0 validators in __post_init__ reject 0/negative pool sizes at startup with the offending value in the error. Consistency polish ------------------ - Migration 006: convert raw `op.execute("ALTER TABLE ... ADD COLUMN")` to `op.batch_alter_table(...).add_column(sa.Column("nonce", sa.Text))` for stylistic consistency with the rewritten 001-005. Downgrade now drops the column instead of being a no-op. - `registered_webhooks.created_at` standardized from `sa.Float` to `sa.BigInteger` (all other `*_at` columns); `store_webhook()` casts `time.time()` → `int`. - `is_sqlite_url()` made case-insensitive. Testing ------- - New `tests/integration/test_storage_postgres.py::test_cleanup_expired_roundtrip` exercises `cleanup_expired_tokens`, `cleanup_expired_sessions`, and `cleanup_expired_browser_sessions` — relies on DELETE rowcount, historically dialect-tricky. - `tests/unit/test_ssl_config.py` extended with `TestDatabaseSSLSettings` + `TestGetDatabaseSSL` classes (9 new tests) mirroring the existing Nextcloud SSL tests one-for-one. Docs ---- - `docs/configuration.md` Centralized-Storage section grew the four new env vars + a homelab example with a private CA. - `docs/ADR-026` grew Distribution, TLS, and `alembic/env.py` async-pattern subsections explaining the non-obvious design choices. Helm chart counterpart in cbcoutinho/helm-charts PR #34 (separate commit on `feat/nextcloud-mcp-server-database-url`). Verification ------------ - `uv run pytest tests/unit/` — 1025 passed. - `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 6 passed (including new cleanup test). - `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean. Tracked on Astrolabe Cloud POC board, card #99. --- _This PR was generated with the help of AI, and reviewed by a Human_ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
292cbb3292
commit
f2b7bf132f
@@ -60,6 +60,19 @@ _DEFAULTS: dict[str, Any] = {
|
||||
# when set. Use postgresql+asyncpg://user:pw@host/db for HA k8s
|
||||
# deployments so pods can be stateless. See ADR-026.
|
||||
"database_url": None,
|
||||
# TLS for the Postgres backend (mirror NEXTCLOUD_VERIFY_SSL pattern).
|
||||
# Default is None — preserve asyncpg's `prefer` mode so cluster-local
|
||||
# Postgres without TLS works out of the box. Set to True for full
|
||||
# verification or False to silence cert errors against self-signed
|
||||
# homelab servers. DATABASE_CA_BUNDLE points at a private-CA PEM.
|
||||
"database_verify_ssl": None,
|
||||
"database_ca_bundle": None,
|
||||
# Postgres connection pool sizing (ADR-026, reviewer feedback on #798).
|
||||
# Per-pod pool defaults to 10 + 20 overflow = 30 max connections.
|
||||
# With many replicas this can blow past managed-Postgres
|
||||
# `max_connections=100`; tune down via env when needed.
|
||||
"database_pool_size": 10,
|
||||
"database_max_overflow": 20,
|
||||
# Webhook delivery authentication (ADR-010): when set, registrations
|
||||
# tell NC to add `Authorization: Bearer <secret>` to webhook deliveries
|
||||
# and the receiver rejects unauthenticated requests.
|
||||
@@ -304,7 +317,30 @@ def is_sqlite_url(url: str) -> bool:
|
||||
"""Return True for SQLite SQLAlchemy URLs (used to gate sqlite-only logic
|
||||
like file-permission hardening and ``sqlite_master`` legacy lookups).
|
||||
"""
|
||||
return url.startswith("sqlite")
|
||||
return url.lower().startswith("sqlite")
|
||||
|
||||
|
||||
def mask_db_password(url: str) -> str:
|
||||
"""Return a logger-safe rendering of a SQLAlchemy URL.
|
||||
|
||||
DATABASE_URL routinely carries a password (e.g.
|
||||
``postgresql+asyncpg://mcp:secret@db/mcp``); logging it raw leaks the
|
||||
secret to stdout/stderr and any aggregator. SQLAlchemy's
|
||||
:func:`make_url` + ``render_as_string(hide_password=True)`` substitutes
|
||||
a fixed ``***`` placeholder while keeping the rest of the URL intact
|
||||
so operators can still see which host / driver they're hitting.
|
||||
"""
|
||||
try:
|
||||
from sqlalchemy.engine.url import make_url # noqa: PLC0415
|
||||
|
||||
return make_url(url).render_as_string(hide_password=True)
|
||||
except Exception:
|
||||
# If parsing fails (e.g. an explicit ssl-disable test URL with an
|
||||
# exotic shape), fall back to a regex that scrubs any
|
||||
# ``://user:password@`` pattern. Never raise from a logging path.
|
||||
import re # noqa: PLC0415
|
||||
|
||||
return re.sub(r"(://[^:/]+):[^@]*@", r"\1:***@", url)
|
||||
|
||||
|
||||
LOGGING_CONFIG = {
|
||||
@@ -470,6 +506,19 @@ class Settings:
|
||||
nextcloud_verify_ssl: bool = True
|
||||
nextcloud_ca_bundle: str | None = None
|
||||
|
||||
# Postgres backend TLS settings (ADR-026). Default verify_ssl is None,
|
||||
# not True: when DATABASE_URL is unset there's nothing to verify, and
|
||||
# when it is set we don't want to break cluster-internal Postgres that
|
||||
# commonly runs without TLS. Operators opt in to verify-full with True
|
||||
# or supply a private-CA bundle.
|
||||
database_verify_ssl: bool | None = None
|
||||
database_ca_bundle: str | None = None
|
||||
# Postgres connection pool sizing (ADR-026). The asyncpg engine maps
|
||||
# these to its underlying QueuePool. Per-pod max = pool_size +
|
||||
# max_overflow. Validate >= 1 in __post_init__.
|
||||
database_pool_size: int = 10
|
||||
database_max_overflow: int = 20
|
||||
|
||||
# ADR-005: Token Audience Validation (required for OAuth mode)
|
||||
nextcloud_mcp_server_url: str | None = None # MCP server URL (used as audience)
|
||||
nextcloud_resource_uri: str | None = None # Nextcloud resource identifier
|
||||
@@ -603,6 +652,35 @@ class Settings:
|
||||
)
|
||||
logger.info("Using custom CA bundle: %s", self.nextcloud_ca_bundle)
|
||||
|
||||
# Validate Postgres backend TLS configuration (ADR-026)
|
||||
if self.database_verify_ssl is False:
|
||||
logger.warning(
|
||||
"DATABASE_VERIFY_SSL is disabled. "
|
||||
"TLS certificate verification is turned off for the Postgres "
|
||||
"backend. Only acceptable for homelab / self-signed setups; "
|
||||
"prefer DATABASE_CA_BUNDLE for production."
|
||||
)
|
||||
if self.database_ca_bundle:
|
||||
if not os.path.isfile(self.database_ca_bundle):
|
||||
raise ValueError(
|
||||
f"DATABASE_CA_BUNDLE path does not exist: {self.database_ca_bundle}"
|
||||
)
|
||||
logger.info(
|
||||
"Using custom CA bundle for Postgres backend: %s",
|
||||
self.database_ca_bundle,
|
||||
)
|
||||
|
||||
# Pool sizing must be sensible — guard against operators accidentally
|
||||
# setting 0 / negative via env (would deadlock at first request).
|
||||
if self.database_pool_size < 1:
|
||||
raise ValueError(
|
||||
f"DATABASE_POOL_SIZE must be >= 1; got {self.database_pool_size}"
|
||||
)
|
||||
if self.database_max_overflow < 0:
|
||||
raise ValueError(
|
||||
f"DATABASE_MAX_OVERFLOW must be >= 0; got {self.database_max_overflow}"
|
||||
)
|
||||
|
||||
# Ensure mutual exclusivity
|
||||
if self.qdrant_url and self.qdrant_location:
|
||||
raise ValueError(
|
||||
@@ -958,6 +1036,12 @@ def get_settings() -> Settings:
|
||||
# Nextcloud SSL/TLS settings
|
||||
"nextcloud_verify_ssl": "NEXTCLOUD_VERIFY_SSL",
|
||||
"nextcloud_ca_bundle": "NEXTCLOUD_CA_BUNDLE",
|
||||
# Postgres backend TLS (ADR-026)
|
||||
"database_verify_ssl": "DATABASE_VERIFY_SSL",
|
||||
"database_ca_bundle": "DATABASE_CA_BUNDLE",
|
||||
# Postgres backend pool sizing (ADR-026)
|
||||
"database_pool_size": "DATABASE_POOL_SIZE",
|
||||
"database_max_overflow": "DATABASE_MAX_OVERFLOW",
|
||||
# ADR-005: Token Audience Validation
|
||||
"nextcloud_mcp_server_url": "NEXTCLOUD_MCP_SERVER_URL",
|
||||
"nextcloud_resource_uri": "NEXTCLOUD_RESOURCE_URI",
|
||||
@@ -1056,3 +1140,33 @@ def get_nextcloud_ssl_verify() -> bool | ssl.SSLContext:
|
||||
ctx = ssl.create_default_context(cafile=settings.nextcloud_ca_bundle)
|
||||
return ctx
|
||||
return True
|
||||
|
||||
|
||||
def get_database_ssl() -> bool | ssl.SSLContext | None:
|
||||
"""Return the asyncpg ``ssl`` arg for the Postgres backend (ADR-026).
|
||||
|
||||
Returns:
|
||||
- ``None`` when both DATABASE_VERIFY_SSL and DATABASE_CA_BUNDLE are
|
||||
unset — caller skips passing ``ssl`` so asyncpg keeps its default
|
||||
(``prefer``). Preserves PR #798 behavior for cluster-local
|
||||
Postgres without TLS.
|
||||
- ``False`` if DATABASE_VERIFY_SSL=false (silence cert errors).
|
||||
- ``ssl.SSLContext`` if DATABASE_CA_BUNDLE is set (custom private
|
||||
CA, implies verify-full).
|
||||
- ``True`` if DATABASE_VERIFY_SSL=true and no bundle (verify-full
|
||||
against system trust store).
|
||||
|
||||
DATABASE_VERIFY_SSL=false wins over DATABASE_CA_BUNDLE so an operator
|
||||
can quickly silence cert errors during incident response without
|
||||
having to delete the bundle path from their secret store. Matches the
|
||||
Nextcloud-pattern precedence for symmetry with
|
||||
:func:`get_nextcloud_ssl_verify`.
|
||||
"""
|
||||
settings = get_settings()
|
||||
if settings.database_verify_ssl is False:
|
||||
return False
|
||||
if settings.database_ca_bundle:
|
||||
return ssl.create_default_context(cafile=settings.database_ca_bundle)
|
||||
if settings.database_verify_ssl is True:
|
||||
return True
|
||||
return None
|
||||
|
||||
Reference in New Issue
Block a user