fix(storage): address PR #798 review feedback (credentials, asyncpg extra, TLS, pool)

Round-2 fixes after the bot review on PR #798 plus two user follow-ups
(self-signed Postgres support; asyncpg should be a PyPI extra). Folded
into the same PR rather than a follow-up since the work is still
unmerged.

Security
--------
- Mask database credentials in all 5 log call sites (storage.py × 4,
  migrations.py × 1) via a new `mask_db_password()` helper in config.py.
  Uses SQLAlchemy's `make_url(...).render_as_string(hide_password=True)`
  with a regex fallback so the masking path never raises.
- New `tests/unit/test_storage_logging.py` asserts a sentinel password
  never appears in `caplog` during `RefreshTokenStorage.initialize()`.

Distribution
------------
- `asyncpg` moved to `[project.optional-dependencies] postgres` so a
  vanilla `pip install nextcloud-mcp-server` no longer pulls in the
  ~5 MB C extension. The Docker image runs `uv sync --extra postgres`,
  so containerized deployments are unchanged.
- When `DATABASE_URL=postgresql+asyncpg://...` is set on a venv missing
  the extra, `RefreshTokenStorage.initialize()` raises a friendly
  RuntimeError pointing at `[postgres]` rather than the generic
  ModuleNotFoundError.

TLS for the Postgres backend
----------------------------
- New `DATABASE_VERIFY_SSL` + `DATABASE_CA_BUNDLE` env vars mirror the
  existing `NEXTCLOUD_VERIFY_SSL` / `NEXTCLOUD_CA_BUNDLE` pattern
  (validators in Settings.__post_init__, `get_database_ssl()` helper
  alongside `get_nextcloud_ssl_verify()`). `DATABASE_VERIFY_SSL=false`
  wins over `DATABASE_CA_BUNDLE` for incident-response convenience.
- Default is **None** rather than True — keeps PR #798's behavior
  intact for cluster-internal Postgres that runs without TLS. Operators
  opt into verify-full or supply a private CA. ADR-026 records the
  reasoning vs the Nextcloud HTTPS default.
- Engine factory in `storage.py` passes `ssl` via `connect_args` only
  when `get_database_ssl()` returns non-None; otherwise asyncpg's
  default (`prefer`) applies.
- Storage logs which TLS mode is active at INFO (no secret material).

Configurable connection pool
----------------------------
- `DATABASE_POOL_SIZE` (default 10) and `DATABASE_MAX_OVERFLOW`
  (default 20) replace the hardcoded engine values. With many replicas
  this can blow past managed-Postgres `max_connections=100`; tune down
  for large fleets.
- gte-1 / gte-0 validators in __post_init__ reject 0/negative pool
  sizes at startup with the offending value in the error.

Consistency polish
------------------
- Migration 006: convert raw `op.execute("ALTER TABLE ... ADD COLUMN")`
  to `op.batch_alter_table(...).add_column(sa.Column("nonce", sa.Text))`
  for stylistic consistency with the rewritten 001-005. Downgrade now
  drops the column instead of being a no-op.
- `registered_webhooks.created_at` standardized from `sa.Float` to
  `sa.BigInteger` (all other `*_at` columns); `store_webhook()` casts
  `time.time()` → `int`.
- `is_sqlite_url()` made case-insensitive.

Testing
-------
- New `tests/integration/test_storage_postgres.py::test_cleanup_expired_roundtrip`
  exercises `cleanup_expired_tokens`, `cleanup_expired_sessions`, and
  `cleanup_expired_browser_sessions` — relies on DELETE rowcount,
  historically dialect-tricky.
- `tests/unit/test_ssl_config.py` extended with `TestDatabaseSSLSettings`
  + `TestGetDatabaseSSL` classes (9 new tests) mirroring the existing
  Nextcloud SSL tests one-for-one.

Docs
----
- `docs/configuration.md` Centralized-Storage section grew the four new
  env vars + a homelab example with a private CA.
- `docs/ADR-026` grew Distribution, TLS, and `alembic/env.py` async-pattern
  subsections explaining the non-obvious design choices.

Helm chart counterpart in cbcoutinho/helm-charts PR #34 (separate
commit on `feat/nextcloud-mcp-server-database-url`).

Verification
------------
- `uv run pytest tests/unit/` — 1025 passed.
- `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 6 passed (including new cleanup test).
- `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean.

Tracked on Astrolabe Cloud POC board, card #99.

---

_This PR was generated with the help of AI, and reviewed by a Human_

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-05-16 18:53:45 +02:00
co-authored by Claude Opus 4.7
parent 292cbb3292
commit f2b7bf132f
14 changed files with 510 additions and 22 deletions
@@ -63,6 +63,70 @@ the fly. The seven `INSERT OR REPLACE` statements were rewritten as
portable `INSERT ... ON CONFLICT (...) DO UPDATE` (SQLite ≥ 3.24, Postgres
≥ 9.5; we already require SQLite ≥ 3.35 elsewhere).
### Distribution: asyncpg is an optional extra, bundled in Docker
`asyncpg` carries a compiled C extension (~5 MB plus a build toolchain on
source installs) — too heavy a default for the
`pip install nextcloud-mcp-server` audience, the majority of whom run the
SQLite path. It is shipped as a PyPI optional dependency::
pip install 'nextcloud-mcp-server[postgres]'
The published Docker image runs `uv sync --extra postgres` so the
container always has the driver, matching the HA-deployment audience
that exercises the Postgres backend. When `DATABASE_URL=postgresql+asyncpg://...`
is set on a venv without the extra installed, `RefreshTokenStorage`
raises a clear actionable error before the engine is built — operators
see "install with `[postgres]` extra" rather than a generic
`ModuleNotFoundError: No module named 'asyncpg'`.
### Alembic env.py runs the async engine inside a worker thread
`nextcloud_mcp_server/alembic/env.py` uses
`async_engine_from_config(...)` + `anyio.run(run_async_migrations)`, and
the runtime invokes it from `RefreshTokenStorage.initialize()` via
`anyio.to_thread.run_sync(upgrade_database, ...)`. This is intentional:
- Alembic wants a synchronous entry point (`upgrade_database()`), but
`async_engine_from_config` returns an async engine.
- Running `anyio.run()` directly inside an already-running event loop
would deadlock; we have to be on a different thread.
- `to_thread.run_sync` puts the call on a worker thread, which has no
running event loop — `anyio.run()` is then free to spin up its own.
The pattern is non-obvious; this note exists so a future maintainer
doesn't try to "simplify" it back into the main loop.
### TLS for the Postgres backend
Two settings mirror the existing `NEXTCLOUD_VERIFY_SSL` /
`NEXTCLOUD_CA_BUNDLE` pattern: `DATABASE_VERIFY_SSL` and
`DATABASE_CA_BUNDLE`. `get_database_ssl()` (in `nextcloud_mcp_server/config.py`)
returns the value to pass to asyncpg via SQLAlchemy's `connect_args={"ssl": ...}`.
The default is deliberately **less strict than the Nextcloud HTTPS
default**: `DATABASE_VERIFY_SSL` defaults to `None` rather than `True`.
When both env vars are unset we omit the `ssl` kwarg entirely and asyncpg's
default (`prefer`) applies — TLS if the server offers it, no certificate
validation. The reasoning:
- Cluster-internal Postgres (CNPG via a Service, RDS over a private VPC,
PgBouncer sidecar) is the common HA pattern and frequently runs without
TLS or with cert hostnames asyncpg wouldn't match anyway.
- The HTTPS analogy doesn't carry over: the Nextcloud client talks to
*external* hostnames over public networks where verify-full is the
right default. The database client talks to a controlled peer.
- Just-shipped PR #798 had no TLS knobs and worked against cluster-local
Postgres-test; flipping the default to `True` here would break that
flow on upgrade.
Operators in production with a managed Postgres opt in with
`DATABASE_VERIFY_SSL=true`. Homelab operators with a private CA set
`DATABASE_CA_BUNDLE=/path/to/ca.pem` (which implies `verify=true`).
`DATABASE_VERIFY_SSL=false` is the escape hatch for incident response —
it wins over `DATABASE_CA_BUNDLE` so an operator can quickly silence
cert errors without editing the secret store.
### Encryption stays in Python (Fernet), not the DB
The DB only ever sees ciphertext for sensitive columns
+18
View File
@@ -136,9 +136,27 @@ TOKEN_ENCRYPTION_KEY=<fernet-key>
|----------|----------|-------------|
| `DATABASE_URL` | Optional | SQLAlchemy async URL for any supported backend. When set, wins over `TOKEN_STORAGE_DB`. Primary supported targets: `postgresql+asyncpg://...` (recommended for HA) and `sqlite+aiosqlite:///...` (development). |
| `TOKEN_STORAGE_DB` | Optional | Legacy SQLite-only path. Used when `DATABASE_URL` is unset. Falls back to a per-process ephemeral tempfile when both are unset. |
| `DATABASE_VERIFY_SSL` | Optional | TLS verification toggle for the Postgres backend. Unset (default) → asyncpg's `prefer` mode (TLS if offered, no verification — keeps cluster-internal Postgres working). `true` → full cert verification. `false` → silence cert errors (homelab / self-signed). |
| `DATABASE_CA_BUNDLE` | Optional | Path to a PEM file containing a private CA. Implies `DATABASE_VERIFY_SSL=true`. Use this for self-hosted Postgres signed by your homelab CA instead of disabling verification. |
| `DATABASE_POOL_SIZE` | Optional (default `10`) | Per-pod SQLAlchemy connection pool size for the Postgres backend. Multiplied by `replicas`, this can exceed managed-Postgres `max_connections=100` defaults — tune down for large fleets. |
| `DATABASE_MAX_OVERFLOW` | Optional (default `20`) | Per-pod overflow connections beyond `DATABASE_POOL_SIZE`. Max per-pod connections = `pool_size + max_overflow`. Set to `0` to make `pool_size` a hard cap. |
Homelab example (self-signed Postgres with a private CA):
```env
DATABASE_URL=postgresql+asyncpg://mcp:secret@pg.lan:5432/mcp
DATABASE_CA_BUNDLE=/etc/ssl/certs/homelab-ca.pem
TOKEN_ENCRYPTION_KEY=<fernet-key>
```
Notes:
- **PyPI extra required.** The `asyncpg` driver is an optional extra so
the default `pip install nextcloud-mcp-server` stays lean. Install
with `pip install 'nextcloud-mcp-server[postgres]'` when using a
Postgres URL. The Docker image bundles it by default. When
`DATABASE_URL=postgresql+asyncpg://...` is set without the extra,
the server fails fast with a clear actionable error.
- **Bring-your-own DB.** The MCP server doesn't provision the database;
it just consumes the URL. Use CNPG, RDS, your existing Helm chart's
Postgres sub-chart, etc.