fix(storage): address PR #798 round-3 review (SonarQube + pool sizing + RETURNING test)

Round-3 fixes. Two threads:
- 9 OPEN SonarQube issues caused the "E Security Rating on New Code"
  gate failure. The bot's diagnosis (sa.text(text_sql) → SQL injection)
  was a wrong guess; the actual SQ rules firing were different.
- Bot's substantive concerns: pool defaults too aggressive,
  delete_browser_session RETURNING path untested on Postgres,
  schema_version legacy table created on Postgres, stale module
  docstring.
- User's underlying question on the pool: "isn't 1 connection enough?"
  Right-sized to 2+5 and documented the concurrency model in ADR-026
  so the rationale is durable.

SonarQube quality-gate fixes (clears all 9 OPEN issues)
-------------------------------------------------------
- BLOCKER S6418: rename `SECRET` constant in test_storage_logging.py
  to `SENTINEL_PASSWORD_FRAGMENT` + NOSONAR with rationale.
- CRITICAL S3776: extract `_build_postgres_engine()` from
  `initialize()` (was complexity 26 > 15); incidentally creates a
  clean unit-test seam for engine args.
- CRITICAL S4423: `ssl.create_default_context(cafile=...)` is flagged
  as "weak protocol" — Python 3.10+ already negotiates the strongest
  available protocol. Explicitly pass `purpose=ssl.Purpose.SERVER_AUTH`
  and NOSONAR with the Python-version rationale.
- MAJOR S3358: split the TLS-mode nested ternary in the engine
  factory into a `_describe_ssl_arg()` helper.
- MAJOR S2068 ×3: bind test app-password literals to local vars and
  put `# NOSONAR S2068` on the same line as the literal (anchoring
  requirement) instead of on the closing paren.
- MINOR S7503 ×2: `# NOSONAR S7503` on `_Cursor.__aenter__/__aexit__`
  — they MUST be `async` per the context-manager protocol.

Pool sizing right-sized (answers "why so many connections?")
------------------------------------------------------------
- `DATABASE_POOL_SIZE` default 10 → **2**.
- `DATABASE_MAX_OVERFLOW` default 20 → **5**.
- Per-pod max drops from 30 to 7. With 3 replicas, total = 21
  connections (was 90) — well under managed-Postgres
  `max_connections=100`.
- New INFO log at startup: `Postgres engine ready: pool_size=N
  max_overflow=M (per-pod max K connections)`. Surfaces the active
  sizing without grepping config.
- New ADR-026 § "Concurrency model and pool sizing" explains
  asyncpg's single-flight connection semantics, the MCP workload
  shape (read-mostly point lookups), why-not-1 (multi-user
  serialization), and the tune-up/tune-down recipe.
- `docs/configuration.md` table updated with new defaults +
  homelab-vs-prod tuning guidance, linking the ADR.

RETURNING path covered on Postgres
----------------------------------
- New `test_browser_session_delete_returning` exercises the
  `DELETE … RETURNING user_id` path — the only RETURNING clause in
  the storage layer and the most dialect-sensitive SQL in this PR.
  Asserts both present-row (returns True, row gone) and absent-row
  (returns False) branches.

Schema portability polish
-------------------------
- `alembic 001`: gate `schema_version` table creation on
  `op.get_bind().dialect.name == "sqlite"`. The table exists purely
  to match the fingerprint of pre-Alembic SQLite databases; fresh
  Postgres installs no longer carry the dead legacy table.

Misc polish
-----------
- Module docstring: "SQLite-based" → "SQL-backed", with a sentence
  on the DATABASE_URL opt-in and an ADR-026 link.
- Comment on `_wrap_row` noting `row._mapping` is the documented
  RowMapping accessor in SQLAlchemy 2.x despite the underscore.

Skipped (rationale in PR reply)
-------------------------------
- `_qmark_to_named` SQL-comment handling: docstring already notes
  the limitation; no `?` in storage SQL comments today.
- Module-level `anyio.Lock()`: established precedent confirmed by
  the bot itself.
- `get_audit_logs` `SELECT *`: pre-existing pattern, out of scope.

Verification
------------
- `uv run pytest tests/unit/` — 1025 passed.
- `TEST_DATABASE_URL=… uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 7 passed.
- `ruff check && ruff format --check && ty check` — clean.
- Confirmed `schema_version` absent on fresh Postgres, still present
  on fresh SQLite.

Tracked on Astrolabe Cloud POC board, card #99.

---

_This PR was generated with the help of AI, and reviewed by a Human_

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-05-16 19:33:23 +02:00
co-authored by Claude Opus 4.7
parent f2b7bf132f
commit 51419329b0
7 changed files with 229 additions and 78 deletions
@@ -111,14 +111,18 @@ def upgrade() -> None:
["mcp_authorization_code"],
)
# Legacy schema-version table; superseded by alembic_version. Retained
# so pre-Alembic databases that get stamped into the migration chain
# still match the schema fingerprint they had on disk.
op.create_table(
"schema_version",
sa.Column("version", sa.Integer, primary_key=True, autoincrement=False),
sa.Column("applied_at", sa.Float, nullable=False),
)
# Legacy schema-version table; superseded by alembic_version. Only
# created on SQLite because it exists *purely* to match the
# fingerprint of pre-Alembic SQLite databases that get stamped into
# the migration chain (see ``RefreshTokenStorage.initialize()``).
# Fresh Postgres installs have no pre-Alembic history and don't
# need it. PR #798 round-3 review (#4).
if op.get_bind().dialect.name == "sqlite":
op.create_table(
"schema_version",
sa.Column("version", sa.Integer, primary_key=True, autoincrement=False),
sa.Column("applied_at", sa.Float, nullable=False),
)
op.create_table(
"registered_webhooks",
@@ -144,7 +148,9 @@ def downgrade() -> None:
op.drop_index("idx_webhooks_created", table_name="registered_webhooks")
op.drop_index("idx_webhooks_preset", table_name="registered_webhooks")
op.drop_table("registered_webhooks")
op.drop_table("schema_version")
# ``schema_version`` is only created on SQLite (see ``upgrade()``).
if op.get_bind().dialect.name == "sqlite":
op.drop_table("schema_version")
op.drop_index("idx_oauth_sessions_mcp_code", table_name="oauth_sessions")
op.drop_table("oauth_sessions")
op.drop_table("oauth_clients")
+92 -45
View File
@@ -1,8 +1,14 @@
"""
Persistent Storage for MCP Server State
This module provides SQLite-based storage for multiple concerns across both
BasicAuth and OAuth authentication modes:
This module provides SQL-backed storage for multiple concerns across both
BasicAuth and OAuth authentication modes. The default backend is SQLite
(file-based or per-process tempfile); set ``DATABASE_URL`` to a
``postgresql+asyncpg://...`` URL for HA k8s deployments where pods need
to be stateless. See :doc:`ADR-026 </docs/ADR-026-pluggable-database-backend>`
for the design.
Concerns covered:
1. **Refresh Tokens** (OAuth mode only, for background jobs)
- Securely stores encrypted refresh tokens for offline access
@@ -145,9 +151,26 @@ class _Row:
def _wrap_row(row) -> _Row | None:
if row is None:
return None
# ``row._mapping`` is the documented public RowMapping accessor in
# SQLAlchemy 2.x (the leading underscore is historical); it returns
# a column-name → value mapping that survives the row being
# tuple-iterated. See SQLAlchemy 2.x ``Row.mapping`` docs.
return _Row(tuple(row), dict(row._mapping))
def _describe_ssl_arg(ssl_arg: object) -> str:
"""Render the ``ssl`` value for the startup log line.
Split out of the engine factory to avoid a nested-ternary
SonarQube finding (``S3358``) and to make the cases readable.
"""
if ssl_arg is False:
return "disabled"
if isinstance(ssl_arg, bool):
return "verify-full (system CAs)"
return "custom CA bundle"
def _wrap_rows(rows) -> list[_Row]:
"""Wrap a list of SQLAlchemy rows; iterator never yields ``None``."""
return [_Row(tuple(r), dict(r._mapping)) for r in rows]
@@ -182,10 +205,14 @@ class _Cursor:
async def fetchall(self) -> list[_Row]:
return _wrap_rows(self._result.fetchall())
async def __aenter__(self) -> "_Cursor":
# NOSONAR S7503 on the next two methods — Python's async-context-manager
# protocol *requires* ``__aenter__`` / ``__aexit__`` to be coroutines
# even when the body has nothing to await; dropping ``async`` would
# break ``async with _Cursor(...)``.
async def __aenter__(self) -> "_Cursor": # NOSONAR S7503
return self
async def __aexit__(self, *exc: object) -> None:
async def __aexit__(self, *exc: object) -> None: # NOSONAR S7503
# SQLAlchemy Result closes when the connection closes; no-op here.
return None
@@ -419,8 +446,8 @@ class RefreshTokenStorage:
# Create the shared async engine for the chosen backend. SQLite uses
# NullPool (per-call connections, matches the prior aiosqlite-direct
# behavior); Postgres uses the default pool with pre-ping so dropped
# connections from idle k8s networks are retried transparently.
# behavior); Postgres uses a small bounded pool — see
# ``_build_postgres_engine`` for sizing rationale.
if is_sqlite:
self.engine = create_async_engine(
self.database_url,
@@ -429,45 +456,7 @@ class RefreshTokenStorage:
future=True,
)
else:
# Postgres ships as an optional PyPI extra (`[postgres]`) so the
# default `pip install nextcloud-mcp-server` audience doesn't
# pull in asyncpg's C extension. The Docker image bundles it.
# Surface a clear actionable error when the driver is missing
# rather than the generic ModuleNotFoundError SQLAlchemy emits.
if "+asyncpg" in self.database_url.lower() and (
importlib.util.find_spec("asyncpg") is None
):
raise RuntimeError(
"DATABASE_URL points at Postgres via asyncpg but the "
"'asyncpg' driver is not installed. Install with "
"`pip install nextcloud-mcp-server[postgres]` or use "
"the Docker image, which bundles it. See ADR-026."
)
# Conditionally pass TLS config through to asyncpg. When
# get_database_ssl() returns None we omit ``ssl`` entirely so
# asyncpg's default (``prefer``) applies — keeps cluster-local
# Postgres without TLS working out of the box. See ADR-026.
connect_args: dict[str, object] = {}
ssl_arg = get_database_ssl()
if ssl_arg is not None:
connect_args["ssl"] = ssl_arg
logger.info(
"Postgres backend TLS: %s",
"disabled"
if ssl_arg is False
else "custom CA bundle"
if not isinstance(ssl_arg, bool)
else "verify-full (system CAs)",
)
settings = get_settings()
self.engine = create_async_engine(
self.database_url,
pool_size=settings.database_pool_size,
max_overflow=settings.database_max_overflow,
pool_pre_ping=True,
connect_args=connect_args,
future=True,
)
self.engine = self._build_postgres_engine()
self._dialect = self.engine.dialect.name
# Check database state with the SQLAlchemy inspector so the legacy
@@ -511,6 +500,64 @@ class RefreshTokenStorage:
mask_db_password(self.database_url),
)
def _build_postgres_engine(self) -> AsyncEngine:
"""Construct the AsyncEngine for a Postgres ``DATABASE_URL``.
Split out from :meth:`initialize` so cognitive complexity stays
under the SonarQube ``S3776`` threshold and so a future
engine-arg unit test has a single seam to mock.
Defaults to ``pool_size=2, max_overflow=5`` (max 7 connections
per pod) — see ADR-026 § "Concurrency model and pool sizing"
for the rationale. asyncpg connections are single-flight, so
the pool only needs to cover the typical multi-user MCP burst,
not every potential in-flight tool call.
"""
# asyncpg ships as an optional PyPI extra (`[postgres]`) so the
# default `pip install nextcloud-mcp-server` audience doesn't
# pull in the C extension. The Docker image bundles it. Surface
# a clear actionable error when the driver is missing rather
# than the generic ``ModuleNotFoundError`` SQLAlchemy emits.
if "+asyncpg" in self.database_url.lower() and (
importlib.util.find_spec("asyncpg") is None
):
raise RuntimeError(
"DATABASE_URL points at Postgres via asyncpg but the "
"'asyncpg' driver is not installed. Install with "
"`pip install nextcloud-mcp-server[postgres]` or use "
"the Docker image, which bundles it. See ADR-026."
)
# Conditionally pass TLS config through to asyncpg. When
# ``get_database_ssl()`` returns None we omit ``ssl`` entirely
# so asyncpg's default (``prefer``) applies — keeps
# cluster-local Postgres without TLS working out of the box.
connect_args: dict[str, object] = {}
ssl_arg = get_database_ssl()
if ssl_arg is not None:
connect_args["ssl"] = ssl_arg
logger.info("Postgres backend TLS: %s", _describe_ssl_arg(ssl_arg))
settings = get_settings()
engine = create_async_engine(
self.database_url,
pool_size=settings.database_pool_size,
max_overflow=settings.database_max_overflow,
pool_pre_ping=True,
connect_args=connect_args,
future=True,
)
# Log the configured sizing so operators can spot
# over-allocation at startup without grepping config.
logger.info(
"Postgres engine ready: pool_size=%d max_overflow=%d "
"(per-pod max %d connections)",
settings.database_pool_size,
settings.database_max_overflow,
settings.database_pool_size + settings.database_max_overflow,
)
return engine
@asynccontextmanager
async def _db(self):
"""Open a backend-agnostic connection.
+23 -10
View File
@@ -67,12 +67,14 @@ _DEFAULTS: dict[str, Any] = {
# homelab servers. DATABASE_CA_BUNDLE points at a private-CA PEM.
"database_verify_ssl": None,
"database_ca_bundle": None,
# Postgres connection pool sizing (ADR-026, reviewer feedback on #798).
# Per-pod pool defaults to 10 + 20 overflow = 30 max connections.
# With many replicas this can blow past managed-Postgres
# `max_connections=100`; tune down via env when needed.
"database_pool_size": 10,
"database_max_overflow": 20,
# Postgres connection pool sizing (ADR-026 → "Concurrency model and
# pool sizing"). Per-pod defaults to 2 + 5 overflow = 7 max
# connections. asyncpg connections are single-flight, so the pool
# only needs to cover typical multi-user MCP burst — not every
# potential in-flight tool call. Tune up with DATABASE_POOL_SIZE /
# DATABASE_MAX_OVERFLOW for high-traffic prod fleets.
"database_pool_size": 2,
"database_max_overflow": 5,
# Webhook delivery authentication (ADR-010): when set, registrations
# tell NC to add `Authorization: Bearer <secret>` to webhook deliveries
# and the receiver rejects unauthenticated requests.
@@ -515,9 +517,12 @@ class Settings:
database_ca_bundle: str | None = None
# Postgres connection pool sizing (ADR-026). The asyncpg engine maps
# these to its underlying QueuePool. Per-pod max = pool_size +
# max_overflow. Validate >= 1 in __post_init__.
database_pool_size: int = 10
database_max_overflow: int = 20
# max_overflow. Defaults are intentionally small (2 + 5 = 7) because
# asyncpg connections are single-flight and the typical MCP workload
# is light read-mostly point lookups. Validate >= 1 / >= 0 in
# __post_init__.
database_pool_size: int = 2
database_max_overflow: int = 5
# ADR-005: Token Audience Validation (required for OAuth mode)
nextcloud_mcp_server_url: str | None = None # MCP server URL (used as audience)
@@ -1166,7 +1171,15 @@ def get_database_ssl() -> bool | ssl.SSLContext | None:
if settings.database_verify_ssl is False:
return False
if settings.database_ca_bundle:
return ssl.create_default_context(cafile=settings.database_ca_bundle)
# ``ssl.create_default_context()`` on Python 3.10+ already negotiates
# the strongest available protocol (TLS 1.2+ with secure ciphers);
# we pin Python 3.11+ in pyproject.toml. ``purpose=SERVER_AUTH`` is
# the default but spelt out here so static analysers (SonarQube
# ``S4423``) can see it explicitly. NOSONAR S4423
return ssl.create_default_context( # NOSONAR S4423
purpose=ssl.Purpose.SERVER_AUTH,
cafile=settings.database_ca_bundle,
)
if settings.database_verify_ssl is True:
return True
return None