fix(storage): address PR #798 round-3 review (SonarQube + pool sizing + RETURNING test)
Round-3 fixes. Two threads: - 9 OPEN SonarQube issues caused the "E Security Rating on New Code" gate failure. The bot's diagnosis (sa.text(text_sql) → SQL injection) was a wrong guess; the actual SQ rules firing were different. - Bot's substantive concerns: pool defaults too aggressive, delete_browser_session RETURNING path untested on Postgres, schema_version legacy table created on Postgres, stale module docstring. - User's underlying question on the pool: "isn't 1 connection enough?" Right-sized to 2+5 and documented the concurrency model in ADR-026 so the rationale is durable. SonarQube quality-gate fixes (clears all 9 OPEN issues) ------------------------------------------------------- - BLOCKER S6418: rename `SECRET` constant in test_storage_logging.py to `SENTINEL_PASSWORD_FRAGMENT` + NOSONAR with rationale. - CRITICAL S3776: extract `_build_postgres_engine()` from `initialize()` (was complexity 26 > 15); incidentally creates a clean unit-test seam for engine args. - CRITICAL S4423: `ssl.create_default_context(cafile=...)` is flagged as "weak protocol" — Python 3.10+ already negotiates the strongest available protocol. Explicitly pass `purpose=ssl.Purpose.SERVER_AUTH` and NOSONAR with the Python-version rationale. - MAJOR S3358: split the TLS-mode nested ternary in the engine factory into a `_describe_ssl_arg()` helper. - MAJOR S2068 ×3: bind test app-password literals to local vars and put `# NOSONAR S2068` on the same line as the literal (anchoring requirement) instead of on the closing paren. - MINOR S7503 ×2: `# NOSONAR S7503` on `_Cursor.__aenter__/__aexit__` — they MUST be `async` per the context-manager protocol. Pool sizing right-sized (answers "why so many connections?") ------------------------------------------------------------ - `DATABASE_POOL_SIZE` default 10 → **2**. - `DATABASE_MAX_OVERFLOW` default 20 → **5**. - Per-pod max drops from 30 to 7. With 3 replicas, total = 21 connections (was 90) — well under managed-Postgres `max_connections=100`. - New INFO log at startup: `Postgres engine ready: pool_size=N max_overflow=M (per-pod max K connections)`. Surfaces the active sizing without grepping config. - New ADR-026 § "Concurrency model and pool sizing" explains asyncpg's single-flight connection semantics, the MCP workload shape (read-mostly point lookups), why-not-1 (multi-user serialization), and the tune-up/tune-down recipe. - `docs/configuration.md` table updated with new defaults + homelab-vs-prod tuning guidance, linking the ADR. RETURNING path covered on Postgres ---------------------------------- - New `test_browser_session_delete_returning` exercises the `DELETE … RETURNING user_id` path — the only RETURNING clause in the storage layer and the most dialect-sensitive SQL in this PR. Asserts both present-row (returns True, row gone) and absent-row (returns False) branches. Schema portability polish ------------------------- - `alembic 001`: gate `schema_version` table creation on `op.get_bind().dialect.name == "sqlite"`. The table exists purely to match the fingerprint of pre-Alembic SQLite databases; fresh Postgres installs no longer carry the dead legacy table. Misc polish ----------- - Module docstring: "SQLite-based" → "SQL-backed", with a sentence on the DATABASE_URL opt-in and an ADR-026 link. - Comment on `_wrap_row` noting `row._mapping` is the documented RowMapping accessor in SQLAlchemy 2.x despite the underscore. Skipped (rationale in PR reply) ------------------------------- - `_qmark_to_named` SQL-comment handling: docstring already notes the limitation; no `?` in storage SQL comments today. - Module-level `anyio.Lock()`: established precedent confirmed by the bot itself. - `get_audit_logs` `SELECT *`: pre-existing pattern, out of scope. Verification ------------ - `uv run pytest tests/unit/` — 1025 passed. - `TEST_DATABASE_URL=… uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 7 passed. - `ruff check && ruff format --check && ty check` — clean. - Confirmed `schema_version` absent on fresh Postgres, still present on fresh SQLite. Tracked on Astrolabe Cloud POC board, card #99. --- _This PR was generated with the help of AI, and reviewed by a Human_ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
f2b7bf132f
commit
51419329b0
@@ -1,8 +1,14 @@
|
||||
"""
|
||||
Persistent Storage for MCP Server State
|
||||
|
||||
This module provides SQLite-based storage for multiple concerns across both
|
||||
BasicAuth and OAuth authentication modes:
|
||||
This module provides SQL-backed storage for multiple concerns across both
|
||||
BasicAuth and OAuth authentication modes. The default backend is SQLite
|
||||
(file-based or per-process tempfile); set ``DATABASE_URL`` to a
|
||||
``postgresql+asyncpg://...`` URL for HA k8s deployments where pods need
|
||||
to be stateless. See :doc:`ADR-026 </docs/ADR-026-pluggable-database-backend>`
|
||||
for the design.
|
||||
|
||||
Concerns covered:
|
||||
|
||||
1. **Refresh Tokens** (OAuth mode only, for background jobs)
|
||||
- Securely stores encrypted refresh tokens for offline access
|
||||
@@ -145,9 +151,26 @@ class _Row:
|
||||
def _wrap_row(row) -> _Row | None:
|
||||
if row is None:
|
||||
return None
|
||||
# ``row._mapping`` is the documented public RowMapping accessor in
|
||||
# SQLAlchemy 2.x (the leading underscore is historical); it returns
|
||||
# a column-name → value mapping that survives the row being
|
||||
# tuple-iterated. See SQLAlchemy 2.x ``Row.mapping`` docs.
|
||||
return _Row(tuple(row), dict(row._mapping))
|
||||
|
||||
|
||||
def _describe_ssl_arg(ssl_arg: object) -> str:
|
||||
"""Render the ``ssl`` value for the startup log line.
|
||||
|
||||
Split out of the engine factory to avoid a nested-ternary
|
||||
SonarQube finding (``S3358``) and to make the cases readable.
|
||||
"""
|
||||
if ssl_arg is False:
|
||||
return "disabled"
|
||||
if isinstance(ssl_arg, bool):
|
||||
return "verify-full (system CAs)"
|
||||
return "custom CA bundle"
|
||||
|
||||
|
||||
def _wrap_rows(rows) -> list[_Row]:
|
||||
"""Wrap a list of SQLAlchemy rows; iterator never yields ``None``."""
|
||||
return [_Row(tuple(r), dict(r._mapping)) for r in rows]
|
||||
@@ -182,10 +205,14 @@ class _Cursor:
|
||||
async def fetchall(self) -> list[_Row]:
|
||||
return _wrap_rows(self._result.fetchall())
|
||||
|
||||
async def __aenter__(self) -> "_Cursor":
|
||||
# NOSONAR S7503 on the next two methods — Python's async-context-manager
|
||||
# protocol *requires* ``__aenter__`` / ``__aexit__`` to be coroutines
|
||||
# even when the body has nothing to await; dropping ``async`` would
|
||||
# break ``async with _Cursor(...)``.
|
||||
async def __aenter__(self) -> "_Cursor": # NOSONAR S7503
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *exc: object) -> None:
|
||||
async def __aexit__(self, *exc: object) -> None: # NOSONAR S7503
|
||||
# SQLAlchemy Result closes when the connection closes; no-op here.
|
||||
return None
|
||||
|
||||
@@ -419,8 +446,8 @@ class RefreshTokenStorage:
|
||||
|
||||
# Create the shared async engine for the chosen backend. SQLite uses
|
||||
# NullPool (per-call connections, matches the prior aiosqlite-direct
|
||||
# behavior); Postgres uses the default pool with pre-ping so dropped
|
||||
# connections from idle k8s networks are retried transparently.
|
||||
# behavior); Postgres uses a small bounded pool — see
|
||||
# ``_build_postgres_engine`` for sizing rationale.
|
||||
if is_sqlite:
|
||||
self.engine = create_async_engine(
|
||||
self.database_url,
|
||||
@@ -429,45 +456,7 @@ class RefreshTokenStorage:
|
||||
future=True,
|
||||
)
|
||||
else:
|
||||
# Postgres ships as an optional PyPI extra (`[postgres]`) so the
|
||||
# default `pip install nextcloud-mcp-server` audience doesn't
|
||||
# pull in asyncpg's C extension. The Docker image bundles it.
|
||||
# Surface a clear actionable error when the driver is missing
|
||||
# rather than the generic ModuleNotFoundError SQLAlchemy emits.
|
||||
if "+asyncpg" in self.database_url.lower() and (
|
||||
importlib.util.find_spec("asyncpg") is None
|
||||
):
|
||||
raise RuntimeError(
|
||||
"DATABASE_URL points at Postgres via asyncpg but the "
|
||||
"'asyncpg' driver is not installed. Install with "
|
||||
"`pip install nextcloud-mcp-server[postgres]` or use "
|
||||
"the Docker image, which bundles it. See ADR-026."
|
||||
)
|
||||
# Conditionally pass TLS config through to asyncpg. When
|
||||
# get_database_ssl() returns None we omit ``ssl`` entirely so
|
||||
# asyncpg's default (``prefer``) applies — keeps cluster-local
|
||||
# Postgres without TLS working out of the box. See ADR-026.
|
||||
connect_args: dict[str, object] = {}
|
||||
ssl_arg = get_database_ssl()
|
||||
if ssl_arg is not None:
|
||||
connect_args["ssl"] = ssl_arg
|
||||
logger.info(
|
||||
"Postgres backend TLS: %s",
|
||||
"disabled"
|
||||
if ssl_arg is False
|
||||
else "custom CA bundle"
|
||||
if not isinstance(ssl_arg, bool)
|
||||
else "verify-full (system CAs)",
|
||||
)
|
||||
settings = get_settings()
|
||||
self.engine = create_async_engine(
|
||||
self.database_url,
|
||||
pool_size=settings.database_pool_size,
|
||||
max_overflow=settings.database_max_overflow,
|
||||
pool_pre_ping=True,
|
||||
connect_args=connect_args,
|
||||
future=True,
|
||||
)
|
||||
self.engine = self._build_postgres_engine()
|
||||
self._dialect = self.engine.dialect.name
|
||||
|
||||
# Check database state with the SQLAlchemy inspector so the legacy
|
||||
@@ -511,6 +500,64 @@ class RefreshTokenStorage:
|
||||
mask_db_password(self.database_url),
|
||||
)
|
||||
|
||||
def _build_postgres_engine(self) -> AsyncEngine:
|
||||
"""Construct the AsyncEngine for a Postgres ``DATABASE_URL``.
|
||||
|
||||
Split out from :meth:`initialize` so cognitive complexity stays
|
||||
under the SonarQube ``S3776`` threshold and so a future
|
||||
engine-arg unit test has a single seam to mock.
|
||||
|
||||
Defaults to ``pool_size=2, max_overflow=5`` (max 7 connections
|
||||
per pod) — see ADR-026 § "Concurrency model and pool sizing"
|
||||
for the rationale. asyncpg connections are single-flight, so
|
||||
the pool only needs to cover the typical multi-user MCP burst,
|
||||
not every potential in-flight tool call.
|
||||
"""
|
||||
# asyncpg ships as an optional PyPI extra (`[postgres]`) so the
|
||||
# default `pip install nextcloud-mcp-server` audience doesn't
|
||||
# pull in the C extension. The Docker image bundles it. Surface
|
||||
# a clear actionable error when the driver is missing rather
|
||||
# than the generic ``ModuleNotFoundError`` SQLAlchemy emits.
|
||||
if "+asyncpg" in self.database_url.lower() and (
|
||||
importlib.util.find_spec("asyncpg") is None
|
||||
):
|
||||
raise RuntimeError(
|
||||
"DATABASE_URL points at Postgres via asyncpg but the "
|
||||
"'asyncpg' driver is not installed. Install with "
|
||||
"`pip install nextcloud-mcp-server[postgres]` or use "
|
||||
"the Docker image, which bundles it. See ADR-026."
|
||||
)
|
||||
|
||||
# Conditionally pass TLS config through to asyncpg. When
|
||||
# ``get_database_ssl()`` returns None we omit ``ssl`` entirely
|
||||
# so asyncpg's default (``prefer``) applies — keeps
|
||||
# cluster-local Postgres without TLS working out of the box.
|
||||
connect_args: dict[str, object] = {}
|
||||
ssl_arg = get_database_ssl()
|
||||
if ssl_arg is not None:
|
||||
connect_args["ssl"] = ssl_arg
|
||||
logger.info("Postgres backend TLS: %s", _describe_ssl_arg(ssl_arg))
|
||||
|
||||
settings = get_settings()
|
||||
engine = create_async_engine(
|
||||
self.database_url,
|
||||
pool_size=settings.database_pool_size,
|
||||
max_overflow=settings.database_max_overflow,
|
||||
pool_pre_ping=True,
|
||||
connect_args=connect_args,
|
||||
future=True,
|
||||
)
|
||||
# Log the configured sizing so operators can spot
|
||||
# over-allocation at startup without grepping config.
|
||||
logger.info(
|
||||
"Postgres engine ready: pool_size=%d max_overflow=%d "
|
||||
"(per-pod max %d connections)",
|
||||
settings.database_pool_size,
|
||||
settings.database_max_overflow,
|
||||
settings.database_pool_size + settings.database_max_overflow,
|
||||
)
|
||||
return engine
|
||||
|
||||
@asynccontextmanager
|
||||
async def _db(self):
|
||||
"""Open a backend-agnostic connection.
|
||||
|
||||
Reference in New Issue
Block a user