admin/mcp-nextcloud
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(storage): pluggable database backend via DATABASE_URL (ADR-026)

Browse Source 
Adds a `DATABASE_URL` setting that lets `RefreshTokenStorage` run against
any SQLAlchemy async backend, primarily `postgresql+asyncpg://...` for
HA k8s deployments. Default behavior is unchanged: when `DATABASE_URL` is
unset the server falls back to the existing `TOKEN_STORAGE_DB` path /
ephemeral SQLite tempfile.

Why
---
Today every MCP pod needs its own PVC to hold the SQLite file, which
pins the Deployment to one replica and blocks horizontal scaling. With
this change, operators can point all replicas at a shared Postgres
(CNPG, RDS, etc.) and the pods become stateless. Encryption stays in
Python (Fernet); the database only sees ciphertext.

What changed
------------
- `config.get_database_url()` resolves DATABASE_URL → TOKEN_STORAGE_DB →
  ephemeral tempfile in that priority order.
- `RefreshTokenStorage` builds a process-shared `AsyncEngine` in
  `initialize()`. SQLite gets NullPool; Postgres gets pool_size=10,
  max_overflow=20, pool_pre_ping=True. 30 aiosqlite call sites adapted
  via a thin `_DBConn` / `_Cursor` / `_Row` / `_ExecuteCtx` shim so
  existing method bodies need no churn beyond the connection
  context-manager swap.
- 7 `INSERT OR REPLACE` statements rewritten as portable
  `INSERT ... ON CONFLICT (...) DO UPDATE` (SQLite ≥ 3.24, Postgres ≥ 9.5).
- `sqlite_master` legacy-detection lookup replaced with SQLAlchemy
  inspector so the path works against either backend.
- File-permission hardening + parent-dir creation gated on
  `is_sqlite_url(...)` — centralized backends manage their own filesystem.
- Alembic migrations 001/002/003/005 converted from raw `op.execute(SQL)`
  to portable `op.create_table()` / `op.create_index()` with SQLAlchemy
  types. All timestamp columns are `sa.BigInteger` so Postgres allocates
  BIGINT (unix epochs don't fit in INT4). SQLite treats BIGINT as
  INTEGER, so existing deployments at revision 006 see no schema drift.
- `migrations.py` + CLI take URLs; `db {upgrade,downgrade,current,history}`
  gain `--database-url / -u` alongside the legacy `--database-path / -d`.
  `get_current_revision()` uses SQLAlchemy inspector instead of raw
  sqlite3, so the CLI works against Postgres too.
- `docker-compose.yml` adds a `postgres-test` service under the
  `postgres` profile (pinned `postgres:16-alpine` digest) for
  integration testing.
- Unit storage tests parametrized over backends via shared
  `tests/fixtures/storage_backend.py` — every test in
  `test_app_password_storage.py` and `test_webhook_storage.py` runs
  once per backend that is available. Postgres is opted in by
  `TEST_DATABASE_URL`.
- New `tests/integration/test_storage_postgres.py` (5 tests, marked
  `postgres` + `integration`) covers refresh-token, app-password,
  OAuth-session, webhook, and audit-log paths end-to-end on Postgres.
- New `docs/ADR-026-pluggable-database-backend.md` records the decision;
  `docs/configuration.md` documents `DATABASE_URL` with examples.

Out of scope
------------
- No SQLite → Postgres data migration tool (clean cutover; tokens reissue
  on next login, webhooks re-register on next sync tick).
- This repo does not provision Postgres. The matching helm chart change
  lives in cbcoutinho/helm-charts (database.url / existingSecret values).

Verification
------------
- `uv run pytest tests/unit/` — 1012 passed, SQLite path unchanged.
- `docker compose --profile postgres up -d postgres-test`
- `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres -v`
  — 5 passed.
- `TEST_DATABASE_URL=... uv run pytest tests/unit/test_app_password_storage.py
  tests/unit/test_webhook_storage.py` — 50 passed (25 per backend).
- `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean.

Tracked on Astrolabe Cloud POC board, card #99.

---

_This PR was generated with the help of AI, and reviewed by a Human_

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-05-16 18:06:42 +02:00
co-authored by Claude Opus 4.7
parent 15a7e680a6
commit 292cbb3292
19 changed files with 1402 additions and 588 deletions
Download Patch File Download Diff File Expand all files Collapse all files
nextcloud_mcp_server/alembic/versions/20251217_2200_001_initial_schema.py
+105 -140
View File
@@ -8,12 +8,19 @@ This migration creates the initial database schema including:
- registered_webhooks: Webhook registration tracking (both OAuth and BasicAuth)
- schema_version: Legacy schema version tracking (deprecated, use alembic_version)
Uses Alembic's portable schema-DDL helpers (``op.create_table`` /
``op.create_index``) with SQLAlchemy types so the DDL is emitted correctly
for both SQLite (BLOB / INTEGER PRIMARY KEY AUTOINCREMENT) and Postgres
(BYTEA / SERIAL). See ADR-026.
Revision ID: 001
Revises:
Create Date: 2025-12-17 22:00:00.000000
"""
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
@@ -24,143 +31,104 @@ depends_on = None
def upgrade() -> None:
"""Create initial database schema."""
"""Create initial database schema.
# Refresh tokens table (OAuth mode only, for background jobs)
op.execute(
"""
CREATE TABLE IF NOT EXISTS refresh_tokens (
user_id TEXT PRIMARY KEY,
encrypted_token BLOB NOT NULL,
expires_at INTEGER,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL,
-- ADR-004 Progressive Consent fields
flow_type TEXT DEFAULT 'hybrid',
token_audience TEXT DEFAULT 'nextcloud',
provisioned_at INTEGER,
provisioning_client_id TEXT,
scopes TEXT,
-- Browser session profile cache
user_profile TEXT,
profile_cached_at INTEGER
)
"""
All ``*_at`` / expiration / timestamp columns use :class:`sa.BigInteger`
so Postgres allocates BIGINT (8-byte) and unix epoch values don't
overflow the 32-bit int32 INTEGER range on long-lived sessions. SQLite
treats BIGINT and INTEGER identically (dynamic typing), so this is
backwards compatible.
"""
op.create_table(
"refresh_tokens",
sa.Column("user_id", sa.Text, primary_key=True),
sa.Column("encrypted_token", sa.LargeBinary, nullable=False),
sa.Column("expires_at", sa.BigInteger),
sa.Column("created_at", sa.BigInteger, nullable=False),
sa.Column("updated_at", sa.BigInteger, nullable=False),
# ADR-004 Progressive Consent fields
sa.Column("flow_type", sa.Text, server_default="hybrid"),
sa.Column("token_audience", sa.Text, server_default="nextcloud"),
sa.Column("provisioned_at", sa.BigInteger),
sa.Column("provisioning_client_id", sa.Text),
sa.Column("scopes", sa.Text),
# Browser session profile cache
sa.Column("user_profile", sa.Text),
sa.Column("profile_cached_at", sa.BigInteger),
)
# Audit logs table (both OAuth and BasicAuth modes)
op.execute(
"""
CREATE TABLE IF NOT EXISTS audit_logs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp INTEGER NOT NULL,
event TEXT NOT NULL,
user_id TEXT NOT NULL,
resource_type TEXT,
resource_id TEXT,
auth_method TEXT,
hostname TEXT
)
"""
op.create_table(
"audit_logs",
sa.Column("id", sa.Integer, primary_key=True, autoincrement=True),
sa.Column("timestamp", sa.BigInteger, nullable=False),
sa.Column("event", sa.Text, nullable=False),
sa.Column("user_id", sa.Text, nullable=False),
sa.Column("resource_type", sa.Text),
sa.Column("resource_id", sa.Text),
sa.Column("auth_method", sa.Text),
sa.Column("hostname", sa.Text),
)
op.create_index("idx_audit_user_timestamp", "audit_logs", ["user_id", "timestamp"])
op.create_table(
"oauth_clients",
sa.Column("id", sa.Integer, primary_key=True, autoincrement=False),
sa.Column("client_id", sa.Text, nullable=False, unique=True),
sa.Column("encrypted_client_secret", sa.LargeBinary, nullable=False),
sa.Column("client_id_issued_at", sa.BigInteger, nullable=False),
sa.Column("client_secret_expires_at", sa.BigInteger, nullable=False),
sa.Column("redirect_uris", sa.Text, nullable=False),
sa.Column("encrypted_registration_access_token", sa.LargeBinary),
sa.Column("registration_client_uri", sa.Text),
sa.Column("created_at", sa.BigInteger, nullable=False),
sa.Column("updated_at", sa.BigInteger, nullable=False),
)
# Index on audit logs for efficient queries
op.execute(
"""
CREATE INDEX IF NOT EXISTS idx_audit_user_timestamp
ON audit_logs(user_id, timestamp)
"""
op.create_table(
"oauth_sessions",
sa.Column("session_id", sa.Text, primary_key=True),
sa.Column("client_id", sa.Text),
sa.Column("client_redirect_uri", sa.Text, nullable=False),
sa.Column("state", sa.Text),
sa.Column("code_challenge", sa.Text),
sa.Column("code_challenge_method", sa.Text),
sa.Column("mcp_authorization_code", sa.Text, unique=True),
sa.Column("idp_access_token", sa.Text),
sa.Column("idp_refresh_token", sa.Text),
sa.Column("user_id", sa.Text),
sa.Column("created_at", sa.BigInteger, nullable=False),
sa.Column("expires_at", sa.BigInteger, nullable=False),
# ADR-004 Progressive Consent fields
sa.Column("flow_type", sa.Text, server_default="hybrid"),
sa.Column("requested_scopes", sa.Text),
sa.Column("granted_scopes", sa.Text),
sa.Column("is_provisioning", sa.Boolean, server_default=sa.false()),
)
op.create_index(
"idx_oauth_sessions_mcp_code",
"oauth_sessions",
["mcp_authorization_code"],
)
# OAuth client credentials storage (OAuth mode only)
op.execute(
"""
CREATE TABLE IF NOT EXISTS oauth_clients (
id INTEGER PRIMARY KEY,
client_id TEXT UNIQUE NOT NULL,
encrypted_client_secret BLOB NOT NULL,
client_id_issued_at INTEGER NOT NULL,
client_secret_expires_at INTEGER NOT NULL,
redirect_uris TEXT NOT NULL,
encrypted_registration_access_token BLOB,
registration_client_uri TEXT,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL
)
"""
# Legacy schema-version table; superseded by alembic_version. Retained
# so pre-Alembic databases that get stamped into the migration chain
# still match the schema fingerprint they had on disk.
op.create_table(
"schema_version",
sa.Column("version", sa.Integer, primary_key=True, autoincrement=False),
sa.Column("applied_at", sa.Float, nullable=False),
)
# OAuth flow sessions (ADR-004 Progressive Consent)
op.execute(
"""
CREATE TABLE IF NOT EXISTS oauth_sessions (
session_id TEXT PRIMARY KEY,
client_id TEXT,
client_redirect_uri TEXT NOT NULL,
state TEXT,
code_challenge TEXT,
code_challenge_method TEXT,
mcp_authorization_code TEXT UNIQUE,
idp_access_token TEXT,
idp_refresh_token TEXT,
user_id TEXT,
created_at INTEGER NOT NULL,
expires_at INTEGER NOT NULL,
-- ADR-004 Progressive Consent fields
flow_type TEXT DEFAULT 'hybrid',
requested_scopes TEXT,
granted_scopes TEXT,
is_provisioning BOOLEAN DEFAULT FALSE
)
"""
)
# Index for MCP authorization code lookups
op.execute(
"""
CREATE INDEX IF NOT EXISTS idx_oauth_sessions_mcp_code
ON oauth_sessions(mcp_authorization_code)
"""
)
# Legacy schema version tracking table
# NOTE: This is deprecated in favor of Alembic's alembic_version table
# Kept for backward compatibility with pre-Alembic databases
op.execute(
"""
CREATE TABLE IF NOT EXISTS schema_version (
version INTEGER PRIMARY KEY,
applied_at REAL NOT NULL
)
"""
)
# Registered webhooks tracking (both BasicAuth and OAuth modes)
op.execute(
"""
CREATE TABLE IF NOT EXISTS registered_webhooks (
id INTEGER PRIMARY KEY AUTOINCREMENT,
webhook_id INTEGER NOT NULL UNIQUE,
preset_id TEXT NOT NULL,
created_at REAL NOT NULL
)
"""
)
# Indexes for efficient webhook queries
op.execute(
"""
CREATE INDEX IF NOT EXISTS idx_webhooks_preset
ON registered_webhooks(preset_id)
"""
)
op.execute(
"""
CREATE INDEX IF NOT EXISTS idx_webhooks_created
ON registered_webhooks(created_at)
"""
op.create_table(
"registered_webhooks",
sa.Column("id", sa.Integer, primary_key=True, autoincrement=True),
sa.Column("webhook_id", sa.Integer, nullable=False, unique=True),
sa.Column("preset_id", sa.Text, nullable=False),
sa.Column("created_at", sa.Float, nullable=False),
)
op.create_index("idx_webhooks_preset", "registered_webhooks", ["preset_id"])
op.create_index("idx_webhooks_created", "registered_webhooks", ["created_at"])
def downgrade() -> None:
@@ -170,16 +138,13 @@ def downgrade() -> None:
Use with extreme caution.
"""
# Drop indexes first
op.execute("DROP INDEX IF EXISTS idx_webhooks_created")
op.execute("DROP INDEX IF EXISTS idx_webhooks_preset")
op.execute("DROP INDEX IF EXISTS idx_oauth_sessions_mcp_code")
op.execute("DROP INDEX IF EXISTS idx_audit_user_timestamp")
# Drop tables
op.execute("DROP TABLE IF EXISTS registered_webhooks")
op.execute("DROP TABLE IF EXISTS schema_version")
op.execute("DROP TABLE IF EXISTS oauth_sessions")
op.execute("DROP TABLE IF EXISTS oauth_clients")
op.execute("DROP TABLE IF EXISTS audit_logs")
op.execute("DROP TABLE IF EXISTS refresh_tokens")
op.drop_index("idx_webhooks_created", table_name="registered_webhooks")
op.drop_index("idx_webhooks_preset", table_name="registered_webhooks")
op.drop_table("registered_webhooks")
op.drop_table("schema_version")
op.drop_index("idx_oauth_sessions_mcp_code", table_name="oauth_sessions")
op.drop_table("oauth_sessions")
op.drop_table("oauth_clients")
op.drop_index("idx_audit_user_timestamp", table_name="audit_logs")
op.drop_table("audit_logs")
op.drop_table("refresh_tokens")
nextcloud_mcp_server/alembic/versions/20260113_1200_002_add_app_passwords.py
+12 -20
View File
@@ -10,6 +10,8 @@ Create Date: 2026-01-13 12:00:00.000000
"""
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
@@ -22,29 +24,19 @@ depends_on = None
def upgrade() -> None:
"""Add app_passwords table for multi-user BasicAuth mode."""
# App passwords table for multi-user BasicAuth background sync
op.execute(
"""
CREATE TABLE IF NOT EXISTS app_passwords (
user_id TEXT PRIMARY KEY,
encrypted_password BLOB NOT NULL,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL
)
"""
)
# Index for efficient user lookups
op.execute(
"""
CREATE INDEX IF NOT EXISTS idx_app_passwords_updated
ON app_passwords(updated_at)
"""
op.create_table(
"app_passwords",
sa.Column("user_id", sa.Text, primary_key=True),
sa.Column("encrypted_password", sa.LargeBinary, nullable=False),
# BigInteger to keep unix epochs in range on Postgres (see 001).
sa.Column("created_at", sa.BigInteger, nullable=False),
sa.Column("updated_at", sa.BigInteger, nullable=False),
)
op.create_index("idx_app_passwords_updated", "app_passwords", ["updated_at"])
def downgrade() -> None:
"""Drop app_passwords table."""
op.execute("DROP INDEX IF EXISTS idx_app_passwords_updated")
op.execute("DROP TABLE IF EXISTS app_passwords")
op.drop_index("idx_app_passwords_updated", table_name="app_passwords")
op.drop_table("app_passwords")
nextcloud_mcp_server/alembic/versions/20260227_1200_003_add_scopes_and_login_flow.py
+28 -61
View File
@@ -12,6 +12,8 @@ Create Date: 2026-02-27 12:00:00.000000
"""
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
@@ -24,72 +26,37 @@ depends_on = None
def upgrade() -> None:
"""Add scopes/username to app_passwords and create login_flow_sessions."""
# Add scopes column (nullable JSON array, NULL = all scopes allowed)
op.execute(
"""
ALTER TABLE app_passwords ADD COLUMN scopes TEXT
"""
)
# Nullable scope columns on the existing app_passwords table.
op.add_column("app_passwords", sa.Column("scopes", sa.Text))
op.add_column("app_passwords", sa.Column("username", sa.Text))
# Add username column (Nextcloud loginName from Login Flow v2)
op.execute(
"""
ALTER TABLE app_passwords ADD COLUMN username TEXT
"""
op.create_table(
"login_flow_sessions",
sa.Column("user_id", sa.Text, primary_key=True),
sa.Column("encrypted_poll_token", sa.LargeBinary, nullable=False),
sa.Column("poll_endpoint", sa.Text, nullable=False),
sa.Column("requested_scopes", sa.Text),
# BigInteger to keep unix epochs in range on Postgres (see 001).
sa.Column("created_at", sa.BigInteger, nullable=False),
sa.Column("expires_at", sa.BigInteger, nullable=False),
)
# Login Flow v2 session tracking
op.execute(
"""
CREATE TABLE IF NOT EXISTS login_flow_sessions (
user_id TEXT PRIMARY KEY,
encrypted_poll_token BLOB NOT NULL,
poll_endpoint TEXT NOT NULL,
requested_scopes TEXT,
created_at INTEGER NOT NULL,
expires_at INTEGER NOT NULL
)
"""
)
# Index for efficient cleanup of expired sessions
op.execute(
"""
CREATE INDEX IF NOT EXISTS idx_login_flow_sessions_expires
ON login_flow_sessions(expires_at)
"""
op.create_index(
"idx_login_flow_sessions_expires",
"login_flow_sessions",
["expires_at"],
)
def downgrade() -> None:
"""Drop login_flow_sessions and remove added columns."""
"""Drop login_flow_sessions and remove added columns.
op.execute("DROP INDEX IF EXISTS idx_login_flow_sessions_expires")
op.execute("DROP TABLE IF EXISTS login_flow_sessions")
``batch_alter_table`` handles SQLite's pre-3.35 lack of ``DROP COLUMN``
by recreating the table; on Postgres it issues a native ``DROP COLUMN``.
"""
# SQLite doesn't support DROP COLUMN before 3.35.0
# Recreate app_passwords without the new columns
op.execute(
"""
CREATE TABLE app_passwords_backup (
user_id TEXT PRIMARY KEY,
encrypted_password BLOB NOT NULL,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL
)
"""
)
op.execute(
"""
INSERT INTO app_passwords_backup (user_id, encrypted_password, created_at, updated_at)
SELECT user_id, encrypted_password, created_at, updated_at FROM app_passwords
"""
)
op.execute("DROP TABLE app_passwords")
op.execute("ALTER TABLE app_passwords_backup RENAME TO app_passwords")
op.execute(
"""
CREATE INDEX IF NOT EXISTS idx_app_passwords_updated
ON app_passwords(updated_at)
"""
)
op.drop_index("idx_login_flow_sessions_expires", table_name="login_flow_sessions")
op.drop_table("login_flow_sessions")
with op.batch_alter_table("app_passwords") as batch_op:
batch_op.drop_column("username")
batch_op.drop_column("scopes")
nextcloud_mcp_server/alembic/versions/20260502_1500_005_add_browser_sessions.py
+14 -24
View File
@@ -10,6 +10,8 @@ Revises: 004
Create Date: 2026-05-02 15:00:00.000000
"""
import sqlalchemy as sa
from alembic import op
revision = "005"
@@ -19,31 +21,19 @@ depends_on = None
def upgrade() -> None:
op.execute(
"""
CREATE TABLE IF NOT EXISTS browser_sessions (
session_id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
created_at INTEGER NOT NULL,
expires_at INTEGER NOT NULL
)
"""
)
op.execute(
"""
CREATE INDEX IF NOT EXISTS idx_browser_sessions_user
ON browser_sessions(user_id)
"""
)
op.execute(
"""
CREATE INDEX IF NOT EXISTS idx_browser_sessions_expires
ON browser_sessions(expires_at)
"""
op.create_table(
"browser_sessions",
sa.Column("session_id", sa.Text, primary_key=True),
sa.Column("user_id", sa.Text, nullable=False),
# BigInteger to keep unix epochs in range on Postgres (see 001).
sa.Column("created_at", sa.BigInteger, nullable=False),
sa.Column("expires_at", sa.BigInteger, nullable=False),
)
op.create_index("idx_browser_sessions_user", "browser_sessions", ["user_id"])
op.create_index("idx_browser_sessions_expires", "browser_sessions", ["expires_at"])
def downgrade() -> None:
op.execute("DROP INDEX IF EXISTS idx_browser_sessions_expires")
op.execute("DROP INDEX IF EXISTS idx_browser_sessions_user")
op.execute("DROP TABLE IF EXISTS browser_sessions")
op.drop_index("idx_browser_sessions_expires", table_name="browser_sessions")
op.drop_index("idx_browser_sessions_user", table_name="browser_sessions")
op.drop_table("browser_sessions")