CI runs `uv run --frozen ty check -- nextcloud_mcp_server` and `uv run pytest -m
unit`, which install the default + dev groups but not the `[postgres]` optional
extra. vector/queue/procrastinate.py imports procrastinate at module scope (the
task registration needs App/Blueprint), so without it installed ty fails on
unresolved imports and the procrastinate unit tests fail to collect.
Add procrastinate + psycopg to the dev group (kept in the [postgres] extra for
production opt-in) so dev/CI always type-check and test against them, while
SQLite/personal installs stay free of the Postgres deps. Matches the repo's
optional-DB-driver philosophy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-architect document ingest from the shared NATS-glued document-processor to a
per-tenant, in-process model owned by nextcloud-mcp-server (Deck #183). The MCP
server now owns both sides of ingest:
- Producer (api role): the scanner defers one job per changed document into the
app's Postgres via procrastinate (queueing_lock dedup; no execution lock, so a
crashed worker can't deadlock a doc — Qdrant upserts are idempotent).
- Consumer (worker role): `nextcloud-mcp-server worker` drains the queue and runs
the existing process_document pipeline; a periodic task reclaims jobs orphaned
in `doing` by a crash.
INGEST_QUEUE selects the transport (auto: postgres when DATABASE_URL is Postgres,
else the in-process anyio queue for SQLite/dev). procrastinate manages its own
tables (applied on a fresh DB at startup and by `db upgrade`). The vector-sync
status surface reads job counts from Postgres in postgres mode. procrastinate +
psycopg3 ship in the [postgres] extra; the app's own engine still uses asyncpg
(driver unification is a follow-up handled in the rendered Helm chart).
NATS JetStream, the Postgres-queue stub, the bus status subscriber, and nats-py
are removed.
BREAKING CHANGE: the external-NATS-ingest env vars are removed
(INGEST_MODE, STATUS_BACKEND, INGEST_BUS_URL, INGEST_BUS_NUM_REPLICAS,
FACT_EVENT_EMITTER). Use INGEST_QUEUE (memory|postgres) and the `worker`
command instead. TENANT_ID is retained (no longer NATS-subject-charset-validated).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the seven §10.2 hook-point modules + five env vars so Astrolabe Cloud can
offload document processing to the external document-processor / embedding
gateway. Purely additive: with every setting unset the server behaves exactly
as today, so self-hosters are unaffected (Deck #92).
Hook points (all default to current monolith behavior):
- config: EMBEDDING_PROVIDER, INGEST_MODE, STATUS_BACKEND,
COLLECTION_METADATA_SOURCE, FACT_EVENT_EMITTER (+ supporting settings),
validated in Settings.__post_init__ (fail-fast STATUS_BACKEND=local with
INGEST_MODE=external); shared canonical.py.
- vector/payload_keys.py + acl_hash.py: cross-impl NAMESPACE/point_id (§2.2)
and BLAKE2b-128 ACL hash (§11), pinned by fixtures shared with the
document-processor repo.
- embedding/gateway_client.py: OpenAI-compatible GatewayProvider authenticating
via M2M OIDC client-credentials (separate realm); manual-only registry entry.
- vector/collection_metadata.py: sentinel-point / API metadata source with env
fallback.
- vector/queue/: hexagonal ingest producer ports + memory/NATS adapters
(Postgres seam); INGEST_MODE=external publishes mcp.ingest.requested.{tenant}
instead of the in-memory stream and skips the in-process processor pool. The
lifespan becomes a composition root across both deployment branches.
- vector/queue/status.py: STATUS_BACKEND=bus subscriber feeding a StatusStore
the vector-sync status endpoint reads.
- admin/payload_backfill.py: POST /api/v1/admin/payload-backfill (admin scope);
processor writes the new payload keys; query-side ACL pre-filter gated behind
ACL_PREFILTER_ENABLED (default off).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-2 fixes after the bot review on PR #798 plus two user follow-ups
(self-signed Postgres support; asyncpg should be a PyPI extra). Folded
into the same PR rather than a follow-up since the work is still
unmerged.
Security
--------
- Mask database credentials in all 5 log call sites (storage.py × 4,
migrations.py × 1) via a new `mask_db_password()` helper in config.py.
Uses SQLAlchemy's `make_url(...).render_as_string(hide_password=True)`
with a regex fallback so the masking path never raises.
- New `tests/unit/test_storage_logging.py` asserts a sentinel password
never appears in `caplog` during `RefreshTokenStorage.initialize()`.
Distribution
------------
- `asyncpg` moved to `[project.optional-dependencies] postgres` so a
vanilla `pip install nextcloud-mcp-server` no longer pulls in the
~5 MB C extension. The Docker image runs `uv sync --extra postgres`,
so containerized deployments are unchanged.
- When `DATABASE_URL=postgresql+asyncpg://...` is set on a venv missing
the extra, `RefreshTokenStorage.initialize()` raises a friendly
RuntimeError pointing at `[postgres]` rather than the generic
ModuleNotFoundError.
TLS for the Postgres backend
----------------------------
- New `DATABASE_VERIFY_SSL` + `DATABASE_CA_BUNDLE` env vars mirror the
existing `NEXTCLOUD_VERIFY_SSL` / `NEXTCLOUD_CA_BUNDLE` pattern
(validators in Settings.__post_init__, `get_database_ssl()` helper
alongside `get_nextcloud_ssl_verify()`). `DATABASE_VERIFY_SSL=false`
wins over `DATABASE_CA_BUNDLE` for incident-response convenience.
- Default is **None** rather than True — keeps PR #798's behavior
intact for cluster-internal Postgres that runs without TLS. Operators
opt into verify-full or supply a private CA. ADR-026 records the
reasoning vs the Nextcloud HTTPS default.
- Engine factory in `storage.py` passes `ssl` via `connect_args` only
when `get_database_ssl()` returns non-None; otherwise asyncpg's
default (`prefer`) applies.
- Storage logs which TLS mode is active at INFO (no secret material).
Configurable connection pool
----------------------------
- `DATABASE_POOL_SIZE` (default 10) and `DATABASE_MAX_OVERFLOW`
(default 20) replace the hardcoded engine values. With many replicas
this can blow past managed-Postgres `max_connections=100`; tune down
for large fleets.
- gte-1 / gte-0 validators in __post_init__ reject 0/negative pool
sizes at startup with the offending value in the error.
Consistency polish
------------------
- Migration 006: convert raw `op.execute("ALTER TABLE ... ADD COLUMN")`
to `op.batch_alter_table(...).add_column(sa.Column("nonce", sa.Text))`
for stylistic consistency with the rewritten 001-005. Downgrade now
drops the column instead of being a no-op.
- `registered_webhooks.created_at` standardized from `sa.Float` to
`sa.BigInteger` (all other `*_at` columns); `store_webhook()` casts
`time.time()` → `int`.
- `is_sqlite_url()` made case-insensitive.
Testing
-------
- New `tests/integration/test_storage_postgres.py::test_cleanup_expired_roundtrip`
exercises `cleanup_expired_tokens`, `cleanup_expired_sessions`, and
`cleanup_expired_browser_sessions` — relies on DELETE rowcount,
historically dialect-tricky.
- `tests/unit/test_ssl_config.py` extended with `TestDatabaseSSLSettings`
+ `TestGetDatabaseSSL` classes (9 new tests) mirroring the existing
Nextcloud SSL tests one-for-one.
Docs
----
- `docs/configuration.md` Centralized-Storage section grew the four new
env vars + a homelab example with a private CA.
- `docs/ADR-026` grew Distribution, TLS, and `alembic/env.py` async-pattern
subsections explaining the non-obvious design choices.
Helm chart counterpart in cbcoutinho/helm-charts PR #34 (separate
commit on `feat/nextcloud-mcp-server-database-url`).
Verification
------------
- `uv run pytest tests/unit/` — 1025 passed.
- `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 6 passed (including new cleanup test).
- `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean.
Tracked on Astrolabe Cloud POC board, card #99.
---
_This PR was generated with the help of AI, and reviewed by a Human_
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a `DATABASE_URL` setting that lets `RefreshTokenStorage` run against
any SQLAlchemy async backend, primarily `postgresql+asyncpg://...` for
HA k8s deployments. Default behavior is unchanged: when `DATABASE_URL` is
unset the server falls back to the existing `TOKEN_STORAGE_DB` path /
ephemeral SQLite tempfile.
Why
---
Today every MCP pod needs its own PVC to hold the SQLite file, which
pins the Deployment to one replica and blocks horizontal scaling. With
this change, operators can point all replicas at a shared Postgres
(CNPG, RDS, etc.) and the pods become stateless. Encryption stays in
Python (Fernet); the database only sees ciphertext.
What changed
------------
- `config.get_database_url()` resolves DATABASE_URL → TOKEN_STORAGE_DB →
ephemeral tempfile in that priority order.
- `RefreshTokenStorage` builds a process-shared `AsyncEngine` in
`initialize()`. SQLite gets NullPool; Postgres gets pool_size=10,
max_overflow=20, pool_pre_ping=True. 30 aiosqlite call sites adapted
via a thin `_DBConn` / `_Cursor` / `_Row` / `_ExecuteCtx` shim so
existing method bodies need no churn beyond the connection
context-manager swap.
- 7 `INSERT OR REPLACE` statements rewritten as portable
`INSERT ... ON CONFLICT (...) DO UPDATE` (SQLite ≥ 3.24, Postgres ≥ 9.5).
- `sqlite_master` legacy-detection lookup replaced with SQLAlchemy
inspector so the path works against either backend.
- File-permission hardening + parent-dir creation gated on
`is_sqlite_url(...)` — centralized backends manage their own filesystem.
- Alembic migrations 001/002/003/005 converted from raw `op.execute(SQL)`
to portable `op.create_table()` / `op.create_index()` with SQLAlchemy
types. All timestamp columns are `sa.BigInteger` so Postgres allocates
BIGINT (unix epochs don't fit in INT4). SQLite treats BIGINT as
INTEGER, so existing deployments at revision 006 see no schema drift.
- `migrations.py` + CLI take URLs; `db {upgrade,downgrade,current,history}`
gain `--database-url / -u` alongside the legacy `--database-path / -d`.
`get_current_revision()` uses SQLAlchemy inspector instead of raw
sqlite3, so the CLI works against Postgres too.
- `docker-compose.yml` adds a `postgres-test` service under the
`postgres` profile (pinned `postgres:16-alpine` digest) for
integration testing.
- Unit storage tests parametrized over backends via shared
`tests/fixtures/storage_backend.py` — every test in
`test_app_password_storage.py` and `test_webhook_storage.py` runs
once per backend that is available. Postgres is opted in by
`TEST_DATABASE_URL`.
- New `tests/integration/test_storage_postgres.py` (5 tests, marked
`postgres` + `integration`) covers refresh-token, app-password,
OAuth-session, webhook, and audit-log paths end-to-end on Postgres.
- New `docs/ADR-026-pluggable-database-backend.md` records the decision;
`docs/configuration.md` documents `DATABASE_URL` with examples.
Out of scope
------------
- No SQLite → Postgres data migration tool (clean cutover; tokens reissue
on next login, webhooks re-register on next sync tick).
- This repo does not provision Postgres. The matching helm chart change
lives in cbcoutinho/helm-charts (database.url / existingSecret values).
Verification
------------
- `uv run pytest tests/unit/` — 1012 passed, SQLite path unchanged.
- `docker compose --profile postgres up -d postgres-test`
- `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres -v`
— 5 passed.
- `TEST_DATABASE_URL=... uv run pytest tests/unit/test_app_password_storage.py
tests/unit/test_webhook_storage.py` — 50 passed (25 per backend).
- `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean.
Tracked on Astrolabe Cloud POC board, card #99.
---
_This PR was generated with the help of AI, and reviewed by a Human_
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The CalDAV REPORT in `_search_events_by_date` unconditionally requested
server-side `<C:expand>`. Per RFC 4791 §9.6.5 the server then normalizes
every expanded DTSTART/DTEND to UTC `Z`, which destroyed two pieces of
information on the read path:
- RFC 5545 floating local times came back as fake-UTC (a `+00:00` suffix
that did not match the stored value), so a 2:30 PM floating event was
indistinguishable from a 14:30 UTC event in the MCP response.
- TZID-bound events lost their IANA TZID context — a "10am America/New_York"
event came back as `14:00:00+00:00`, making it impossible for callers to
reconstruct DST-aware recurrence semantics.
Replace `<C:expand>` with client-side recurrence expansion via the
`recurring-ical-events` library (promoted from transitive to direct dep),
so the wire response retains its original DTSTART format. Surface the
TZID parameter as new `start_tz`/`end_tz` fields on `CalendarEventSummary`.
Add an optional `timezone` (IANA name) parameter to `nc_calendar_create_event`
and `nc_calendar_update_event` so callers can pin a TZID for naive input;
the helper attaches `ZoneInfo(...)` and emits a paired `VTIMEZONE`
component. Naive input without `timezone` continues to store as RFC 5545
floating local time (with a warning logged). Offset-aware input continues
to store as UTC `Z`.
Drive-by: switch the update path's DTSTART/DTEND assignment from raw
`datetime` to `vDDDTypes(dt)` wrappers — the previous code produced invalid
iCal like `DTSTART:2026-05-14 10:00:00+00:00` for any TZ-aware update.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
