The vector index has always been strictly per-user: every Qdrant payload
carries a `user_id` and the search filter is `user_id == querying_user`.
A file Alice indexed cannot be discovered by Bob even if she has shared
it with him — Bob would have to re-index it under his own user_id to
make it searchable, which means duplicate index entries for every share
recipient.
Switch to ownership-with-ACL-expansion:
- New `nextcloud_mcp_server.search.access_filter` module:
- `list_accessible_owners(sharing_client, user_id)` calls the OCS
Sharing API (`shared_with_me=true`) and returns
`{user_id} ∪ {uid_owner of each share}`. Fails open to `[user_id]`
so a misbehaving Sharing API doesn't black-hole search.
- `build_ownership_filter(user_id, accessible_owners)` returns a
Qdrant `Filter` whose `should` branch matches either the new
`owner_id IN accessible_owners` field or the legacy `user_id` field.
The legacy branch keeps points indexed before this change reachable
without a migration backfill.
- Indexer payload (`vector/processor.py`) now writes `owner_id` alongside
`user_id`. `DocumentTask` gains an optional `owner_id` field; today the
scanner always runs as the owner so the processor falls back to
`user_id`, but the field is plumbed so a future shared-with-me crawler
can set the true owner without reshaping the payload contract.
- `SemanticSearchAlgorithm.search` and `BM25HybridSearchAlgorithm.search`
accept `accessible_owners` via kwargs and use the new ownership filter.
Default behaviour with no kwarg is unchanged (self-only).
- Both user-facing callers — the MCP tool path (`server/semantic.py`) and
the visualization Starlette route (`auth/viz_routes.py`) — compute
`accessible_owners` from the authenticated Nextcloud client before
invoking the search algorithm. Eviction, scanner deletion, placeholder,
and chunk-context paths intentionally keep the legacy `user_id`
semantics (those are "operations on a specific user's records", not
cross-user reads).
- 10 new unit tests in `tests/unit/search/test_access_filter.py` cover
self-only default, owner expansion, dedup, fallback fields, OCS
failure, and the legacy `should`-branch shape.
Pairs with cbcoutinho/astrolabe#89 — together they let an Astrolabe user
find content owners have shared with them without going through any
re-authorization flow or re-indexing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The startup sweep was reading settings.qdrant_collection (the raw config
value, default "nextcloud_content") instead of settings.get_collection_name(),
which is what every other vector-sync operation uses. When QDRANT_COLLECTION
is not overridden, get_collection_name() auto-generates a
{deployment-id}-{model-name} name; the sweep was targeting a non-existent
collection and silently returning (0, 0).
Also adds the AsyncQdrantClient type annotation that was missing on
sweep_orphan_placeholders, and renames its parameter from collection_name
to collection to make it clear the value must be the resolved name.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the per-tenant nextcloud-mcp-server Pod OOMKills mid-batch, the
in-memory anyio processor queue is lost but the placeholder Qdrant
points (is_placeholder=true, status=pending) survive. The next Pod's
scanner re-runs, sees the existing placeholders, applies the
5 × VECTOR_SYNC_SCAN_INTERVAL staleness gate (~5h with the deployed
1h scan interval), and skips them. Result: 0 documents indexed for
the duration of the gate after every restart.
Stamps a process-level instance_id (UUID per Pod-process) onto every
placeholder write. A new sweep_orphan_placeholders helper, called
once from starlette_lifespan after the Qdrant client is initialised
and before the scanner / user-manager spawns, scrolls the collection
and deletes any placeholder whose instance_id doesn't match the
current Pod's (including placeholders with no instance_id field —
back-compat for pre-fix Pod versions). The scanner's next cycle
naturally re-creates fresh placeholders and queues work normally;
no DocumentTask reconstruction needed.
Sweep is one-shot at startup, not periodic — the existing staleness
gate still covers same-Pod recovery, and the cross-Pod-restart gap
was the only failure mode. Failure is non-fatal (logged via
vector_sync.orphan_sweep_failed) so a transient Qdrant hiccup at
boot doesn't prevent the scanner from running.
Both lifespan branches (single-user BasicAuth, OAuth / multi-user
BasicAuth) call the sweep via a module-local helper. A new
VECTOR_SYNC_ORPHAN_SWEEP_ENABLED setting (default True) provides
an escape hatch.
Closes Deck #101.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #719 fixed the contact-create path so all documented fields persist to the
vCard, but the read path (list/search via MCP) still returned ``organization:
null`` / ``note: null`` / ``title: null`` because pythonvCard4 has no typed
parser for ORG/TITLE — they land in ``Contact.custom`` — and the server-side
mapper never read ``note``/``urls``/``categories``/``photo`` even when present.
Reads now surface what the write side persisted:
- ``client/contacts.py``: new ``_first_custom`` helper pulls raw values from
``Contact.custom`` for ORG/TITLE/unencoded PHOTO. ``list_contacts``
extends its per-contact dict with org/title/note/url/categories/photo.
- ``server/contacts.py``: ``_raw_contact_to_model`` maps the new keys onto
``Contact.organization`` / ``.title`` / ``.note`` / ``.urls`` / ``.categories``
/ ``.photo``. URL accepts both list and plain-string shapes; categories
accepts comma-separated strings for forward-compat.
Coverage:
- Unit: ``TestFirstCustom`` (five cases incl. bare-string library shape) and
three new ``_raw_contact_to_model`` cases covering the full field set,
plain-string URL, and comma-string categories.
- Integration: ``test_mcp_contacts_workflow`` now decodes the
``nc_contacts_search_contacts`` response and asserts
``organization`` / ``note`` round-trip — direct regression coverage for
elvisdragonmao's report on issue #716.
Verified end-to-end against the local single-user docker stack: creating a
contact with ``{organization, title, note, url, categories}`` and reading it
back via ``nc_contacts_search_contacts`` returns every field populated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Astrolabe (and any other PHP-side client) sends a stable User-Agent on
every outbound call to the MCP server. Capture it at the middleware
layer so backend access logs can attribute each request to a specific
client build — e.g. ``Nextcloud-Astrolabe/0.14.1``.
The middleware fires only for /api/v1/* and /webhooks/nextcloud,
which is the surface PHP-side clients hit; /mcp and /health stay
silent. The structured ``extra`` ({user_agent, http_method, http_path})
flows into OTel spans so the field is queryable in Grafana / Loki.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Nextcloud Deck PR #7910 added IWebhookCompatibleEvent to CardCreated/
Updated/DeletedEvent and BoardUpdatedEvent, so Deck can finally emit
real-time webhooks via core's webhook_listeners app. Wire this into
the existing preset → parser → DocumentTask pipeline that already
backs Notes / Calendar / Tables / Forms / Files sync.
- Add deck_sync preset (app=deck, 4 events) and drop the stale
"Deck does not support webhooks" comment.
- Teach webhook_parser to convert Deck card events into
DocumentTask(doc_type=deck_card, operation=index|delete) with
stack_id metadata. BoardUpdatedEvent logs delivery at INFO and
returns None — the polling scanner reconciles affected cards.
- Cover three new unit tests for the deck create/delete/board-update
paths plus symmetric fail-open tests for missing card.id /
node.id in _parse_deck_event and _parse_file_event.
The astrolabe admin UI auto-discovers the new preset via
filter_presets_by_installed_apps(); no astrolabe-side wiring is
required for it to appear in the Webhook Management card grid.
Note: requires Deck ≥1.18.x (where PR #7910 lands); the preset is
hidden when the Deck app isn't installed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Split the Docker image build into a per-platform matrix and merge job,
producing a single multi-arch manifest (linux/amd64 + linux/arm64) without
QEMU emulation. The arm64 build runs on the native ubuntu-24.04-arm runner.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The ``BM25SparseEmbeddingProvider.__init__`` calls
``fastembed.SparseTextEmbedding(model_name="Qdrant/bm25")`` which
downloads ~50 MB of model weights from HuggingFace and loads them
into memory — observed >5 s wall-clock in production. The inference
methods (``encode_async``, ``encode_batch_async``) already wrap work
in ``anyio.to_thread.run_sync``, so the design intent is clearly to
keep FastEmbed off the event loop. That protection just didn't
cover the constructor.
Symptom in the Astrolabe Cloud per-tenant deploy (deck #102 smoke):
~30–90 s after a user enables semantic search, the pod tips into a
SIGKILL-restart cycle. Loki shows a single log line
Initializing BM25 sparse embedding provider: Qdrant/bm25
followed by nothing else from the event loop until exitCode 137.
Kubernetes ``/health/live`` httpGet probe timeout=5s fires 6 times
in a row, kubelet kills the container, restart, repeat.
Fix: switch ``get_bm25_service()`` to an async accessor that wraps
the first-time construction in ``anyio.to_thread.run_sync``. Two
existing call sites (``vector/processor.py:603``,
``search/bm25_hybrid.py:123``) update to ``await``. Both are
already inside async functions so the await is free.
New unit test pins the invariant by monkey-patching
``BM25SparseEmbeddingProvider.__init__`` with ``time.sleep(1)`` and
asserting a concurrent ``anyio.sleep(0.05)`` finishes promptly —
the test fails if the constructor ever runs back on the event loop.
Same pattern exists in ``OllamaEmbeddingProvider.__init__`` (sync
``httpx.get`` health-check). Ollama isn't enabled in any current
deploy; filed as a follow-up.
Refs:
- Astrolabe Cloud deck card #102 (smoke discovery)
- Sibling fix#799 (NullPool for cross-loop-asyncpg, same class
of "anyio bites you in production" bug)
Verified:
- ``uv run pytest tests/unit/`` — 1027 passed
- ``uv run ruff check`` clean on touched files
- ``uv run ty check`` clean on touched files
- New tests pass
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
claude-review on #799 flagged:
1. Stale inline comment in ``initialize()`` (line 466) still said
"Postgres uses a small bounded pool". Updated to reflect both
backends now use NullPool.
2. Stale ``close()`` docstring referenced pool-size starving
max_connections — irrelevant with NullPool. Replaced with the
NullPool-aware rationale (dispose still tears down in-flight
asyncpg connections cleanly).
3. ``docs/configuration.md`` actively directed operators to tune
DATABASE_POOL_SIZE / DATABASE_MAX_OVERFLOW, with worked
examples and pool math. Both are now deprecated no-ops; the
table entries explain the deprecation and link to PR #799.
Operators reading the docs will no longer be confused into
tuning settings that don't do anything.
4. ``config.py`` comment for the deprecated fields updated to
record the deprecation. Validators are intentionally kept
(still reject < 1 / < 0) so misconfigured deploys fail loudly
rather than silently — the reviewer flagged this as a minor
UX wart but explicitly "not a blocker"; the docs change in (3)
keeps operators away from the config altogether.
5. New ``tests/unit/test_storage_engine.py`` with three tests:
- ``test_postgres_engine_uses_nullpool`` — pins ``isinstance(
engine.pool, NullPool)`` so a refactor back to QueuePool /
SingletonThreadPool can't silently re-introduce the cross-
event-loop crashes.
- ``test_postgres_engine_ignores_pool_sizing_settings`` —
setting DATABASE_POOL_SIZE / DATABASE_MAX_OVERFLOW to huge
values must not change pool type (proves the deprecated
fields are wired-up no-ops).
- ``test_postgres_engine_missing_asyncpg_driver_message`` —
guards the existing actionable-error branch when the
optional ``[postgres]`` extra isn't installed.
Verified:
- ``uv run pytest tests/unit/`` — 1028 passed
- ``uv run ruff check`` clean on the touched python files
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Postgres backend hit a hard crashloop in production deployments
where the MCP server runs under anyio TaskGroups with multiple
concurrent background tasks (`vector.oauth_sync.user_manager_task`,
processors, etc.) alongside the request-path code. Symptom in the
pod logs:
RuntimeError: Task <Task pending name='nextcloud_mcp_server.vector.oauth_sync.user_manager_task'>
got Future <Future pending cb=[BaseProtocol._on_waiter_completed()]>
attached to a different loop
followed seconds later by
RuntimeError: Event loop is closed
while SQLAlchemy's pool tries to clean up the failed connection.
The event loop becomes increasingly unresponsive as asyncpg
protocol Futures pile up holding references to closed loops; the
`/health/live` endpoint eventually misses its probe window and
the kubelet SIGKILLs the pod (exitCode 137), restart-looping the
backend.
Root cause: the engine was built with the default
`AsyncAdaptedQueuePool` (`pool_size=2, max_overflow=5`) and
`pool_pre_ping=True`. asyncpg connection objects are bound to the
event loop they were created on. When the process holds a
singleton engine and tasks running under different anyio
TaskGroups check out connections from that pool, the pre-ping
probe runs on a cached connection whose underlying transport
references a different loop's selector → cross-loop access →
crash.
Switch to `NullPool` — one fresh asyncpg connection per
`engine.connect()`, no caching, no cross-loop bookkeeping to get
wrong. asyncpg connection setup is ~5 ms over LAN and a single
round-trip in the local-Postgres case, so the throughput cost is
negligible for the MCP server's traffic shape (low concurrency,
bursty per-user requests). This matches what the SQLite branch
already does (see `initialize()`) and what Alembic's `env.py`
uses for migrations, so the codebase is now consistent across
all backends.
`DATABASE_POOL_SIZE` / `DATABASE_MAX_OVERFLOW` config knobs are
preserved for backward compatibility but no longer affect the
Postgres engine. The validators in `config.py` continue to
reject values < 1 / < 0, so misconfigured deploys still fail
loudly. A follow-up could mark them deprecated in
`docs/configuration.md`; out of scope here.
Discovered while smoke-testing the per-tenant Postgres flow in
Astrolabe Cloud (every-tenant pod fresh-provisions a database
via the ADR-026 backend → hits this crashloop within ~5 min of
the first MCP-routed request).
Refs:
- ADR-026 § "Concurrency model and pool sizing" (the original
QueuePool rationale, now superseded by this finding)
- nextcloud_mcp_server/alembic/env.py (NullPool for migrations)
- SQLAlchemy docs: NullPool is the documented choice when
connection objects don't survive across the lifecycle of the
pool's logical "owner" (here: the event loop)
Verified:
- `uv run ruff check nextcloud_mcp_server/auth/storage.py` clean
- `uv run pytest tests/unit/test_*storage*.py` → 29 passed
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses all 8 items in the round-4 bot review plus 4 remaining
SonarQube OPEN issues that were silently broken by round 3's
malformed NOSONAR markers.
NOSONAR syntax fix (clears the remaining 4 OPEN SQ issues)
----------------------------------------------------------
Round 3 used ``# NOSONAR S<rule_key>`` form. SonarQube Python doesn't
recognize the rule-key suffix — it treats the whole thing as a
malformed suppression directive (S7632) AND lets the underlying rule
keep firing (S7503 on ``_Cursor.__aenter__/__aexit__``).
Switch every marker to bare ``# NOSONAR``, with the rationale moved
into a preceding comment block. Affected sites:
- storage.py: ``_Cursor.__aenter__``, ``_Cursor.__aexit__``
- config.py: ``get_database_ssl()`` ``return False`` + ``ssl.create_default_context()``
- test_storage_logging.py: ``SENTINEL_PASSWORD_FRAGMENT`` constant
- test_storage_postgres.py: three ``bob_pw_v1`` / ``bob_pw_v2`` / ``carol_pw`` literals
Bot 🔴#1 — defensive NOSONAR on get_database_ssl `return False`
--------------------------------------------------------------
Bot predicted S4830 fires on the operator-opt-out path. SQ output
shows it doesn't currently fire, but bare NOSONAR added defensively
with rationale comment.
Bot 🔴#2 — defensive NOSONAR on f-string SQL
--------------------------------------------
``update_oauth_session`` builds its SET clause via ``f"{', '.join(update_fields)}"``;
``get_audit_logs`` builds its WHERE clause via string concatenation.
Both are safe (the fragments only come from this function's own
branches, no user input), but the patterns trip taint analysers.
Annotated both with bare NOSONAR + safety comment explaining the
hardcoded-fragments invariant. Note: S2077 doesn't currently fire
on these; defensive.
Bot 🟡#3 — pg_advisory_lock for concurrent migrations
-----------------------------------------------------
Without coordination, two pods rolling-updating simultaneously can
both observe ``has_alembic=False`` and both try to apply migrations
from scratch — the second crashes with "relation already exists".
New ``_migration_lock()`` async context manager:
- On Postgres: ``SELECT pg_advisory_lock(:lock_id)`` on a fresh
connection (separate from the engine pool so it survives the
``to_thread.run_sync`` worker), held across BOTH the schema-inspect
AND the migration call. Without that span, two pods could each
observe "no alembic_version" before either started migrating,
defeating the lock.
- On SQLite: yields immediately (file-level locking serializes
writes natively).
Lock ID derived from
``sha256(b"nextcloud-mcp-server:migrations")[:8]`` as a stable signed
int64 so we can't collide with other apps sharing the same Postgres.
Bot 🟡#4 — RefreshTokenStorage.close() + lifespan wiring
--------------------------------------------------------
New idempotent ``close()`` method calls ``await engine.dispose()``,
nulls the engine, resets ``_initialized``. Wired into both
``app_lifespan_basic`` (BasicAuth) and the OAuth lifespan teardown,
each wrapped in ``try/except Exception`` with ``logger.warning`` so a
buggy dispose can't block SIGTERM. Without this, pooled asyncpg
connections leak server-side slots until
``idle_in_transaction_session_timeout`` reaps them — with small pool
defaults and frequent k8s rolling restarts this can starve
``max_connections``.
Bot 🟢#5 — is_sqlite_url docstring on :memory:
----------------------------------------------
Updated docstring to note both file-backed and in-memory forms are
recognized; caller is responsible for ``:memory:`` magic.
Bot 🟢#6 — db_path via make_url(...).database
---------------------------------------------
Replaced ``database_url.split("///", 1)[1]`` hack with SQLAlchemy's
own URL parsing. Naturally handles in-memory (``.database is None``
→ falls back to ``""``). Same lazy-import pattern as the existing
``mask_db_password`` to avoid module-import-time cost.
Bot 🟢#7 — _to_sync_url unrecognized-driver guard
-------------------------------------------------
Pulled ``_KNOWN_ASYNC_DRIVERS = ("aiosqlite", "asyncpg")`` into a
module constant. When an unrecognized ``+<driver>`` token survives
the strip, emits ``logger.warning`` with the known-supported list.
Behavior unchanged for valid URLs.
Bot 🟢#8 — get_audit_logs SELECT * → explicit columns
-----------------------------------------------------
Replaced ``SELECT *`` with explicit column list. Future schema
additions stay out of the dict return.
New tests
---------
- ``test_close_disposes_engine``: pins the public contract — engine
nulled, state reset, second call is a no-op.
- ``test_concurrent_initialize_serialized_by_advisory_lock``: spawns
3 concurrent inits against a fresh schema; asserts no "relation
already exists" and exactly one ``alembic_version`` row at the end.
Without the lock, this reliably fails on the second concurrent
task.
Docs
----
- ADR-026: new "Concurrent migrations across pods" subsection
documents the advisory-lock approach + lock-ID derivation.
Verification
------------
- ``uv run pytest tests/unit/`` — 1025 passed.
- ``TEST_DATABASE_URL=… uv run pytest tests/integration/test_storage_postgres.py -m postgres`` — 9 passed (was 7).
- ``ruff check && ruff format --check && ty check`` — clean.
Expected post-push: SQ scan reports 0 OPEN issues (was 4).
Tracked on Astrolabe Cloud POC board, card #99.
---
_This PR was generated with the help of AI, and reviewed by a Human_
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Round-3 fixes. Two threads:
- 9 OPEN SonarQube issues caused the "E Security Rating on New Code"
gate failure. The bot's diagnosis (sa.text(text_sql) → SQL injection)
was a wrong guess; the actual SQ rules firing were different.
- Bot's substantive concerns: pool defaults too aggressive,
delete_browser_session RETURNING path untested on Postgres,
schema_version legacy table created on Postgres, stale module
docstring.
- User's underlying question on the pool: "isn't 1 connection enough?"
Right-sized to 2+5 and documented the concurrency model in ADR-026
so the rationale is durable.
SonarQube quality-gate fixes (clears all 9 OPEN issues)
-------------------------------------------------------
- BLOCKER S6418: rename `SECRET` constant in test_storage_logging.py
to `SENTINEL_PASSWORD_FRAGMENT` + NOSONAR with rationale.
- CRITICAL S3776: extract `_build_postgres_engine()` from
`initialize()` (was complexity 26 > 15); incidentally creates a
clean unit-test seam for engine args.
- CRITICAL S4423: `ssl.create_default_context(cafile=...)` is flagged
as "weak protocol" — Python 3.10+ already negotiates the strongest
available protocol. Explicitly pass `purpose=ssl.Purpose.SERVER_AUTH`
and NOSONAR with the Python-version rationale.
- MAJOR S3358: split the TLS-mode nested ternary in the engine
factory into a `_describe_ssl_arg()` helper.
- MAJOR S2068 ×3: bind test app-password literals to local vars and
put `# NOSONAR S2068` on the same line as the literal (anchoring
requirement) instead of on the closing paren.
- MINOR S7503 ×2: `# NOSONAR S7503` on `_Cursor.__aenter__/__aexit__`
— they MUST be `async` per the context-manager protocol.
Pool sizing right-sized (answers "why so many connections?")
------------------------------------------------------------
- `DATABASE_POOL_SIZE` default 10 → **2**.
- `DATABASE_MAX_OVERFLOW` default 20 → **5**.
- Per-pod max drops from 30 to 7. With 3 replicas, total = 21
connections (was 90) — well under managed-Postgres
`max_connections=100`.
- New INFO log at startup: `Postgres engine ready: pool_size=N
max_overflow=M (per-pod max K connections)`. Surfaces the active
sizing without grepping config.
- New ADR-026 § "Concurrency model and pool sizing" explains
asyncpg's single-flight connection semantics, the MCP workload
shape (read-mostly point lookups), why-not-1 (multi-user
serialization), and the tune-up/tune-down recipe.
- `docs/configuration.md` table updated with new defaults +
homelab-vs-prod tuning guidance, linking the ADR.
RETURNING path covered on Postgres
----------------------------------
- New `test_browser_session_delete_returning` exercises the
`DELETE … RETURNING user_id` path — the only RETURNING clause in
the storage layer and the most dialect-sensitive SQL in this PR.
Asserts both present-row (returns True, row gone) and absent-row
(returns False) branches.
Schema portability polish
-------------------------
- `alembic 001`: gate `schema_version` table creation on
`op.get_bind().dialect.name == "sqlite"`. The table exists purely
to match the fingerprint of pre-Alembic SQLite databases; fresh
Postgres installs no longer carry the dead legacy table.
Misc polish
-----------
- Module docstring: "SQLite-based" → "SQL-backed", with a sentence
on the DATABASE_URL opt-in and an ADR-026 link.
- Comment on `_wrap_row` noting `row._mapping` is the documented
RowMapping accessor in SQLAlchemy 2.x despite the underscore.
Skipped (rationale in PR reply)
-------------------------------
- `_qmark_to_named` SQL-comment handling: docstring already notes
the limitation; no `?` in storage SQL comments today.
- Module-level `anyio.Lock()`: established precedent confirmed by
the bot itself.
- `get_audit_logs` `SELECT *`: pre-existing pattern, out of scope.
Verification
------------
- `uv run pytest tests/unit/` — 1025 passed.
- `TEST_DATABASE_URL=… uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 7 passed.
- `ruff check && ruff format --check && ty check` — clean.
- Confirmed `schema_version` absent on fresh Postgres, still present
on fresh SQLite.
Tracked on Astrolabe Cloud POC board, card #99.
---
_This PR was generated with the help of AI, and reviewed by a Human_
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Round-2 fixes after the bot review on PR #798 plus two user follow-ups
(self-signed Postgres support; asyncpg should be a PyPI extra). Folded
into the same PR rather than a follow-up since the work is still
unmerged.
Security
--------
- Mask database credentials in all 5 log call sites (storage.py × 4,
migrations.py × 1) via a new `mask_db_password()` helper in config.py.
Uses SQLAlchemy's `make_url(...).render_as_string(hide_password=True)`
with a regex fallback so the masking path never raises.
- New `tests/unit/test_storage_logging.py` asserts a sentinel password
never appears in `caplog` during `RefreshTokenStorage.initialize()`.
Distribution
------------
- `asyncpg` moved to `[project.optional-dependencies] postgres` so a
vanilla `pip install nextcloud-mcp-server` no longer pulls in the
~5 MB C extension. The Docker image runs `uv sync --extra postgres`,
so containerized deployments are unchanged.
- When `DATABASE_URL=postgresql+asyncpg://...` is set on a venv missing
the extra, `RefreshTokenStorage.initialize()` raises a friendly
RuntimeError pointing at `[postgres]` rather than the generic
ModuleNotFoundError.
TLS for the Postgres backend
----------------------------
- New `DATABASE_VERIFY_SSL` + `DATABASE_CA_BUNDLE` env vars mirror the
existing `NEXTCLOUD_VERIFY_SSL` / `NEXTCLOUD_CA_BUNDLE` pattern
(validators in Settings.__post_init__, `get_database_ssl()` helper
alongside `get_nextcloud_ssl_verify()`). `DATABASE_VERIFY_SSL=false`
wins over `DATABASE_CA_BUNDLE` for incident-response convenience.
- Default is **None** rather than True — keeps PR #798's behavior
intact for cluster-internal Postgres that runs without TLS. Operators
opt into verify-full or supply a private CA. ADR-026 records the
reasoning vs the Nextcloud HTTPS default.
- Engine factory in `storage.py` passes `ssl` via `connect_args` only
when `get_database_ssl()` returns non-None; otherwise asyncpg's
default (`prefer`) applies.
- Storage logs which TLS mode is active at INFO (no secret material).
Configurable connection pool
----------------------------
- `DATABASE_POOL_SIZE` (default 10) and `DATABASE_MAX_OVERFLOW`
(default 20) replace the hardcoded engine values. With many replicas
this can blow past managed-Postgres `max_connections=100`; tune down
for large fleets.
- gte-1 / gte-0 validators in __post_init__ reject 0/negative pool
sizes at startup with the offending value in the error.
Consistency polish
------------------
- Migration 006: convert raw `op.execute("ALTER TABLE ... ADD COLUMN")`
to `op.batch_alter_table(...).add_column(sa.Column("nonce", sa.Text))`
for stylistic consistency with the rewritten 001-005. Downgrade now
drops the column instead of being a no-op.
- `registered_webhooks.created_at` standardized from `sa.Float` to
`sa.BigInteger` (all other `*_at` columns); `store_webhook()` casts
`time.time()` → `int`.
- `is_sqlite_url()` made case-insensitive.
Testing
-------
- New `tests/integration/test_storage_postgres.py::test_cleanup_expired_roundtrip`
exercises `cleanup_expired_tokens`, `cleanup_expired_sessions`, and
`cleanup_expired_browser_sessions` — relies on DELETE rowcount,
historically dialect-tricky.
- `tests/unit/test_ssl_config.py` extended with `TestDatabaseSSLSettings`
+ `TestGetDatabaseSSL` classes (9 new tests) mirroring the existing
Nextcloud SSL tests one-for-one.
Docs
----
- `docs/configuration.md` Centralized-Storage section grew the four new
env vars + a homelab example with a private CA.
- `docs/ADR-026` grew Distribution, TLS, and `alembic/env.py` async-pattern
subsections explaining the non-obvious design choices.
Helm chart counterpart in cbcoutinho/helm-charts PR #34 (separate
commit on `feat/nextcloud-mcp-server-database-url`).
Verification
------------
- `uv run pytest tests/unit/` — 1025 passed.
- `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 6 passed (including new cleanup test).
- `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean.
Tracked on Astrolabe Cloud POC board, card #99.
---
_This PR was generated with the help of AI, and reviewed by a Human_
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a `DATABASE_URL` setting that lets `RefreshTokenStorage` run against
any SQLAlchemy async backend, primarily `postgresql+asyncpg://...` for
HA k8s deployments. Default behavior is unchanged: when `DATABASE_URL` is
unset the server falls back to the existing `TOKEN_STORAGE_DB` path /
ephemeral SQLite tempfile.
Why
---
Today every MCP pod needs its own PVC to hold the SQLite file, which
pins the Deployment to one replica and blocks horizontal scaling. With
this change, operators can point all replicas at a shared Postgres
(CNPG, RDS, etc.) and the pods become stateless. Encryption stays in
Python (Fernet); the database only sees ciphertext.
What changed
------------
- `config.get_database_url()` resolves DATABASE_URL → TOKEN_STORAGE_DB →
ephemeral tempfile in that priority order.
- `RefreshTokenStorage` builds a process-shared `AsyncEngine` in
`initialize()`. SQLite gets NullPool; Postgres gets pool_size=10,
max_overflow=20, pool_pre_ping=True. 30 aiosqlite call sites adapted
via a thin `_DBConn` / `_Cursor` / `_Row` / `_ExecuteCtx` shim so
existing method bodies need no churn beyond the connection
context-manager swap.
- 7 `INSERT OR REPLACE` statements rewritten as portable
`INSERT ... ON CONFLICT (...) DO UPDATE` (SQLite ≥ 3.24, Postgres ≥ 9.5).
- `sqlite_master` legacy-detection lookup replaced with SQLAlchemy
inspector so the path works against either backend.
- File-permission hardening + parent-dir creation gated on
`is_sqlite_url(...)` — centralized backends manage their own filesystem.
- Alembic migrations 001/002/003/005 converted from raw `op.execute(SQL)`
to portable `op.create_table()` / `op.create_index()` with SQLAlchemy
types. All timestamp columns are `sa.BigInteger` so Postgres allocates
BIGINT (unix epochs don't fit in INT4). SQLite treats BIGINT as
INTEGER, so existing deployments at revision 006 see no schema drift.
- `migrations.py` + CLI take URLs; `db {upgrade,downgrade,current,history}`
gain `--database-url / -u` alongside the legacy `--database-path / -d`.
`get_current_revision()` uses SQLAlchemy inspector instead of raw
sqlite3, so the CLI works against Postgres too.
- `docker-compose.yml` adds a `postgres-test` service under the
`postgres` profile (pinned `postgres:16-alpine` digest) for
integration testing.
- Unit storage tests parametrized over backends via shared
`tests/fixtures/storage_backend.py` — every test in
`test_app_password_storage.py` and `test_webhook_storage.py` runs
once per backend that is available. Postgres is opted in by
`TEST_DATABASE_URL`.
- New `tests/integration/test_storage_postgres.py` (5 tests, marked
`postgres` + `integration`) covers refresh-token, app-password,
OAuth-session, webhook, and audit-log paths end-to-end on Postgres.
- New `docs/ADR-026-pluggable-database-backend.md` records the decision;
`docs/configuration.md` documents `DATABASE_URL` with examples.
Out of scope
------------
- No SQLite → Postgres data migration tool (clean cutover; tokens reissue
on next login, webhooks re-register on next sync tick).
- This repo does not provision Postgres. The matching helm chart change
lives in cbcoutinho/helm-charts (database.url / existingSecret values).
Verification
------------
- `uv run pytest tests/unit/` — 1012 passed, SQLite path unchanged.
- `docker compose --profile postgres up -d postgres-test`
- `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres -v`
— 5 passed.
- `TEST_DATABASE_URL=... uv run pytest tests/unit/test_app_password_storage.py
tests/unit/test_webhook_storage.py` — 50 passed (25 per backend).
- `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean.
Tracked on Astrolabe Cloud POC board, card #99.
---
_This PR was generated with the help of AI, and reviewed by a Human_
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`_get_background_operations_enabled()` was emitting three advisory log
lines (1 INFO + 2 deprecation WARNINGs) on every call. Because
`get_settings()` is intentionally non-cached and runs on every MCP tool
invocation via `get_client()`, the "Automatically enabled background
operations for semantic search in multi-user mode" INFO line was
firing per-request — 569 entries/hour in one production tenant.
Gate the three log emissions behind a module-level
`_bg_ops_advisories_logged` flag, mirroring the existing
`_warn_missing_secret_once` precedent in
`vector/webhook_receiver.py`. The boolean-derivation path stays
unchanged, so the `Settings` value remains fresh per call.
Extends the autouse `_reload_dynaconf_after_test` fixture to reset the
new flag between tests, and adds two regression tests that call
`get_settings()` five times and assert each advisory fires exactly once.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to #787/#789 (ADR-022 cleanup). After
`oauth_enabled ↔ enable_login_flow` became an invariant, the
`use_basic_auth=False` branch in `vector/oauth_sync.py` — and the
parameter wiring that fed it — was no longer reachable from any
supported deployment mode. This commit removes the dead code.
- nextcloud_mcp_server/vector/oauth_sync.py:
- Deleted `get_user_client_oauth` (the OAuth-token refresh helper) and
its `VECTOR_SYNC_SCOPES` constant.
- Deleted the `get_user_client` dispatcher. Internal callers now call
`get_user_client_basic_auth` directly.
- Dropped the `use_basic_auth: bool` parameter from `user_scanner_task`,
`multi_user_processor_task`, `_run_user_scanner_with_scope`, and
`user_manager_task`.
- Dropped the `token_broker` parameter from the same four functions —
they no longer need it now that the OAuth-refresh path is gone. The
`TokenBrokerService` constructed in `app.py` is still used by the
management API revoke endpoint, just not by background sync.
- Simplified the user-list query in `user_manager_task` to always read
from the `app_passwords` table.
- Replaced all `mode_label = "BasicAuth" if use_basic_auth else "OAuth"`
with a literal `[BasicAuth]` log prefix (keeps existing log filters
working).
- Updated the module docstring to describe the post-cleanup shape.
- Dropped the now-unused `TYPE_CHECKING` import of `TokenBrokerService`.
- nextcloud_mcp_server/app.py: dropped the `use_basic_auth = True` block
and the now-stale `token_broker if not use_basic_auth else None` /
`use_basic_auth` positional args from the two `tg.start(...)` calls in
the multi-user vector-sync lifespan. Token broker construction stays —
still consumed by the management API revoke endpoint via
`app.state.oauth_context["token_broker"]`.
- tests/integration/test_app_password_provisioning.py: deleted four tests
that exercised the now-removed OAuth-refresh path
(`test_oauth_mode_uses_refresh_token_only`,
`test_oauth_mode_raises_error_without_token`,
`test_get_user_client_oauth_function`,
`test_oauth_mode_requires_token_broker`) plus the
`test_get_user_client_dispatches_to_basic_auth` test for the deleted
dispatcher. Updated the module docstring + imports accordingly. The
BasicAuth-mode tests (`test_basic_auth_mode_uses_local_storage`,
`test_multiple_users_basic_auth_mode`, etc.) all remain.
No runtime-behaviour change in any supported deployment mode — the deleted
branches were already unreachable post-PR #787. 3 files changed,
+59 / -301; 1010 unit tests pass; integration jobs for
`mcp-login-flow` and `mcp-multi-user-basic` are the critical regression
gates before merge.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
