6cff2df008337f5e790fe75bd3ddc2ca09aac3ae
588
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris CoutinhoandGitHub
|
baf4d4d225 |
Merge pull request #809 from cbcoutinho/feat/deck-webhook-presets
feat(webhooks): add Deck card sync preset with vector indexing |
||
|
Chris CoutinhoandClaude Opus 4.7
|
c03223a41c |
fix(vector-sync): use resolved collection name in orphan sweep
The startup sweep was reading settings.qdrant_collection (the raw config
value, default "nextcloud_content") instead of settings.get_collection_name(),
which is what every other vector-sync operation uses. When QDRANT_COLLECTION
is not overridden, get_collection_name() auto-generates a
{deployment-id}-{model-name} name; the sweep was targeting a non-existent
collection and silently returning (0, 0).
Also adds the AsyncQdrantClient type annotation that was missing on
sweep_orphan_placeholders, and renames its parameter from collection_name
to collection to make it clear the value must be the resolved name.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
a5cbe91b29 |
fix(vector-sync): sweep placeholder orphans at Pod startup (#101)
When the per-tenant nextcloud-mcp-server Pod OOMKills mid-batch, the in-memory anyio processor queue is lost but the placeholder Qdrant points (is_placeholder=true, status=pending) survive. The next Pod's scanner re-runs, sees the existing placeholders, applies the 5 × VECTOR_SYNC_SCAN_INTERVAL staleness gate (~5h with the deployed 1h scan interval), and skips them. Result: 0 documents indexed for the duration of the gate after every restart. Stamps a process-level instance_id (UUID per Pod-process) onto every placeholder write. A new sweep_orphan_placeholders helper, called once from starlette_lifespan after the Qdrant client is initialised and before the scanner / user-manager spawns, scrolls the collection and deletes any placeholder whose instance_id doesn't match the current Pod's (including placeholders with no instance_id field — back-compat for pre-fix Pod versions). The scanner's next cycle naturally re-creates fresh placeholders and queues work normally; no DocumentTask reconstruction needed. Sweep is one-shot at startup, not periodic — the existing staleness gate still covers same-Pod recovery, and the cross-Pod-restart gap was the only failure mode. Failure is non-fatal (logged via vector_sync.orphan_sweep_failed) so a transient Qdrant hiccup at boot doesn't prevent the scanner from running. Both lifespan branches (single-user BasicAuth, OAuth / multi-user BasicAuth) call the sweep via a module-local helper. A new VECTOR_SYNC_ORPHAN_SWEEP_ENABLED setting (default True) provides an escape hatch. Closes Deck #101. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
688a03f00a |
fix(contacts): surface ORG/TITLE/NOTE/URL/CATEGORIES/PHOTO on read (refs #716)
PR #719 fixed the contact-create path so all documented fields persist to the vCard, but the read path (list/search via MCP) still returned ``organization: null`` / ``note: null`` / ``title: null`` because pythonvCard4 has no typed parser for ORG/TITLE — they land in ``Contact.custom`` — and the server-side mapper never read ``note``/``urls``/``categories``/``photo`` even when present. Reads now surface what the write side persisted: - ``client/contacts.py``: new ``_first_custom`` helper pulls raw values from ``Contact.custom`` for ORG/TITLE/unencoded PHOTO. ``list_contacts`` extends its per-contact dict with org/title/note/url/categories/photo. - ``server/contacts.py``: ``_raw_contact_to_model`` maps the new keys onto ``Contact.organization`` / ``.title`` / ``.note`` / ``.urls`` / ``.categories`` / ``.photo``. URL accepts both list and plain-string shapes; categories accepts comma-separated strings for forward-compat. Coverage: - Unit: ``TestFirstCustom`` (five cases incl. bare-string library shape) and three new ``_raw_contact_to_model`` cases covering the full field set, plain-string URL, and comma-string categories. - Integration: ``test_mcp_contacts_workflow`` now decodes the ``nc_contacts_search_contacts`` response and asserts ``organization`` / ``note`` round-trip — direct regression coverage for elvisdragonmao's report on issue #716. Verified end-to-end against the local single-user docker stack: creating a contact with ``{organization, title, note, url, categories}`` and reading it back via ``nc_contacts_search_contacts`` returns every field populated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
7da72a3888 |
feat(api): log inbound User-Agent on management API and webhook receiver
Astrolabe (and any other PHP-side client) sends a stable User-Agent on
every outbound call to the MCP server. Capture it at the middleware
layer so backend access logs can attribute each request to a specific
client build — e.g. ``Nextcloud-Astrolabe/0.14.1``.
The middleware fires only for /api/v1/* and /webhooks/nextcloud,
which is the surface PHP-side clients hit; /mcp and /health stay
silent. The structured ``extra`` ({user_agent, http_method, http_path})
flows into OTel spans so the field is queryable in Grafana / Loki.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
fdcbd7bd3f |
feat(webhooks): add Deck card sync preset with vector indexing
Nextcloud Deck PR #7910 added IWebhookCompatibleEvent to CardCreated/ Updated/DeletedEvent and BoardUpdatedEvent, so Deck can finally emit real-time webhooks via core's webhook_listeners app. Wire this into the existing preset → parser → DocumentTask pipeline that already backs Notes / Calendar / Tables / Forms / Files sync. - Add deck_sync preset (app=deck, 4 events) and drop the stale "Deck does not support webhooks" comment. - Teach webhook_parser to convert Deck card events into DocumentTask(doc_type=deck_card, operation=index|delete) with stack_id metadata. BoardUpdatedEvent logs delivery at INFO and returns None — the polling scanner reconciles affected cards. - Cover three new unit tests for the deck create/delete/board-update paths plus symmetric fail-open tests for missing card.id / node.id in _parse_deck_event and _parse_file_event. The astrolabe admin UI auto-discovers the new preset via filter_presets_by_installed_apps(); no astrolabe-side wiring is required for it to appear in the Webhook Management card grid. Note: requires Deck ≥1.18.x (where PR #7910 lands); the preset is hidden when the Deck app isn't installed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandGitHub
|
1cf00d6be0 |
Merge pull request #719 from cbcoutinho/fix/contacts-create-dropped-fields-716
fix(contacts): persist all documented fields on create (fixes #716) |
||
|
Chris CoutinhoandClaude Opus 4.7
|
e0b7afb4b1 |
fix(embedding): instantiate BM25 singleton off the event loop
The ``BM25SparseEmbeddingProvider.__init__`` calls ``fastembed.SparseTextEmbedding(model_name="Qdrant/bm25")`` which downloads ~50 MB of model weights from HuggingFace and loads them into memory — observed >5 s wall-clock in production. The inference methods (``encode_async``, ``encode_batch_async``) already wrap work in ``anyio.to_thread.run_sync``, so the design intent is clearly to keep FastEmbed off the event loop. That protection just didn't cover the constructor. Symptom in the Astrolabe Cloud per-tenant deploy (deck #102 smoke): ~30–90 s after a user enables semantic search, the pod tips into a SIGKILL-restart cycle. Loki shows a single log line Initializing BM25 sparse embedding provider: Qdrant/bm25 followed by nothing else from the event loop until exitCode 137. Kubernetes ``/health/live`` httpGet probe timeout=5s fires 6 times in a row, kubelet kills the container, restart, repeat. Fix: switch ``get_bm25_service()`` to an async accessor that wraps the first-time construction in ``anyio.to_thread.run_sync``. Two existing call sites (``vector/processor.py:603``, ``search/bm25_hybrid.py:123``) update to ``await``. Both are already inside async functions so the await is free. New unit test pins the invariant by monkey-patching ``BM25SparseEmbeddingProvider.__init__`` with ``time.sleep(1)`` and asserting a concurrent ``anyio.sleep(0.05)`` finishes promptly — the test fails if the constructor ever runs back on the event loop. Same pattern exists in ``OllamaEmbeddingProvider.__init__`` (sync ``httpx.get`` health-check). Ollama isn't enabled in any current deploy; filed as a follow-up. Refs: - Astrolabe Cloud deck card #102 (smoke discovery) - Sibling fix #799 (NullPool for cross-loop-asyncpg, same class of "anyio bites you in production" bug) Verified: - ``uv run pytest tests/unit/`` — 1027 passed - ``uv run ruff check`` clean on touched files - ``uv run ty check`` clean on touched files - New tests pass Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
e98903c502 |
fix(storage): address review on PR #799 (stale comments, docs deprecation, unit test)
claude-review on #799 flagged: 1. Stale inline comment in ``initialize()`` (line 466) still said "Postgres uses a small bounded pool". Updated to reflect both backends now use NullPool. 2. Stale ``close()`` docstring referenced pool-size starving max_connections — irrelevant with NullPool. Replaced with the NullPool-aware rationale (dispose still tears down in-flight asyncpg connections cleanly). 3. ``docs/configuration.md`` actively directed operators to tune DATABASE_POOL_SIZE / DATABASE_MAX_OVERFLOW, with worked examples and pool math. Both are now deprecated no-ops; the table entries explain the deprecation and link to PR #799. Operators reading the docs will no longer be confused into tuning settings that don't do anything. 4. ``config.py`` comment for the deprecated fields updated to record the deprecation. Validators are intentionally kept (still reject < 1 / < 0) so misconfigured deploys fail loudly rather than silently — the reviewer flagged this as a minor UX wart but explicitly "not a blocker"; the docs change in (3) keeps operators away from the config altogether. 5. New ``tests/unit/test_storage_engine.py`` with three tests: - ``test_postgres_engine_uses_nullpool`` — pins ``isinstance( engine.pool, NullPool)`` so a refactor back to QueuePool / SingletonThreadPool can't silently re-introduce the cross- event-loop crashes. - ``test_postgres_engine_ignores_pool_sizing_settings`` — setting DATABASE_POOL_SIZE / DATABASE_MAX_OVERFLOW to huge values must not change pool type (proves the deprecated fields are wired-up no-ops). - ``test_postgres_engine_missing_asyncpg_driver_message`` — guards the existing actionable-error branch when the optional ``[postgres]`` extra isn't installed. Verified: - ``uv run pytest tests/unit/`` — 1028 passed - ``uv run ruff check`` clean on the touched python files Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
8cd3092e87 |
fix(storage): use NullPool for Postgres engine (cross-loop crashes under anyio TaskGroup)
The Postgres backend hit a hard crashloop in production deployments
where the MCP server runs under anyio TaskGroups with multiple
concurrent background tasks (`vector.oauth_sync.user_manager_task`,
processors, etc.) alongside the request-path code. Symptom in the
pod logs:
RuntimeError: Task <Task pending name='nextcloud_mcp_server.vector.oauth_sync.user_manager_task'>
got Future <Future pending cb=[BaseProtocol._on_waiter_completed()]>
attached to a different loop
followed seconds later by
RuntimeError: Event loop is closed
while SQLAlchemy's pool tries to clean up the failed connection.
The event loop becomes increasingly unresponsive as asyncpg
protocol Futures pile up holding references to closed loops; the
`/health/live` endpoint eventually misses its probe window and
the kubelet SIGKILLs the pod (exitCode 137), restart-looping the
backend.
Root cause: the engine was built with the default
`AsyncAdaptedQueuePool` (`pool_size=2, max_overflow=5`) and
`pool_pre_ping=True`. asyncpg connection objects are bound to the
event loop they were created on. When the process holds a
singleton engine and tasks running under different anyio
TaskGroups check out connections from that pool, the pre-ping
probe runs on a cached connection whose underlying transport
references a different loop's selector → cross-loop access →
crash.
Switch to `NullPool` — one fresh asyncpg connection per
`engine.connect()`, no caching, no cross-loop bookkeeping to get
wrong. asyncpg connection setup is ~5 ms over LAN and a single
round-trip in the local-Postgres case, so the throughput cost is
negligible for the MCP server's traffic shape (low concurrency,
bursty per-user requests). This matches what the SQLite branch
already does (see `initialize()`) and what Alembic's `env.py`
uses for migrations, so the codebase is now consistent across
all backends.
`DATABASE_POOL_SIZE` / `DATABASE_MAX_OVERFLOW` config knobs are
preserved for backward compatibility but no longer affect the
Postgres engine. The validators in `config.py` continue to
reject values < 1 / < 0, so misconfigured deploys still fail
loudly. A follow-up could mark them deprecated in
`docs/configuration.md`; out of scope here.
Discovered while smoke-testing the per-tenant Postgres flow in
Astrolabe Cloud (every-tenant pod fresh-provisions a database
via the ADR-026 backend → hits this crashloop within ~5 min of
the first MCP-routed request).
Refs:
- ADR-026 § "Concurrency model and pool sizing" (the original
QueuePool rationale, now superseded by this finding)
- nextcloud_mcp_server/alembic/env.py (NullPool for migrations)
- SQLAlchemy docs: NullPool is the documented choice when
connection objects don't survive across the lifecycle of the
pool's logical "owner" (here: the event loop)
Verified:
- `uv run ruff check nextcloud_mcp_server/auth/storage.py` clean
- `uv run pytest tests/unit/test_*storage*.py` → 29 passed
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
d717c64750 |
fix(storage): address PR #798 round-4 review (NOSONAR syntax + pg_advisory_lock + engine dispose + nits)
Addresses all 8 items in the round-4 bot review plus 4 remaining SonarQube OPEN issues that were silently broken by round 3's malformed NOSONAR markers. NOSONAR syntax fix (clears the remaining 4 OPEN SQ issues) ---------------------------------------------------------- Round 3 used ``# NOSONAR S<rule_key>`` form. SonarQube Python doesn't recognize the rule-key suffix — it treats the whole thing as a malformed suppression directive (S7632) AND lets the underlying rule keep firing (S7503 on ``_Cursor.__aenter__/__aexit__``). Switch every marker to bare ``# NOSONAR``, with the rationale moved into a preceding comment block. Affected sites: - storage.py: ``_Cursor.__aenter__``, ``_Cursor.__aexit__`` - config.py: ``get_database_ssl()`` ``return False`` + ``ssl.create_default_context()`` - test_storage_logging.py: ``SENTINEL_PASSWORD_FRAGMENT`` constant - test_storage_postgres.py: three ``bob_pw_v1`` / ``bob_pw_v2`` / ``carol_pw`` literals Bot 🔴#1 — defensive NOSONAR on get_database_ssl `return False` -------------------------------------------------------------- Bot predicted S4830 fires on the operator-opt-out path. SQ output shows it doesn't currently fire, but bare NOSONAR added defensively with rationale comment. Bot 🔴#2 — defensive NOSONAR on f-string SQL -------------------------------------------- ``update_oauth_session`` builds its SET clause via ``f"{', '.join(update_fields)}"``; ``get_audit_logs`` builds its WHERE clause via string concatenation. Both are safe (the fragments only come from this function's own branches, no user input), but the patterns trip taint analysers. Annotated both with bare NOSONAR + safety comment explaining the hardcoded-fragments invariant. Note: S2077 doesn't currently fire on these; defensive. Bot 🟡#3 — pg_advisory_lock for concurrent migrations ----------------------------------------------------- Without coordination, two pods rolling-updating simultaneously can both observe ``has_alembic=False`` and both try to apply migrations from scratch — the second crashes with "relation already exists". New ``_migration_lock()`` async context manager: - On Postgres: ``SELECT pg_advisory_lock(:lock_id)`` on a fresh connection (separate from the engine pool so it survives the ``to_thread.run_sync`` worker), held across BOTH the schema-inspect AND the migration call. Without that span, two pods could each observe "no alembic_version" before either started migrating, defeating the lock. - On SQLite: yields immediately (file-level locking serializes writes natively). Lock ID derived from ``sha256(b"nextcloud-mcp-server:migrations")[:8]`` as a stable signed int64 so we can't collide with other apps sharing the same Postgres. Bot 🟡#4 — RefreshTokenStorage.close() + lifespan wiring -------------------------------------------------------- New idempotent ``close()`` method calls ``await engine.dispose()``, nulls the engine, resets ``_initialized``. Wired into both ``app_lifespan_basic`` (BasicAuth) and the OAuth lifespan teardown, each wrapped in ``try/except Exception`` with ``logger.warning`` so a buggy dispose can't block SIGTERM. Without this, pooled asyncpg connections leak server-side slots until ``idle_in_transaction_session_timeout`` reaps them — with small pool defaults and frequent k8s rolling restarts this can starve ``max_connections``. Bot 🟢#5 — is_sqlite_url docstring on :memory: ---------------------------------------------- Updated docstring to note both file-backed and in-memory forms are recognized; caller is responsible for ``:memory:`` magic. Bot 🟢#6 — db_path via make_url(...).database --------------------------------------------- Replaced ``database_url.split("///", 1)[1]`` hack with SQLAlchemy's own URL parsing. Naturally handles in-memory (``.database is None`` → falls back to ``""``). Same lazy-import pattern as the existing ``mask_db_password`` to avoid module-import-time cost. Bot 🟢#7 — _to_sync_url unrecognized-driver guard ------------------------------------------------- Pulled ``_KNOWN_ASYNC_DRIVERS = ("aiosqlite", "asyncpg")`` into a module constant. When an unrecognized ``+<driver>`` token survives the strip, emits ``logger.warning`` with the known-supported list. Behavior unchanged for valid URLs. Bot 🟢#8 — get_audit_logs SELECT * → explicit columns ----------------------------------------------------- Replaced ``SELECT *`` with explicit column list. Future schema additions stay out of the dict return. New tests --------- - ``test_close_disposes_engine``: pins the public contract — engine nulled, state reset, second call is a no-op. - ``test_concurrent_initialize_serialized_by_advisory_lock``: spawns 3 concurrent inits against a fresh schema; asserts no "relation already exists" and exactly one ``alembic_version`` row at the end. Without the lock, this reliably fails on the second concurrent task. Docs ---- - ADR-026: new "Concurrent migrations across pods" subsection documents the advisory-lock approach + lock-ID derivation. Verification ------------ - ``uv run pytest tests/unit/`` — 1025 passed. - ``TEST_DATABASE_URL=… uv run pytest tests/integration/test_storage_postgres.py -m postgres`` — 9 passed (was 7). - ``ruff check && ruff format --check && ty check`` — clean. Expected post-push: SQ scan reports 0 OPEN issues (was 4). Tracked on Astrolabe Cloud POC board, card #99. --- _This PR was generated with the help of AI, and reviewed by a Human_ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
51419329b0 |
fix(storage): address PR #798 round-3 review (SonarQube + pool sizing + RETURNING test)
Round-3 fixes. Two threads: - 9 OPEN SonarQube issues caused the "E Security Rating on New Code" gate failure. The bot's diagnosis (sa.text(text_sql) → SQL injection) was a wrong guess; the actual SQ rules firing were different. - Bot's substantive concerns: pool defaults too aggressive, delete_browser_session RETURNING path untested on Postgres, schema_version legacy table created on Postgres, stale module docstring. - User's underlying question on the pool: "isn't 1 connection enough?" Right-sized to 2+5 and documented the concurrency model in ADR-026 so the rationale is durable. SonarQube quality-gate fixes (clears all 9 OPEN issues) ------------------------------------------------------- - BLOCKER S6418: rename `SECRET` constant in test_storage_logging.py to `SENTINEL_PASSWORD_FRAGMENT` + NOSONAR with rationale. - CRITICAL S3776: extract `_build_postgres_engine()` from `initialize()` (was complexity 26 > 15); incidentally creates a clean unit-test seam for engine args. - CRITICAL S4423: `ssl.create_default_context(cafile=...)` is flagged as "weak protocol" — Python 3.10+ already negotiates the strongest available protocol. Explicitly pass `purpose=ssl.Purpose.SERVER_AUTH` and NOSONAR with the Python-version rationale. - MAJOR S3358: split the TLS-mode nested ternary in the engine factory into a `_describe_ssl_arg()` helper. - MAJOR S2068 ×3: bind test app-password literals to local vars and put `# NOSONAR S2068` on the same line as the literal (anchoring requirement) instead of on the closing paren. - MINOR S7503 ×2: `# NOSONAR S7503` on `_Cursor.__aenter__/__aexit__` — they MUST be `async` per the context-manager protocol. Pool sizing right-sized (answers "why so many connections?") ------------------------------------------------------------ - `DATABASE_POOL_SIZE` default 10 → **2**. - `DATABASE_MAX_OVERFLOW` default 20 → **5**. - Per-pod max drops from 30 to 7. With 3 replicas, total = 21 connections (was 90) — well under managed-Postgres `max_connections=100`. - New INFO log at startup: `Postgres engine ready: pool_size=N max_overflow=M (per-pod max K connections)`. Surfaces the active sizing without grepping config. - New ADR-026 § "Concurrency model and pool sizing" explains asyncpg's single-flight connection semantics, the MCP workload shape (read-mostly point lookups), why-not-1 (multi-user serialization), and the tune-up/tune-down recipe. - `docs/configuration.md` table updated with new defaults + homelab-vs-prod tuning guidance, linking the ADR. RETURNING path covered on Postgres ---------------------------------- - New `test_browser_session_delete_returning` exercises the `DELETE … RETURNING user_id` path — the only RETURNING clause in the storage layer and the most dialect-sensitive SQL in this PR. Asserts both present-row (returns True, row gone) and absent-row (returns False) branches. Schema portability polish ------------------------- - `alembic 001`: gate `schema_version` table creation on `op.get_bind().dialect.name == "sqlite"`. The table exists purely to match the fingerprint of pre-Alembic SQLite databases; fresh Postgres installs no longer carry the dead legacy table. Misc polish ----------- - Module docstring: "SQLite-based" → "SQL-backed", with a sentence on the DATABASE_URL opt-in and an ADR-026 link. - Comment on `_wrap_row` noting `row._mapping` is the documented RowMapping accessor in SQLAlchemy 2.x despite the underscore. Skipped (rationale in PR reply) ------------------------------- - `_qmark_to_named` SQL-comment handling: docstring already notes the limitation; no `?` in storage SQL comments today. - Module-level `anyio.Lock()`: established precedent confirmed by the bot itself. - `get_audit_logs` `SELECT *`: pre-existing pattern, out of scope. Verification ------------ - `uv run pytest tests/unit/` — 1025 passed. - `TEST_DATABASE_URL=… uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 7 passed. - `ruff check && ruff format --check && ty check` — clean. - Confirmed `schema_version` absent on fresh Postgres, still present on fresh SQLite. Tracked on Astrolabe Cloud POC board, card #99. --- _This PR was generated with the help of AI, and reviewed by a Human_ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
f2b7bf132f |
fix(storage): address PR #798 review feedback (credentials, asyncpg extra, TLS, pool)
Round-2 fixes after the bot review on PR #798 plus two user follow-ups (self-signed Postgres support; asyncpg should be a PyPI extra). Folded into the same PR rather than a follow-up since the work is still unmerged. Security -------- - Mask database credentials in all 5 log call sites (storage.py × 4, migrations.py × 1) via a new `mask_db_password()` helper in config.py. Uses SQLAlchemy's `make_url(...).render_as_string(hide_password=True)` with a regex fallback so the masking path never raises. - New `tests/unit/test_storage_logging.py` asserts a sentinel password never appears in `caplog` during `RefreshTokenStorage.initialize()`. Distribution ------------ - `asyncpg` moved to `[project.optional-dependencies] postgres` so a vanilla `pip install nextcloud-mcp-server` no longer pulls in the ~5 MB C extension. The Docker image runs `uv sync --extra postgres`, so containerized deployments are unchanged. - When `DATABASE_URL=postgresql+asyncpg://...` is set on a venv missing the extra, `RefreshTokenStorage.initialize()` raises a friendly RuntimeError pointing at `[postgres]` rather than the generic ModuleNotFoundError. TLS for the Postgres backend ---------------------------- - New `DATABASE_VERIFY_SSL` + `DATABASE_CA_BUNDLE` env vars mirror the existing `NEXTCLOUD_VERIFY_SSL` / `NEXTCLOUD_CA_BUNDLE` pattern (validators in Settings.__post_init__, `get_database_ssl()` helper alongside `get_nextcloud_ssl_verify()`). `DATABASE_VERIFY_SSL=false` wins over `DATABASE_CA_BUNDLE` for incident-response convenience. - Default is **None** rather than True — keeps PR #798's behavior intact for cluster-internal Postgres that runs without TLS. Operators opt into verify-full or supply a private CA. ADR-026 records the reasoning vs the Nextcloud HTTPS default. - Engine factory in `storage.py` passes `ssl` via `connect_args` only when `get_database_ssl()` returns non-None; otherwise asyncpg's default (`prefer`) applies. - Storage logs which TLS mode is active at INFO (no secret material). Configurable connection pool ---------------------------- - `DATABASE_POOL_SIZE` (default 10) and `DATABASE_MAX_OVERFLOW` (default 20) replace the hardcoded engine values. With many replicas this can blow past managed-Postgres `max_connections=100`; tune down for large fleets. - gte-1 / gte-0 validators in __post_init__ reject 0/negative pool sizes at startup with the offending value in the error. Consistency polish ------------------ - Migration 006: convert raw `op.execute("ALTER TABLE ... ADD COLUMN")` to `op.batch_alter_table(...).add_column(sa.Column("nonce", sa.Text))` for stylistic consistency with the rewritten 001-005. Downgrade now drops the column instead of being a no-op. - `registered_webhooks.created_at` standardized from `sa.Float` to `sa.BigInteger` (all other `*_at` columns); `store_webhook()` casts `time.time()` → `int`. - `is_sqlite_url()` made case-insensitive. Testing ------- - New `tests/integration/test_storage_postgres.py::test_cleanup_expired_roundtrip` exercises `cleanup_expired_tokens`, `cleanup_expired_sessions`, and `cleanup_expired_browser_sessions` — relies on DELETE rowcount, historically dialect-tricky. - `tests/unit/test_ssl_config.py` extended with `TestDatabaseSSLSettings` + `TestGetDatabaseSSL` classes (9 new tests) mirroring the existing Nextcloud SSL tests one-for-one. Docs ---- - `docs/configuration.md` Centralized-Storage section grew the four new env vars + a homelab example with a private CA. - `docs/ADR-026` grew Distribution, TLS, and `alembic/env.py` async-pattern subsections explaining the non-obvious design choices. Helm chart counterpart in cbcoutinho/helm-charts PR #34 (separate commit on `feat/nextcloud-mcp-server-database-url`). Verification ------------ - `uv run pytest tests/unit/` — 1025 passed. - `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 6 passed (including new cleanup test). - `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean. Tracked on Astrolabe Cloud POC board, card #99. --- _This PR was generated with the help of AI, and reviewed by a Human_ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
292cbb3292 |
feat(storage): pluggable database backend via DATABASE_URL (ADR-026)
Adds a `DATABASE_URL` setting that lets `RefreshTokenStorage` run against
any SQLAlchemy async backend, primarily `postgresql+asyncpg://...` for
HA k8s deployments. Default behavior is unchanged: when `DATABASE_URL` is
unset the server falls back to the existing `TOKEN_STORAGE_DB` path /
ephemeral SQLite tempfile.
Why
---
Today every MCP pod needs its own PVC to hold the SQLite file, which
pins the Deployment to one replica and blocks horizontal scaling. With
this change, operators can point all replicas at a shared Postgres
(CNPG, RDS, etc.) and the pods become stateless. Encryption stays in
Python (Fernet); the database only sees ciphertext.
What changed
------------
- `config.get_database_url()` resolves DATABASE_URL → TOKEN_STORAGE_DB →
ephemeral tempfile in that priority order.
- `RefreshTokenStorage` builds a process-shared `AsyncEngine` in
`initialize()`. SQLite gets NullPool; Postgres gets pool_size=10,
max_overflow=20, pool_pre_ping=True. 30 aiosqlite call sites adapted
via a thin `_DBConn` / `_Cursor` / `_Row` / `_ExecuteCtx` shim so
existing method bodies need no churn beyond the connection
context-manager swap.
- 7 `INSERT OR REPLACE` statements rewritten as portable
`INSERT ... ON CONFLICT (...) DO UPDATE` (SQLite ≥ 3.24, Postgres ≥ 9.5).
- `sqlite_master` legacy-detection lookup replaced with SQLAlchemy
inspector so the path works against either backend.
- File-permission hardening + parent-dir creation gated on
`is_sqlite_url(...)` — centralized backends manage their own filesystem.
- Alembic migrations 001/002/003/005 converted from raw `op.execute(SQL)`
to portable `op.create_table()` / `op.create_index()` with SQLAlchemy
types. All timestamp columns are `sa.BigInteger` so Postgres allocates
BIGINT (unix epochs don't fit in INT4). SQLite treats BIGINT as
INTEGER, so existing deployments at revision 006 see no schema drift.
- `migrations.py` + CLI take URLs; `db {upgrade,downgrade,current,history}`
gain `--database-url / -u` alongside the legacy `--database-path / -d`.
`get_current_revision()` uses SQLAlchemy inspector instead of raw
sqlite3, so the CLI works against Postgres too.
- `docker-compose.yml` adds a `postgres-test` service under the
`postgres` profile (pinned `postgres:16-alpine` digest) for
integration testing.
- Unit storage tests parametrized over backends via shared
`tests/fixtures/storage_backend.py` — every test in
`test_app_password_storage.py` and `test_webhook_storage.py` runs
once per backend that is available. Postgres is opted in by
`TEST_DATABASE_URL`.
- New `tests/integration/test_storage_postgres.py` (5 tests, marked
`postgres` + `integration`) covers refresh-token, app-password,
OAuth-session, webhook, and audit-log paths end-to-end on Postgres.
- New `docs/ADR-026-pluggable-database-backend.md` records the decision;
`docs/configuration.md` documents `DATABASE_URL` with examples.
Out of scope
------------
- No SQLite → Postgres data migration tool (clean cutover; tokens reissue
on next login, webhooks re-register on next sync tick).
- This repo does not provision Postgres. The matching helm chart change
lives in cbcoutinho/helm-charts (database.url / existingSecret values).
Verification
------------
- `uv run pytest tests/unit/` — 1012 passed, SQLite path unchanged.
- `docker compose --profile postgres up -d postgres-test`
- `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres -v`
— 5 passed.
- `TEST_DATABASE_URL=... uv run pytest tests/unit/test_app_password_storage.py
tests/unit/test_webhook_storage.py` — 50 passed (25 per backend).
- `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean.
Tracked on Astrolabe Cloud POC board, card #99.
---
_This PR was generated with the help of AI, and reviewed by a Human_
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
79ea4e9e21 |
fix(config): emit background-ops advisory logs once per process
`_get_background_operations_enabled()` was emitting three advisory log lines (1 INFO + 2 deprecation WARNINGs) on every call. Because `get_settings()` is intentionally non-cached and runs on every MCP tool invocation via `get_client()`, the "Automatically enabled background operations for semantic search in multi-user mode" INFO line was firing per-request — 569 entries/hour in one production tenant. Gate the three log emissions behind a module-level `_bg_ops_advisories_logged` flag, mirroring the existing `_warn_missing_secret_once` precedent in `vector/webhook_receiver.py`. The boolean-derivation path stays unchanged, so the `Settings` value remains fresh per call. Extends the autouse `_reload_dynaconf_after_test` fixture to reset the new flag between tests, and adds two regression tests that call `get_settings()` five times and assert each advisory fires exactly once. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris Coutinho
|
55ca44c9ec |
Merge remote-tracking branch 'origin/master' into chore/lazy-logging-g004-sweep
# Conflicts: # nextcloud_mcp_server/vector/oauth_sync.py |
||
|
Chris CoutinhoandClaude Opus 4.7
|
65345fd6eb |
refactor: drop OAuth-refresh background-sync path from oauth_sync.py
Follow-up to #787/#789 (ADR-022 cleanup). After
`oauth_enabled ↔ enable_login_flow` became an invariant, the
`use_basic_auth=False` branch in `vector/oauth_sync.py` — and the
parameter wiring that fed it — was no longer reachable from any
supported deployment mode. This commit removes the dead code.
- nextcloud_mcp_server/vector/oauth_sync.py:
- Deleted `get_user_client_oauth` (the OAuth-token refresh helper) and
its `VECTOR_SYNC_SCOPES` constant.
- Deleted the `get_user_client` dispatcher. Internal callers now call
`get_user_client_basic_auth` directly.
- Dropped the `use_basic_auth: bool` parameter from `user_scanner_task`,
`multi_user_processor_task`, `_run_user_scanner_with_scope`, and
`user_manager_task`.
- Dropped the `token_broker` parameter from the same four functions —
they no longer need it now that the OAuth-refresh path is gone. The
`TokenBrokerService` constructed in `app.py` is still used by the
management API revoke endpoint, just not by background sync.
- Simplified the user-list query in `user_manager_task` to always read
from the `app_passwords` table.
- Replaced all `mode_label = "BasicAuth" if use_basic_auth else "OAuth"`
with a literal `[BasicAuth]` log prefix (keeps existing log filters
working).
- Updated the module docstring to describe the post-cleanup shape.
- Dropped the now-unused `TYPE_CHECKING` import of `TokenBrokerService`.
- nextcloud_mcp_server/app.py: dropped the `use_basic_auth = True` block
and the now-stale `token_broker if not use_basic_auth else None` /
`use_basic_auth` positional args from the two `tg.start(...)` calls in
the multi-user vector-sync lifespan. Token broker construction stays —
still consumed by the management API revoke endpoint via
`app.state.oauth_context["token_broker"]`.
- tests/integration/test_app_password_provisioning.py: deleted four tests
that exercised the now-removed OAuth-refresh path
(`test_oauth_mode_uses_refresh_token_only`,
`test_oauth_mode_raises_error_without_token`,
`test_get_user_client_oauth_function`,
`test_oauth_mode_requires_token_broker`) plus the
`test_get_user_client_dispatches_to_basic_auth` test for the deleted
dispatcher. Updated the module docstring + imports accordingly. The
BasicAuth-mode tests (`test_basic_auth_mode_uses_local_storage`,
`test_multiple_users_basic_auth_mode`, etc.) all remain.
No runtime-behaviour change in any supported deployment mode — the deleted
branches were already unreachable post-PR #787. 3 files changed,
+59 / -301; 1010 unit tests pass; integration jobs for
`mcp-login-flow` and `mcp-multi-user-basic` are the critical regression
gates before merge.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
665cb9b1eb |
refactor: convert f-string logging to lazy %-style format (G004)
Sweep all 1676 G004 violations across 112 files, converting
`logger.<level>(f"…{x}…")` to `logger.<level>("…%s…", x)`.
Why: ruff rule G004 was added to pyproject.toml to enforce lazy
%-style logging — defers formatting until the log level is enabled
and lets structured log tooling match the unformatted template.
Conversion preserves rendered output byte-for-byte:
- `{x}` → `%s` + `x`
- `{x!r}` / `{x!s}` / `{x!a}` → `%r` / `%s` / `%a`
- Format specs (`{x:.2f}`, `{x:>10}`) → `%s` + `format(x, 'spec')`
(printf-style specs aren't 1:1 with Python format specs, so we
delegate to `format()` to keep identical output)
- Literal `%` → `%%`
- Concatenated f-strings (`f"a {x} " "b"`) flattened
- Trailing kwargs (`exc_info=True`) preserved
Verified:
- `uv run ruff check --select G004` → 0 violations
- `uv run ty check -- nextcloud_mcp_server` → passes
- `uv run pytest tests/unit/` → 1010 passed
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
18db9d6cb3 |
refactor: prune dead pre-LOGIN_FLOW config/runtime branches
Two small post-merge cleanups deferred from PR #787 (ADR-022 follow-up). Both were explicitly noted in the reviewer's "acknowledged deferred items" list. 1. config.py: drop `enable_multi_user_basic_auth` and `enable_login_flow` from the dynaconf `_DEFAULTS` dict. They were removed from `_field_map` in PR #787, so `get_settings()` never read them anyway, but their presence in `_DEFAULTS` was visually misleading — readers might think they could be set via TOML when in fact `Settings.__post_init__` derives them from `MCP_DEPLOYMENT_MODE`. Replaced with a NOTE comment pointing at the canonical derivation site. 2. app.py: the lifespan code had `use_basic_auth = not oauth_enabled or settings.enable_login_flow`, which became always-True once PR #787 enforced `oauth_enabled ↔ enable_login_flow` via __post_init__. Hard-coded to `True` with a comment explaining the invariant and pointing at the separate follow-up that will prune the now-unreachable `use_basic_auth=False` code paths in `vector/oauth_sync.py` (which includes deleting the `use_basic_auth` parameter from `user_manager_task` / `oauth_processor_task` and the OAuth-token-refresh branch in `get_user_client`). Kept the variable name and the call-site conditionals as-is for now so that follow-up is a clean mechanical diff. No runtime behaviour change: `use_basic_auth` already evaluated to True in every supported mode after PR #787, and the `_DEFAULTS` entries were already shadowed by `__post_init__`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
0fb21b5c6d |
chore: address review-round-5 — stale _sync_derived_flags reference + missing migration-hint test
Two findings from the reviewer's latest pass: - config_validators.py:329: comment in the LOGIN_FLOW validation block still referenced `_sync_derived_flags` (removed in commit 5; derivation now lives in `Settings.__post_init__`). Updated the comment to point at the correct location so a future reader grepping for the function name doesn't come up empty. - tests/unit/test_config_validators.py: added `test_oauth_single_audience_migration_hint` next to the existing `test_invalid_deployment_mode_raises_error`. The new test pins the ADR-022 rename-hint branch in `detect_auth_mode` by setting `MCP_DEPLOYMENT_MODE=oauth_single_audience` and asserting the ValueError mentions both the old and new mode names plus "ADR-022". Without this, a future refactor could drop the hint without any test catching it (the prior `invalid_mode` test only asserts the generic "Valid values:" prefix). No functional changes; 1010 unit tests now pass (+1 from the new hint test). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
1fa4c82fd2 |
chore: address review-round-4 nits — stale delenv, upgrade hint, in-sync notes
Four small follow-ups from the reviewer's latest pass:
- tests/unit/test_stdio.py:18: the single_user_env fixture used
monkeypatch.delenv("ENABLE_MULTI_USER_BASIC_AUTH", ...). That env var
is no longer read after the ADR-022 follow-up; switched to delenv of
MCP_DEPLOYMENT_MODE which is the canonical mode-selection input today.
Comment updated to match.
- config_validators.py: when detect_auth_mode rejects an invalid
MCP_DEPLOYMENT_MODE, surface a one-line ADR-022 migration hint if the
rejected value is exactly "oauth_single_audience" (the most common
upgrade pain — users carrying that value over from ADR-021 .env files).
Other invalid values get the regular "Valid values: …" message
unchanged.
- config.py + config_validators.py: added cross-reference comments on
both mode-resolution sites (Settings.__post_init__ and
detect_auth_mode) noting that they each compute the canonical mode
independently and must be kept in sync when a new mode is added.
Surfaces the parallel-duplication intentionally so the next maintainer
doesn't have to discover it.
- docs/ADR-021-configuration-consolidation.md:92: appended a trailing
comment to the historical "valid values" example, marking
oauth_single_audience and oauth_token_exchange as removed in ADR-022.
ADR-021 stays as the historical record; the trailer points future
readers at the current state.
No functional changes; 1009 unit tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
ade42b55dc |
docs: clear review-round-3 nits — stale login_flow_v2, duplicates, field comments
Five small findings from the reviewer's third round, plus a SonarCloud
quality-gate failure on a test fixture.
- docs/troubleshooting.md, docs/configuration.md: six pre-PR references
to a non-existent `login_flow_v2` mode value (the actual enum value is
`login_flow`). They predated this PR but became actively misleading
once `detect_auth_mode` started raising ValueError for anything not in
the mode_map. Replaced with `login_flow` via sed.
- docs/configuration-migration-v2.md: removed a duplicate
`MCP_DEPLOYMENT_MODE=multi_user_basic` line in the troubleshooting
section (around line 447) — same shape as the round-2 duplicate
caught earlier in the migration-steps section. Also dropped the
`oauth_token_exchange` row from the mode-value table around line 364
(that enum value was removed in
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
6e7c821761 |
fix(config): derive mode flags in Settings.__post_init__; address review round 2
The integration jobs for `mcp-multi-user-basic` and `mcp-login-flow`
were failing with HTTP 500s. Root cause: `get_settings()` builds a
fresh Settings on every call (not cached). Commits 3 and 4 set the
derived `enable_login_flow` / `enable_multi_user_basic_auth` flags as
a side effect of `detect_auth_mode`. detect_auth_mode runs once at
startup, against the Settings instance owned by `validate_configuration`.
Every per-request call site that does `settings = get_settings()` got
a fresh Settings with both flags at their default `False` (since the
env-var aliases were dropped), causing the multi-user dispatcher in
`context.py` to take the wrong branch and crash.
Fix: move the derivation into `Settings.__post_init__`. Every Settings
instance now carries correct flags from the moment it's constructed —
no caching needed, no mutation-after-construction race. detect_auth_mode
becomes a pure reader of the already-derived state.
The legacy env-var deprecation check moves with it. It also picks up
the reviewer's truthy-string fix: previously `os.getenv(legacy)` fired
for the literal string "false" (a non-empty Python string is truthy),
which would have errored on any user with a leftover
`ENABLE_LOGIN_FLOW=false` in their `.env`. The check now only fires
when the value lowercases to one of {"1", "true", "yes", "on"}.
- nextcloud_mcp_server/config.py: extend Settings.__post_init__ with
the legacy-deprecation block and the derived-flag derivation
(resolve mode from deployment_mode + username/password, set flags).
- nextcloud_mcp_server/config_validators.py: drop the
`_sync_derived_flags` helper (superseded by __post_init__). Drop the
legacy-env-var deprecation block (moved). `detect_auth_mode` is now
pure — no mutation. Drop the now-unused `import os`.
- tests/unit/test_config_validators.py: legacy-env-var tests now
expect `ValueError` at `Settings(...)` construction (via `get_settings()`),
not at `detect_auth_mode` call. Added two new tests:
* `test_legacy_env_var_check_ignores_falsy_strings` — pins the
truthy-string fix (reviewer round 2 finding).
* `test_derived_flags_stable_across_get_settings_calls` — regression
test pinning the integration-test fix (two consecutive
`get_settings()` calls return Settings instances with the same
derived flags).
Also reworked `test_login_flow_mode_auto_derives_enable_login_flow_flag`
to assert at-construction derivation (not the old mutation pattern).
- docs/configuration-migration-v2.md: dropped the duplicate
`MCP_DEPLOYMENT_MODE=multi_user_basic` line (review round 2 nit — a
sed artifact from commit 4).
- docs/ADR-021-configuration-consolidation.md: sed-replaced the in-body
`MCP_DEPLOYMENT_MODE=oauth_single_audience` examples with `login_flow`
(review round 2 nit — only the status header was updated in commit 4).
- tests/conftest.py: docstring comment for the multi-user-basic fixture
switched from `ENABLE_MULTI_USER_BASIC_AUTH=true` to
`MCP_DEPLOYMENT_MODE=multi_user_basic` (review round 2 nit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
282c245da1 |
refactor(config)!: drop ENABLE_MULTI_USER_BASIC_AUTH env var, fail loud on legacy aliases
Same pattern as the ENABLE_LOGIN_FLOW removal in the previous commit:
the deployment mode (MCP_DEPLOYMENT_MODE) is the single source of truth
for selecting an auth flow. The ENABLE_MULTI_USER_BASIC_AUTH env-var
alias is redundant with `MCP_DEPLOYMENT_MODE=multi_user_basic`.
Unlike the ENABLE_LOGIN_FLOW removal — where silent removal was safe
because Login Flow v2 is the auto-detection default — silent removal
here would be a surprise: a user with only ENABLE_MULTI_USER_BASIC_AUTH=true
in their .env would auto-detect into LOGIN_FLOW after upgrade (wrong
runtime mode). Mitigation: detect_auth_mode now reads os.environ
directly for both legacy aliases and raises ValueError with a one-line
migration message if either is set. Applied retroactively to
ENABLE_LOGIN_FLOW as well — loud is better than silent.
- nextcloud_mcp_server/config.py:
- Drop the dynaconf env-var alias entry for ENABLE_MULTI_USER_BASIC_AUTH.
- Update the `enable_multi_user_basic_auth` field docstring to mark it
as derived / not user-settable.
- `_is_multi_user_mode()` (early-config helper, runs before Settings
is built) switched to checking MCP_DEPLOYMENT_MODE directly. Now
consistent with the canonical detection in detect_auth_mode.
- nextcloud_mcp_server/config_validators.py:
- Drop the auto-detection branch (`if settings.enable_multi_user_basic_auth`).
Selection of MULTI_USER_BASIC is now exclusively via the explicit
MCP_DEPLOYMENT_MODE branch.
- Add `enable_multi_user_basic_auth` to `_sync_derived_flags` alongside
`enable_login_flow` — both flags are now derived from the resolved mode.
- Drop `enable_multi_user_basic_auth` from
`MODE_REQUIREMENTS[MULTI_USER_BASIC].required` and from the
`forbidden` lists of SINGLE_USER_BASIC and LOGIN_FLOW (no longer
user input → no meaningful forbidden check).
- Add loud-deprecation `ValueError` block at the top of detect_auth_mode
that errors with a clear migration message when ENABLE_MULTI_USER_BASIC_AUTH
or ENABLE_LOGIN_FLOW is found in os.environ.
- tests/unit/test_config_validators.py:
- Switch ~10 fixtures from `enable_multi_user_basic_auth=True` to
`deployment_mode="multi_user_basic"` (mirrors `enable_login_flow`
treatment from the previous commit).
- Switch two `patch.dict(os.environ, {"ENABLE_MULTI_USER_BASIC_AUTH": "true"})`
blocks to use MCP_DEPLOYMENT_MODE.
- Rename `test_forbidden_multi_user_basic_auth` to
`test_forbidden_multi_user_basic_when_credentials_present` — the
scenario is now an explicit-mode + credentials conflict, not an
env-var-flag conflict.
- Add `test_legacy_enable_multi_user_basic_auth_env_var_errors` and
`test_legacy_enable_login_flow_env_var_errors` to exercise the new
loud-deprecation ValueError path.
- docker-compose.yml: mcp-multi-user-basic profile switched to
`MCP_DEPLOYMENT_MODE=multi_user_basic`.
- env.sample: replaced `#ENABLE_MULTI_USER_BASIC_AUTH=true` example with
`#MCP_DEPLOYMENT_MODE=multi_user_basic`.
- docs/authentication.md, configuration.md, troubleshooting.md,
auth-flows.md, webhook-management-guide.md,
configuration-migration-v2.md, ADR-025: replaced env-var examples
with the canonical MCP_DEPLOYMENT_MODE form.
- docs/ADR-020: marked partly superseded by ADR-022.
- CLAUDE.md: Multi-User BasicAuth section updated to set
MCP_DEPLOYMENT_MODE.
- nextcloud_mcp_server/vector/oauth_sync.py: module docstring updated.
BREAKING CHANGE: ENABLE_MULTI_USER_BASIC_AUTH is no longer read from
the environment, and setting it now raises a startup ValueError with
a migration message. Replace `ENABLE_MULTI_USER_BASIC_AUTH=true` with
`MCP_DEPLOYMENT_MODE=multi_user_basic`. The same loud-deprecation
check is also applied to the recently-removed ENABLE_LOGIN_FLOW —
replace with `MCP_DEPLOYMENT_MODE=login_flow` (or drop both;
`login_flow` is the auto-detect default when no other auth env vars
are set).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
df4994e860 |
refactor(config)!: derive enable_login_flow from mode, remove ENABLE_LOGIN_FLOW env var
Once OAUTH_SINGLE_AUDIENCE was renamed to LOGIN_FLOW and the validation
gate ensured the only meaningful configuration was
`MCP_DEPLOYMENT_MODE=login_flow + ENABLE_LOGIN_FLOW=true`, the two
controls became redundant. Setting the mode is sufficient; the
ENABLE_LOGIN_FLOW env var doesn't add information.
This commit makes the deployment mode the single source of truth for
the Login Flow v2 toggle:
- `nextcloud_mcp_server/config.py`: drop the `ENABLE_LOGIN_FLOW`
dynaconf env-var alias. The `enable_login_flow` field stays as an
internal attribute so the 6 runtime call sites (app.py x4,
context.py, auth/scope_authorization.py) keep working unchanged.
Updated field docstring to flag it as derived.
- `nextcloud_mcp_server/config_validators.py`:
- Drop `enable_login_flow` from `MODE_REQUIREMENTS[LOGIN_FLOW].required`.
- Drop the validation gate that required ENABLE_LOGIN_FLOW=true for
LOGIN_FLOW mode (no longer possible to misconfigure — the flag is
derived, not user input).
- Add `_sync_derived_flags()` helper called at every return path of
`detect_auth_mode` to set `settings.enable_login_flow` from the
resolved mode.
- `tests/unit/test_config_validators.py`: drop `enable_login_flow=True`
from happy-path fixtures (no longer needed — detection sets it).
Repurpose `test_login_flow_requires_enable_login_flow_flag` into
`test_login_flow_mode_auto_derives_enable_login_flow_flag` which
asserts the new auto-derivation behaviour for both LOGIN_FLOW and a
non-LOGIN_FLOW mode.
- `docker-compose.yml`: remove `ENABLE_LOGIN_FLOW=true` from the
`mcp-login-flow` and `mcp-keycloak` profiles.
- `env.sample`: remove the ENABLE_LOGIN_FLOW reference; the comment
on `MCP_DEPLOYMENT_MODE` now notes the derived flag.
- `docs/configuration.md`, `docs/authentication.md`,
`docs/login-flow-v2.md`, `docs/auth-flows.md`,
`docs/troubleshooting.md`, `docs/ADR-025-*.md`: replace
ENABLE_LOGIN_FLOW=true examples and references with
MCP_DEPLOYMENT_MODE=login_flow.
BREAKING CHANGE: `ENABLE_LOGIN_FLOW` is no longer read from the
environment. Anyone who relied on `ENABLE_LOGIN_FLOW=true` to activate
Login Flow v2 should set `MCP_DEPLOYMENT_MODE=login_flow` instead (or
rely on it being the default when no other auth env vars are set).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
cafd318f36 |
refactor(config)!: rename OAUTH_SINGLE_AUDIENCE to LOGIN_FLOW, gate on ENABLE_LOGIN_FLOW
The AuthMode.OAUTH_SINGLE_AUDIENCE enum was a vestige of ADR-021's
original design where it co-existed with OAUTH_TOKEN_EXCHANGE. The
un-augmented OAuth bearer pass-through it represented relied on
Nextcloud-side patches to user_oidc (Bearer token validation on
non-OCS endpoints) that were never merged upstream (see
docs/authentication.md, docs/login-flow-v2.md). The working path —
mcp-login-flow profile — sets ENABLE_LOGIN_FLOW=true on top of this
mode so Login Flow v2 acquires per-user Nextcloud app passwords via
a browser flow. With OAUTH_TOKEN_EXCHANGE removed in
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
9d5ac01f24 |
fix(calendar): preserve floating/TZID semantics across CalDAV roundtrip (#782)
The CalDAV REPORT in `_search_events_by_date` unconditionally requested server-side `<C:expand>`. Per RFC 4791 §9.6.5 the server then normalizes every expanded DTSTART/DTEND to UTC `Z`, which destroyed two pieces of information on the read path: - RFC 5545 floating local times came back as fake-UTC (a `+00:00` suffix that did not match the stored value), so a 2:30 PM floating event was indistinguishable from a 14:30 UTC event in the MCP response. - TZID-bound events lost their IANA TZID context — a "10am America/New_York" event came back as `14:00:00+00:00`, making it impossible for callers to reconstruct DST-aware recurrence semantics. Replace `<C:expand>` with client-side recurrence expansion via the `recurring-ical-events` library (promoted from transitive to direct dep), so the wire response retains its original DTSTART format. Surface the TZID parameter as new `start_tz`/`end_tz` fields on `CalendarEventSummary`. Add an optional `timezone` (IANA name) parameter to `nc_calendar_create_event` and `nc_calendar_update_event` so callers can pin a TZID for naive input; the helper attaches `ZoneInfo(...)` and emits a paired `VTIMEZONE` component. Naive input without `timezone` continues to store as RFC 5545 floating local time (with a warning logged). Offset-aware input continues to store as UTC `Z`. Drive-by: switch the update path's DTSTART/DTEND assignment from raw `datetime` to `vDDDTypes(dt)` wrappers — the previous code produced invalid iCal like `DTSTART:2026-05-14 10:00:00+00:00` for any TZ-aware update. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris Coutinho
|
9072559d26 | chore: Address reviewers feedback | ||
|
Chris CoutinhoandClaude Opus 4.7
|
271904c227 |
fix(deck): address review — notesPath key, scopes, modernize types
PR #781 review round 1: - 🔴 Fix notesPath key: the Notes API returns the folder under camelCase ``notesPath`` (see models/notes.py:43), but `deck_attach_note` was looking up snake_case ``notes_path`` and silently falling back to ``"Notes"``. Users with a non-default notes folder would have produced shares pointing at non-existent files (404 on click in Deck UI). - 🔴 Add wire-through unit test that would have caught the above: extract `_resolve_note_attach_path(client, note_id)` as a testable helper that encapsulates the camelCase-key lookup. Three new tests: custom notesPath honored, missing key falls back to default, null category handled. - 🟡 Modernize new fields on `DeckAttachmentExtendedData` to PEP 604 (`X | None`) per CLAUDE.md. - 🟡 Drop unnecessary string forward reference on `ListAttachmentsResponse.results` — DeckAttachment is defined earlier in the same module. - 🟢 Move `pytestmark = pytest.mark.unit` to module level in test_sharing_client.py to match the convention in test_deck_server.py. Per user request: `deck_attach_file` is now scoped `deck.write` + ``files.read`` (was just `deck.write`) so the generic file-share permission story is consistent — only `deck_attach_note` keeps `notes.read` since it specifically reads from the Notes app. Docstring updated to emphasise the tool is generic over the user's Files (PDFs/images/etc., not just markdown). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
c0a974c498 |
feat(deck): add file/note attachment MCP tools
Adds four tools that expose Deck card attachments via the MCP surface: deck_attach_file, deck_attach_note, deck_list_attachments, and deck_delete_attachment. The attach* variants share an existing Files entry (or Notes-app note) with the card via OCS shareType=12 — same mechanism the Deck UI's "Share from Files" picker uses, no file copy. This replaces the prior workaround of appending bulky activity content as Deck card comments: per-PR/per-event narrative now lives in NC Notes and surfaces on the tracking card as a clickable attachment that opens the original note in place. Implementation reuses existing client methods (SharingClient.create_share, DeckClient.get/delete_attachment, NotesClient.get_settings/get_note); no new client code. _SHARE_TYPE_DECK is centralised with a CI-guard test to prevent silent drift, and SharingClient.create_share's wire format is pinned to what the Deck Vue source sends. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
9cb0b33ec1 |
fix(health): forward api-key to Qdrant /readyz so Cloud probes work
The readiness handler called the configured `qdrant_url/readyz` with a
bare httpx.AsyncClient — no headers. That works against a self-hosted
Qdrant (where /readyz is anonymous), but Qdrant Cloud's auth gateway
returns 403 for any unauthenticated request, including /readyz, /livez
and /healthz. Result: every probe against a Cloud cluster fell into
the "status 403" branch, the handler returned 503, and the Pod never
went Ready — even when the configured `AsyncQdrantClient` itself was
authenticating fine for actual collection traffic.
Forward `settings.qdrant_api_key` as the `api-key` header (mirroring
what `vector/qdrant_client.py:540` already does for the real client).
When the key is unset (self-hosted, anonymous case) we send no header,
so existing self-hosted deployments are unchanged.
Verified end-to-end against Qdrant Cloud:
- Without header: GET /readyz -> 403 {"error":"forbidden"}
- With api-key: same request shape returns 200 (matches what
AsyncQdrantClient.wait() relies on internally).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris Coutinho
|
f3a66cf6bb | chore: ruff format | ||
|
Chris Coutinho
|
189024f1ca | Merge remote-tracking branch 'origin/master' into fix/qdrant-doc-id-keyword-index | ||
|
Chris CoutinhoandClaude Opus 4.7
|
8246d9a088 |
fix(vector): address PR review round 17 + local-mode collection-creation regression
Round 17 reviewer (🟡 Important): 1. docs/configuration.md degraded-migration runbook said `doc_id backfill failed on …` but the actual log line in qdrant_client.py:415 is `doc_id backfill scroll failed on …`. Operators grepping the runbook string would have missed it. Insert the `scroll` qualifier. 2. _create_one_payload_index returned True on the 400 schema-conflict path, so a wrong-type index discovered at create time skipped the consolidated `Payload index creation incomplete` summary — but a wrong-type index discovered via the existing-schema check at line 195-206 did fire it. Tenants whose payload_schema is hidden from their JWT (Qdrant Cloud collection-scoped tokens) only ever observe the create-time path, so they never saw the operator-level summary. Return False so the summary fires in both cases. 3. docs/configuration.md said the upgrade-time delay was `proportional to point count while writes are issued` — overstating the cost. Writes are proportional to int-typed points only; the scroll itself is proportional to total point count. Reword. Local-mode collection-creation regression (root-cause of failing single-user / login-flow / multi-user-basic CI jobs): PR #779 changed the existence probe in get_qdrant_client from collection_exists() (returned bool in both modes) to get_collection() + except UnexpectedResponse(status_code=404). The HTTP-mode client raises UnexpectedResponse with a 404 body, but the local/in-memory client raises ValueError(f"Collection {name} not found") — see qdrant_client/local/async_qdrant_local.py. The narrow except clause let the ValueError propagate, app.py's lifespan re-raised as RuntimeError, and the mcp container crashed on first start. Catch ValueError too, with a `not found` substring guard so genuine programming bugs (bad collection_name, etc.) still surface. Tests: extend the existing 400-path test to assert the new failed_fields contract; add two get_qdrant_client unit tests pinning the local-mode VE catch (positive case + propagation case). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandGitHub
|
bf19c948e2 |
Merge pull request #769 from KuriGohan-Kamehameha/feat/contacts-search-contacts
feat(contacts): add nc_contacts_search_contacts free-text search tool |
||
|
Chris Coutinho
|
363c2a2624 |
Merge remote-tracking branch 'origin/master' into fix/qdrant-doc-id-keyword-index
# Conflicts: # nextcloud_mcp_server/vector/qdrant_client.py |
||
|
Chris CoutinhoandClaude Opus 4.7
|
8f4f5c0079 |
fix(vector): address PR review round 16 — type-aware index check, comments
Detect pre-existing payload indexes with the wrong schema type in `_ensure_payload_indexes`. The previous "field already in existing_schema → skip" branch silently survived a collection migrated from the int-doc_id era where `doc_id` is indexed as INTEGER, letting `MatchValue(value="123")` searches keep failing with HTTP 400 on Qdrant Cloud strict mode — exactly the production failure this PR was meant to fix. New behaviour: compare `existing_schema[field].data_type` against the declared type; on mismatch log a WARNING and append to `failed_fields` so the consolidated end-of-function summary picks it up. No auto-repair (operator intervention only — see docs/configuration.md recovery procedure). New test exercises the doc_id-INTEGER scenario end-to-end and asserts both the per-field WARNING and the summary line. Clarify the `_verify_news_items` malformed-doc_id rationale: the news API has no per-item endpoint, so a malformed doc_id genuinely cannot be verified against the source of truth. We err toward false-positive (keep) over false-negative (drop) — same conservative posture as `_verify_notes` and `_verify_deck_cards`. The producer-side validation is the real security boundary; the verifier is defence-in-depth. Both the inline comment and the WARNING message now spell this out. Add a TODO in `get_last_indexed_timestamp` flagging the O(N) cost on every incremental sync tick. The previous single-page `limit=10_000` silently bounded the scroll; paginating fixed correctness but made the unbounded cost visible. The follow-up tracker (canonical TODO at `api/visualization.py`) covers migrating the max-`indexed_at` to a sentinel point or collection metadata for O(1) lookup. Consolidate the duplicate non-numeric-doc_type TODOs at `api/visualization.py:508` and `auth/viz_routes.py:570` into a single canonical comment in `visualization.py`; `viz_routes.py` is reduced to a back-reference. Removes the rot risk of "fixed in one place, forgotten in the other." The canonical comment also references the O(1) timestamp follow-up in `scanner.py`. Document the `batch_size = 256` (qdrant_client.py) vs `_DELETION_TRACKING_PAGE_SIZE = 1024` (scanner.py) split with cross-referencing comments at each site: the smaller batch is for the read-write backfill upsert path (Qdrant accepts ~256-point chunks comfortably); the larger page is for read-only deletion-tracking scrolls where no per-page write round-trip applies. Replace `assert qdrant_client is not None` in `scan_user_documents` with `cast(AsyncQdrantClient, qdrant_client)` plus an explanatory comment. `assert` is silently elided under `-O`; `cast` is the conventional zero-cost narrower for branches the type checker can't infer from the surrounding `if not initial_sync` ternary. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
04bc2325f2 |
fix(qdrant): use get_collection for startup probe (multi-tenant safe, take 2)
Follow-up to PR #778 — `collection_exists()` is also denied by Qdrant Cloud on a collection-scoped JWT, so the multi-tenant fix needs to go one step further: use `get_collection(name)` (the underlying GET `/collections/{name}` call) and treat a 404 `UnexpectedResponse` as the "doesn't exist" signal. That endpoint is the only existence-probe Qdrant permits on a collection-scoped JWT — listing or probing collection metadata cluster-wide is a tenant-isolation boundary by design. Hit during Astrolabe Cloud smoke17 with the post-#778 image: qdrant_client.http.exceptions.UnexpectedResponse: 403 (Forbidden) raw response: {"error":"forbidden"} File "qdrant_client.py", line 84, in get_qdrant_client collection_present = await _qdrant_client.collection_exists(...) Folds the existence check into the same `get_collection()` call that already runs immediately afterward for dimension validation, so the new path is also one fewer round-trip on the happy path. Cold-start (collection genuinely missing) behavior is unchanged: 404 → `collection_info` is None → fall through to `create_collection()`. Whether `create_collection` succeeds is an orthogonal concern (managed multi-tenant setups pre-provision collections externally; admin-key single-tenant setups can create on the fly). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
f2d4982b2f |
fix(qdrant): use collection_exists for startup probe (multi-tenant safe)
The startup path in `get_qdrant_client()` calls `get_collections()` to
check whether the configured collection already exists. That's a
cluster-wide list operation; in managed multi-tenant Qdrant Cloud
deployments where each tenant's JWT is scoped to a single collection
(by design — `access: [{"collection": "tenant_<id>", "access": "rw"}]`),
the call returns `403 Forbidden` and the FastAPI lifespan crashes:
qdrant_client.http.exceptions.UnexpectedResponse: 403 (Forbidden)
raw response: {"error":"forbidden"}
RuntimeError: Cannot start vector sync - Qdrant initialization failed
Switching to `collection_exists(collection_name)` (per-collection
HEAD-style probe) only requires access to the named collection, which
the tenant JWT has. Single-tenant deployments using an admin/master
key are unaffected — they had access to both forms; this picks the
narrower one.
Doesn't change creation semantics: when the collection isn't present
the code path still calls `create_collection`. In a managed setup
where the collection is pre-provisioned by an external admin (e.g.,
the Astrolabe Cloud control plane's create-tenant workflow), that
branch never fires for an existing tenant; cold-start tenants get
their collection created by the workflow before the Pod boots.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
68506f96c5 |
fix(vector): address PR review round 15 — concurrency, pagination, stale coercion
Defer publication of `_qdrant_client` until after the in-lock backfill +
payload-index migration awaits complete. The fast-path check at the top of
`get_qdrant_client` reads the singleton without holding the init lock, so
publishing the constructed-but-unmigrated client let concurrent fast-path
callers fire filtered searches before `_ensure_payload_indexes` ran —
producing HTTP 400 ("Index required but not found") on Qdrant Cloud strict
mode. Local `provisional` is now used for every await inside the lock; the
global is assigned exactly once, last.
Replace the five hand-rolled `scroll(..., limit=10000)` calls in
`vector/scanner.py` (notes / files / news / deck-cards deletion tracking,
plus the timestamp scroll) with a single paginated `_scroll_all_points`
helper. The previous single-page cap silently dropped deletion-tracking
points beyond the first 10 k for any user past that threshold. Pagination
follows Qdrant's documented contract (loop until `next_page_offset is
None`) with a fixed per-page `_DELETION_TRACKING_PAGE_SIZE = 1024`.
Extract `_create_one_payload_index` from `_ensure_payload_indexes` to drop
its cognitive complexity below the SonarQube limit (17 → ≤ 15) without
losing the per-field error-containment rationale; every comment is
preserved verbatim on the helper.
Drop the stale `SearchResult.id` `int | str` comment and the redundant
`str(d)` coercion in `_verify_news_items` — the contract has been
str-only since the producer-side stringification landed earlier in this
PR.
Fix eight `doc_id=<int>` test calls in `test_chunk_context_offset_gate.py`
that violated the `doc_id: str` signature of `get_chunk_with_context`,
plus align `_make_result` in `test_verification.py` to coerce `id=str(...)`
matching the production contract — and update 30+ assertions from int
sets (`{1, 2, 3}`) to str sets (`{"1", "2", "3"}`) so the tests now model
the post-PR `SearchResult.id: str` reality end-to-end. Previously these
were masked by the `str(d)` coercion now removed from production.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
c5020d9629 |
fix(vector): address PR review round 14 — accurate offset-skip comment + news_item doc_id guard
Round-14 review surfaced one blocking and one important issue. context.py: the comment justifying `skip_offset_lookup` claimed chunk_start/end_offset weren't in _PAYLOAD_INDEX_FIELDS — round 13 indexed both as INTEGER, so the comment now actively misleads. Replace with the real reason: an indexed chunk_index miss is canonical (both paths hit the same Qdrant collection), and skipping the offset filter avoids a redundant round-trip. verification.py: hoist an is_valid_nextcloud_doc_id guard before the `int(d)` cast in _verify_news_items, mirroring the boundary-validation pattern already in _fetch_document_text. Coerce via `str(d)` because SearchResult.id is `int | str` (D1 forward-compat widening). Malformed ids now surface as a logger.warning rather than a generic debug line; fail-open semantics are preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
ae23bbe8b8 |
fix(vector): address PR review round 13 — index offset fields + tighten test
- Add chunk_start_offset / chunk_end_offset to _PAYLOAD_INDEX_FIELDS so the legacy offset-based fallback in search/context.py works on Qdrant Cloud strict mode (pre-#75 clients have no chunk_index payload). - Cover chunk_index / chunk_start_offset / chunk_end_offset in the payload-index summary test; refresh the stale field-list comment. - Flag the is_valid_nextcloud_doc_id gate at both chunk-context handler sites with a TODO for future non-numeric doc_types. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
f9ad7dc52e |
fix(vector): address PR review round 12 — bool guard + strict doc_id validation
- _group_int_doc_ids: use type(value) is not int instead of isinstance, since bool is an int subclass and would otherwise stringify to "True"/"False" and corrupt legacy payloads on backfill. - Replace doc_id.isdigit() guards in 5 boundary sites (api/visualization, auth/viz_routes, search/context note/news_item/ deck_card branches) with a shared is_valid_nextcloud_doc_id helper that rejects "0", leading zeros, and Unicode digit classes (superscripts, Arabic-Indic, Devanagari) which pass isdigit() but cannot be valid MySQL AUTO_INCREMENT IDs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
f3ce46da0f |
fix(vector): address PR review round 11 — broaden offset-skip gate, clarify ordering
- search/context.py: drop the doc_type=='file' guard on skip_offset_lookup so notes / deck cards / news items also bypass the unindexed offset fallback when chunk_index is available. Legacy chunk_index=None data still uses the offset path. - vector/qdrant_client.py: clarify the backfill/_ensure_payload_indexes ordering invariant (backfill rewrites payload values only, never schema or indexes). Acknowledge OSS-vs-Cloud uncertainty in the 400-branch comment and the new-collection call-site comment. - vector/scanner.py: hoist qdrant_client to function scope so the file-scroll block doesn't depend on a name bound inside the notes-scroll block. - tests/unit/test_chunk_context_offset_gate.py: flip the note-with- chunk_index test to assert the offset fallback is skipped. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
47c531969f |
fix(vector): address PR review round 10 — index chunk_index, harden index loop, lazy-init lock
Three coordinated fixes flagged as Important in the round-10 review of PR #773: 1. Index chunk_index. The chunk-context fast path in _get_chunk_by_index_from_qdrant and get_chunk_bbox_and_page_from_qdrant filters on chunk_index, but the field was absent from _PAYLOAD_INDEX_FIELDS. On Qdrant Cloud strict mode every chunk-context lookup via chunk_index would 400 and silently fall back to the document re-fetch path — the exact failure mode the chunk_index shortcut exists to avoid. Added as INTEGER schema. 2. Catch raw network errors in _ensure_payload_indexes. The create_payload_index loop only caught UnexpectedResponse, so an httpx.ConnectError or asyncio.TimeoutError mid-loop would propagate uncaught — leaving _qdrant_client assigned and silently skipping all remaining fields. Added a broad Exception catch with the same per-field containment as the 5xx path: log at ERROR with exc_info, append to failed_fields, continue. New test covers the path. 3. Lazy-initialise _qdrant_init_lock. Constructing anyio.Lock() at module import time works for the asyncio backend but anyio's docs advise instantiating synchronization primitives within an async context, and pyproject.toml's anyio_mode = "auto" means tests can run under trio. Moved the construction into get_qdrant_client; safe under cooperative multitasking because there is no await between the None-check and the assignment. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
d60348e77b |
fix(api): validate doc_id at chunk-context handler boundary
Add an .isdigit() guard at the top of both chunk-context handlers so a
non-numeric doc_id fails fast with a clear 400 ("doc_id must be numeric,
got 'abc'") rather than silently bottoming out as a 404 from deep inside
get_chunk_with_context. The earlier int(doc_id) coercion was removed when
doc_id became a pure pass-through to Qdrant's keyword payload index, which
also dropped this boundary validation.
Also align test_backfill_emits_progress_log_every_20_batches' scroll stub
with real Qdrant: next_offset is now "next-1" (str) instead of 1 (int),
matching the sibling test_backfill_rewrites_int_doc_ids_to_str. Pure
stub-fidelity fix; production code already treats next_offset as opaque.
Addresses both 🟡 Important items from PR #773 review round 10.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
fec1596784 |
fix(vector): address PR review round 9 — drop redundant guard, add init lock, test float doc_id path
Addresses the four 🟡 important findings from claude-bot review on PR #773: str (non-Optional) and the guard would silently skip the Qdrant lookup for an empty string. Removing the guard matches the type signature. (`all([…, doc_id, …])` rejects None and empty string, plus `assert doc_id is not None`). No code change needed. `get_qdrant_client()` with a module-level `anyio.Lock`. Double-checked locking keeps the steady-state hot path lock-free. Without this, parallel cold-start callers could all enter the init block and run `_backfill_doc_id_to_string` + `_ensure_payload_indexes` redundantly (idempotent, but noisy). Pattern matches `auth/storage.py:2071`. behavior with three tests covering the float-warning path (the gap called out in the review), the str/None silent-skip paths, and the int-grouping happy path. Verification: - ruff check / format: clean - ty check -- nextcloud_mcp_server: clean - uv run pytest tests/unit/: 969 passed Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
64f0842977 |
fix(vector): guard _group_int_doc_ids against non-int doc_id values
Skip and warn instead of stringifying floats / unexpected types in the backfill helper. A stray doc_id=3.0 would otherwise be rewritten to "3.0", which producers (str(int)) and the keyword index would never match, and which int() on the verification side would reject. Also add a doc_id=0 case to the backfill test to guard against a future falsy-skip regression. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
a27f738dbf |
fix(vector): tighten get_chunk_bbox_and_page_from_qdrant doc_id to str
🔴 Blocking finding from PR #773 latest review: `get_chunk_bbox_and_page_from_qdrant` (`search/context.py:199`) still declared `doc_id: int | str` and passed the raw value into `MatchValue(value=doc_id)` at lines 239 and 256 without `str()` coercion. After this branch's startup backfill normalises every Qdrant `doc_id` payload to a string, an `int` filter would silently match zero points — the function would return `(None, None)` instead of the chunk bbox / page, and PDF highlight overlays would fail in production. Take option 2 from the reviewer's two suggestions (annotation tightening over inline coercion): the producer side of this PR has already narrowed every other `doc_id` annotation to `str`, so this function is the last hold-out. Pushing the contract into the type system means `ty` will catch any future regression at the call site. Production callers in `api/visualization.py` and `auth/viz_routes.py` already pass `doc_id` (str) verbatim after the recent merge with master's chunk_index-first refactor, so no caller-side changes needed. Update the 9 calls in `tests/unit/test_chunk_bbox_helper.py` to use string literals (`"42"` / `"99"` / `"1"`) instead of integers. The mock doesn't validate `MatchValue` value types, so the tests passed with stale int doc_ids today — but they were exercising a path production no longer takes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
d390b3a4b8 |
fix(vector): address PR review round 8 — anyio convention + cosine-safe sentinel + dedup get_collection
Reviewer findings (1 blocking + 2 important): - 🔴 Replace `import asyncio` / `await asyncio.sleep(0)` with `import anyio` / `await anyio.sleep(0)` in the four async side-effect helpers (_scroll_raises, _upsert_raises, _get_collection_raises, _create_index). CLAUDE.md mandates anyio for all async operations; conftest pins the backend to asyncio so the asyncio.sleep call worked today, but the inconsistency would surface the moment that pin moves. - 🟡 Replace the sentinel's zero dense vector with a single non-zero element (`[1e-9] + [0.0] * (dimension - 1)`). Cosine distance is mathematically undefined for the zero vector and Qdrant Cloud strict mode rejects zero-vector upserts. The exact value doesn't matter (sentinel never participates in a search — no user_id/doc_id/doc_type payload) but the upsert itself must be valid. - 🟡 Avoid the duplicate `get_collection` round-trip on every restart. `_ensure_payload_indexes` now accepts an optional `existing_schema: dict | None` parameter; when None it fetches collection_info itself (and the get_collection-failure swallow still applies), but `get_qdrant_client` already fetches collection_info for dimension validation in the existing-collection branch — pass `collection_info.payload_schema or {}` through to skip the second call. The new-collection branch passes `existing_schema={}` explicitly since a freshly created collection has no payload schema. The 🟡 deck_card iteration-fallback finding doesn't apply: the `isdigit()` guard at context.py:612 returns early before either the fast-path or the iteration fallback runs, so non-numeric doc_ids cannot reach the inner `c.id == int(doc_id)` comparison. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |