db04ae3bcc7e80a7d7ac5476e10d38864d86bc92
212
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris CoutinhoandClaude Opus 4.8
|
ab128bef5b |
feat(search): ADR-027 Phase 2 — file-path filter
Add a path_prefix filter to semantic search, honoured on both the MCP tool and the dense-only visualization/API paths through the shared filter contract. - build_base_filter_conditions: append FieldCondition(file_path, MatchText(path_prefix)) when set. file_path is only on doc_type == "file" points, so a non-empty path_prefix implicitly restricts to files. - Promote path_prefix to an explicit keyword param on the SearchAlgorithm ABC and both algorithms; thread it through nc_semantic_search (blank ⇒ no filter), the /api/v1 search endpoints, and the viz route. - Add a file_path TEXT payload index to _PAYLOAD_INDEX_FIELDS (no content re-index; idempotent startup migration). MatchText tokenizes on server Qdrant and matches by substring on local/embedded qdrant-client — both serve folder scoping. - Update ADR-027 (Phase 2 implemented; readiness table; semantics note). Tests. Refs ADR-027 Phase 2. Deck #177. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
c2c8dc1a08 |
feat(search): ADR-027 Phase 1 — modified-date range filter
Add a modified_after/modified_before date-range filter to semantic search, honoured on both the MCP tool path (BM25HybridSearchAlgorithm) and the dense-only visualization/API path (SemanticSearchAlgorithm) through one shared contract. - Promote modified_after/modified_before to explicit keyword params on the SearchAlgorithm ABC and both concrete algorithms; factor the shared placeholder+ownership+doc_type+date filter into access_filter.build_base_filter_conditions so new filters land in one place. - nc_semantic_search: accept RFC 3339 / ISO 8601 (or Unix seconds) bounds via utils.validation.parse_modified_timestamp; Annotated/Field constraints on the numeric args; explicit McpError guard for after > before. Thread the parsed bounds through the cross-app and per-doc_type dispatch. - /api/v1 search endpoints + viz route parse the same formats and 400 on bad or inverted ranges. - Add a modified_at INTEGER payload index to _PAYLOAD_INDEX_FIELDS; the idempotent _ensure_payload_indexes() startup path migrates existing collections with no content re-index. - Update ADR-027 to resolve the review feedback (validation placement, shared algorithm contract, deferral of nc_semantic_search_answer, payload index, RFC-3339-at-the-boundary rationale). Add unit tests. Refs ADR-027. Deck #177. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
52da297ded |
fix(auth): authenticate stored app passwords with loginName, not UID
Nextcloud authenticates app passwords against the *loginName*, which differs from the UID for OIDC-provisioned users (e.g. user_oidc makes the UID the display name: UID "Ada Lovelace", loginName "ada@example.com"). The runtime consumers of stored app passwords bound the UID as the BasicAuth username, so every Notes/Files/Shares/CalDAV call returned HTTP 401. PR #818 fixed only the provisioning endpoint; the consuming paths were missed. Observed on a login_flow tenant (NC's own OIDC app as IdP): the background-sync scan loop never started ("Credential validation failed ... HTTP 401") and semantic search returned 0 results because the ACL shared_with_me lookup 401'd and degraded to a self-only owner filter. Root cause: NextcloudClient / CalendarClient conflated two identities — the DAV/URL path identity (the user_id the whole system keys on = NC UID) and the auth-credential username (the loginName). Decouple them: - Thread a keyword-only auth_username through NextcloudClient -> CalendarClient (defaults to username, so single-user / OAuth where UID == loginName is unchanged). - get_user_client_basic_auth (background sync + the /api/v1/vector-viz/search endpoint) authenticates as the stored loginName, UID for paths. - _get_client_from_login_flow (the get_client(ctx) MCP-tool path) does the same. - cleanup_invalid_app_passwords validates with the loginName, so it no longer 401s and wrongly deletes a valid OIDC user's password. The loginName is already persisted in app_passwords.username and returned by get_app_password_with_scopes. Adds unit tests covering the UID != loginName split for both client builders, the calendar credential/path split, and the cleanup validation. Also genericises the example user in the #818 comment/test (real name/email -> Ada Lovelace / ada@example.com). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
350358b802 |
fix: PR #813 review — shared-file context in MCP tool path + viz over-fetch cap
🟡 Important: nc_semantic_search's include_context branch did not forward accessible_owners to get_chunk_with_context, so context expansion for shared files stayed self-only, found nothing in Qdrant, and silently fell back to the plain excerpt. Forward accessible_owners (the per-file file_accessible_by_id gate still enforces access). 🟡 Performance: auth/viz_routes.py's multi-doc_type branch sorted but did not cap the candidate pool before verify-on-read, so N doc_types × limit*2 went into verification (N× the Nextcloud round-trips). Cap to limit*2 after the sort, matching server/semantic.py and the cross-app branch. Also clear the SonarCloud gate (new_duplicated_lines_density 5.1% > 3%) the ACL wiring introduced: extract the duplicated /api/v1 client-resolution + owner-expansion + verify-on-read block from unified_search/vector_search into a shared _search_with_acl helper, define a constant for the repeated "Nextcloud host not configured" literal (S1192), and reword the access_filter move_to_end comment so it isn't misread as commented-out code (S125) while adding the other-owner count to its debug log (review nits). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
8deb48e6fa |
fix: address PR #813 review round 4 (log leak, cross-user chunk ctx, algo, overlap)
1. Don't log unverified result titles: both search algorithms logged top-5 titles at DEBUG before verify-on-read; with owner-level share expansion the unverified set can contain other users' docs. Algorithms now log a count only; the verifying callers (server/semantic, viz_routes, api/visualization) log verified titles after verify-on-read. 2. Cross-user FILE chunk context: get_chunk_with_context + the Qdrant chunk helpers now take accessible_owners and use build_ownership_filter. For files the expanded scope is honoured only after a per-file file_accessible_by_id check (accessible_owners is owner-level, so the gate prevents a one-file share recipient from reading any of the owner's cached chunks). note/deck/ news stay self-only (per-user APIs) — a documented gap. Both chunk endpoints pass accessible_owners. 3. Algorithm usage: SemanticSearchAlgorithm is not dead (it backs the dense-only option on the viz/API surfaces); added a clarifying comment in server/ semantic.py. Additionally wired accessible_owners + verify-on-read into the /api/v1 search routes (unified_search, vector_search) so the astrolabe surface is ACL-aware too — degrading gracefully to self-only/unverified for non-provisioned callers instead of 401. 4. Overlapping conditions: build_ownership_filter no longer lists self in the owner_id MatchAny branch (self is already covered by the user_id branch); the owner_id branch carries only the OTHER owners. Tests: build_ownership_filter dedup + chunk-bbox filter-shape updates; new ACL-aware get_indexed_doc_types, cached-chunk lookup, and end-to-end cross-user file chunk-context (recipient gets the chunk, non-recipient denied) tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
423d0a1758 |
fix: address PR #813 latest review (ACL-aware doc-type discovery, robustness)
- get_indexed_doc_types: add optional accessible_owners param and reuse
build_ownership_filter so cross-user doc-type discovery matches the real
search scope (was self-only / ACL-blind); docstring documents the self-only
default. Covered by test_get_indexed_doc_types_is_acl_aware.
- access_filter: build_ownership_filter now omits the owner_id branch entirely
for an empty owner set instead of relying on undocumented MatchAny(any=[])
semantics; updated the empty-list unit test accordingly.
- access_filter: make the uid_owner/owner share-owner extraction explicit
("absent, not empty") to avoid skipping on a falsy-but-present field.
- access_filter: add an operator note that pre-owner_id points need a re-index
to surface to share recipients (ACL search is a no-op for legacy data).
- verification/webdav: lock the file_accessible_by_id(scope="") contract with a
targeted multi-user test (owner + recipient True, non-recipient False).
- viz_routes: comment that verify-on-read eviction runs inline by design (no
lifespan task group available on the Starlette route).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.8
|
b1fac2d7a8 |
fix(search): address PR #813 review (viz verify-on-read, owners cache, docs)
- viz_routes: run verify_search_results before returning results. After the accessible_owners expansion the viz can surface OTHER users' shared docs, so it must drop ones the caller can no longer access (revoked share) — same as the nc_semantic_search tool path. (Blocking review item.) - access_filter: cache list_accessible_owners per user for 30s to keep the OCS shares round-trip off the search hot path (failures aren't cached); document the single-page OCS limitation; add a clear_accessible_owners_cache() test helper. Comment the empty-accessible_owners MatchAny([]) edge case. - verification: comment why cross-user eviction is a deliberate no-op (eviction is scoped to the querying user's id, so a recipient's revoked access never deletes the owner's points; the recipient self-heals via accessible_owners). - algorithms: declare SearchResult.original_score (set by the viz route) so the now-precisely-typed result list type-checks. - tests: cross-user eviction-no-op safety test; autouse owners-cache reset in the access_filter + shared-search tests; replace async-no-await qdrant fakes with AsyncMock (clears SonarCloud S7503). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
ae54956f27 |
fix(auth): login-flow provisioning — public login_url + session app passwords
Two fixes surfaced while testing Login Flow v2 provisioning behind a split internal/external host (Docker: server↔Nextcloud over http://app, browser over http://localhost:8080): 1. login_url pointed at the internal host. Nextcloud builds the login URL from the request host, so the browser-facing URL came back as http://app/login/v2/flow/... — unreachable from the user's browser. The poll endpoint was already rewritten to the internal host (correct, the server polls it); now LoginFlowV2Client also rewrites the login_url origin to settings.nextcloud_public_issuer_url when set (passed at all 5 construction sites). When unset, behaviour is unchanged. 2. The app-password format guard rejected raw session tokens. core/ getapppassword returns a long alphanumeric token, not the dashed 25-char Security-settings format, so the dashed-only regex 400'd the one-click opt-in handoff. Relax APP_PASSWORD_PATTERN to `^[a-zA-Z0-9-]{20,256}$`; the authoritative validation is still the BasicAuth check against Nextcloud. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
37db82613d |
feat(search): ACL-aware vector filter via Nextcloud Shares lookup
The vector index has always been strictly per-user: every Qdrant payload
carries a `user_id` and the search filter is `user_id == querying_user`.
A file Alice indexed cannot be discovered by Bob even if she has shared
it with him — Bob would have to re-index it under his own user_id to
make it searchable, which means duplicate index entries for every share
recipient.
Switch to ownership-with-ACL-expansion:
- New `nextcloud_mcp_server.search.access_filter` module:
- `list_accessible_owners(sharing_client, user_id)` calls the OCS
Sharing API (`shared_with_me=true`) and returns
`{user_id} ∪ {uid_owner of each share}`. Fails open to `[user_id]`
so a misbehaving Sharing API doesn't black-hole search.
- `build_ownership_filter(user_id, accessible_owners)` returns a
Qdrant `Filter` whose `should` branch matches either the new
`owner_id IN accessible_owners` field or the legacy `user_id` field.
The legacy branch keeps points indexed before this change reachable
without a migration backfill.
- Indexer payload (`vector/processor.py`) now writes `owner_id` alongside
`user_id`. `DocumentTask` gains an optional `owner_id` field; today the
scanner always runs as the owner so the processor falls back to
`user_id`, but the field is plumbed so a future shared-with-me crawler
can set the true owner without reshaping the payload contract.
- `SemanticSearchAlgorithm.search` and `BM25HybridSearchAlgorithm.search`
accept `accessible_owners` via kwargs and use the new ownership filter.
Default behaviour with no kwarg is unchanged (self-only).
- Both user-facing callers — the MCP tool path (`server/semantic.py`) and
the visualization Starlette route (`auth/viz_routes.py`) — compute
`accessible_owners` from the authenticated Nextcloud client before
invoking the search algorithm. Eviction, scanner deletion, placeholder,
and chunk-context paths intentionally keep the legacy `user_id`
semantics (those are "operations on a specific user's records", not
cross-user reads).
- 10 new unit tests in `tests/unit/search/test_access_filter.py` cover
self-only default, owner expansion, dedup, fallback fields, OCS
failure, and the legacy `should`-branch shape.
Pairs with cbcoutinho/astrolabe#89 — together they let an Astrolabe user
find content owners have shared with them without going through any
re-authorization flow or re-indexing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
e98903c502 |
fix(storage): address review on PR #799 (stale comments, docs deprecation, unit test)
claude-review on #799 flagged: 1. Stale inline comment in ``initialize()`` (line 466) still said "Postgres uses a small bounded pool". Updated to reflect both backends now use NullPool. 2. Stale ``close()`` docstring referenced pool-size starving max_connections — irrelevant with NullPool. Replaced with the NullPool-aware rationale (dispose still tears down in-flight asyncpg connections cleanly). 3. ``docs/configuration.md`` actively directed operators to tune DATABASE_POOL_SIZE / DATABASE_MAX_OVERFLOW, with worked examples and pool math. Both are now deprecated no-ops; the table entries explain the deprecation and link to PR #799. Operators reading the docs will no longer be confused into tuning settings that don't do anything. 4. ``config.py`` comment for the deprecated fields updated to record the deprecation. Validators are intentionally kept (still reject < 1 / < 0) so misconfigured deploys fail loudly rather than silently — the reviewer flagged this as a minor UX wart but explicitly "not a blocker"; the docs change in (3) keeps operators away from the config altogether. 5. New ``tests/unit/test_storage_engine.py`` with three tests: - ``test_postgres_engine_uses_nullpool`` — pins ``isinstance( engine.pool, NullPool)`` so a refactor back to QueuePool / SingletonThreadPool can't silently re-introduce the cross- event-loop crashes. - ``test_postgres_engine_ignores_pool_sizing_settings`` — setting DATABASE_POOL_SIZE / DATABASE_MAX_OVERFLOW to huge values must not change pool type (proves the deprecated fields are wired-up no-ops). - ``test_postgres_engine_missing_asyncpg_driver_message`` — guards the existing actionable-error branch when the optional ``[postgres]`` extra isn't installed. Verified: - ``uv run pytest tests/unit/`` — 1028 passed - ``uv run ruff check`` clean on the touched python files Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
8cd3092e87 |
fix(storage): use NullPool for Postgres engine (cross-loop crashes under anyio TaskGroup)
The Postgres backend hit a hard crashloop in production deployments
where the MCP server runs under anyio TaskGroups with multiple
concurrent background tasks (`vector.oauth_sync.user_manager_task`,
processors, etc.) alongside the request-path code. Symptom in the
pod logs:
RuntimeError: Task <Task pending name='nextcloud_mcp_server.vector.oauth_sync.user_manager_task'>
got Future <Future pending cb=[BaseProtocol._on_waiter_completed()]>
attached to a different loop
followed seconds later by
RuntimeError: Event loop is closed
while SQLAlchemy's pool tries to clean up the failed connection.
The event loop becomes increasingly unresponsive as asyncpg
protocol Futures pile up holding references to closed loops; the
`/health/live` endpoint eventually misses its probe window and
the kubelet SIGKILLs the pod (exitCode 137), restart-looping the
backend.
Root cause: the engine was built with the default
`AsyncAdaptedQueuePool` (`pool_size=2, max_overflow=5`) and
`pool_pre_ping=True`. asyncpg connection objects are bound to the
event loop they were created on. When the process holds a
singleton engine and tasks running under different anyio
TaskGroups check out connections from that pool, the pre-ping
probe runs on a cached connection whose underlying transport
references a different loop's selector → cross-loop access →
crash.
Switch to `NullPool` — one fresh asyncpg connection per
`engine.connect()`, no caching, no cross-loop bookkeeping to get
wrong. asyncpg connection setup is ~5 ms over LAN and a single
round-trip in the local-Postgres case, so the throughput cost is
negligible for the MCP server's traffic shape (low concurrency,
bursty per-user requests). This matches what the SQLite branch
already does (see `initialize()`) and what Alembic's `env.py`
uses for migrations, so the codebase is now consistent across
all backends.
`DATABASE_POOL_SIZE` / `DATABASE_MAX_OVERFLOW` config knobs are
preserved for backward compatibility but no longer affect the
Postgres engine. The validators in `config.py` continue to
reject values < 1 / < 0, so misconfigured deploys still fail
loudly. A follow-up could mark them deprecated in
`docs/configuration.md`; out of scope here.
Discovered while smoke-testing the per-tenant Postgres flow in
Astrolabe Cloud (every-tenant pod fresh-provisions a database
via the ADR-026 backend → hits this crashloop within ~5 min of
the first MCP-routed request).
Refs:
- ADR-026 § "Concurrency model and pool sizing" (the original
QueuePool rationale, now superseded by this finding)
- nextcloud_mcp_server/alembic/env.py (NullPool for migrations)
- SQLAlchemy docs: NullPool is the documented choice when
connection objects don't survive across the lifecycle of the
pool's logical "owner" (here: the event loop)
Verified:
- `uv run ruff check nextcloud_mcp_server/auth/storage.py` clean
- `uv run pytest tests/unit/test_*storage*.py` → 29 passed
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
d717c64750 |
fix(storage): address PR #798 round-4 review (NOSONAR syntax + pg_advisory_lock + engine dispose + nits)
Addresses all 8 items in the round-4 bot review plus 4 remaining SonarQube OPEN issues that were silently broken by round 3's malformed NOSONAR markers. NOSONAR syntax fix (clears the remaining 4 OPEN SQ issues) ---------------------------------------------------------- Round 3 used ``# NOSONAR S<rule_key>`` form. SonarQube Python doesn't recognize the rule-key suffix — it treats the whole thing as a malformed suppression directive (S7632) AND lets the underlying rule keep firing (S7503 on ``_Cursor.__aenter__/__aexit__``). Switch every marker to bare ``# NOSONAR``, with the rationale moved into a preceding comment block. Affected sites: - storage.py: ``_Cursor.__aenter__``, ``_Cursor.__aexit__`` - config.py: ``get_database_ssl()`` ``return False`` + ``ssl.create_default_context()`` - test_storage_logging.py: ``SENTINEL_PASSWORD_FRAGMENT`` constant - test_storage_postgres.py: three ``bob_pw_v1`` / ``bob_pw_v2`` / ``carol_pw`` literals Bot 🔴#1 — defensive NOSONAR on get_database_ssl `return False` -------------------------------------------------------------- Bot predicted S4830 fires on the operator-opt-out path. SQ output shows it doesn't currently fire, but bare NOSONAR added defensively with rationale comment. Bot 🔴#2 — defensive NOSONAR on f-string SQL -------------------------------------------- ``update_oauth_session`` builds its SET clause via ``f"{', '.join(update_fields)}"``; ``get_audit_logs`` builds its WHERE clause via string concatenation. Both are safe (the fragments only come from this function's own branches, no user input), but the patterns trip taint analysers. Annotated both with bare NOSONAR + safety comment explaining the hardcoded-fragments invariant. Note: S2077 doesn't currently fire on these; defensive. Bot 🟡#3 — pg_advisory_lock for concurrent migrations ----------------------------------------------------- Without coordination, two pods rolling-updating simultaneously can both observe ``has_alembic=False`` and both try to apply migrations from scratch — the second crashes with "relation already exists". New ``_migration_lock()`` async context manager: - On Postgres: ``SELECT pg_advisory_lock(:lock_id)`` on a fresh connection (separate from the engine pool so it survives the ``to_thread.run_sync`` worker), held across BOTH the schema-inspect AND the migration call. Without that span, two pods could each observe "no alembic_version" before either started migrating, defeating the lock. - On SQLite: yields immediately (file-level locking serializes writes natively). Lock ID derived from ``sha256(b"nextcloud-mcp-server:migrations")[:8]`` as a stable signed int64 so we can't collide with other apps sharing the same Postgres. Bot 🟡#4 — RefreshTokenStorage.close() + lifespan wiring -------------------------------------------------------- New idempotent ``close()`` method calls ``await engine.dispose()``, nulls the engine, resets ``_initialized``. Wired into both ``app_lifespan_basic`` (BasicAuth) and the OAuth lifespan teardown, each wrapped in ``try/except Exception`` with ``logger.warning`` so a buggy dispose can't block SIGTERM. Without this, pooled asyncpg connections leak server-side slots until ``idle_in_transaction_session_timeout`` reaps them — with small pool defaults and frequent k8s rolling restarts this can starve ``max_connections``. Bot 🟢#5 — is_sqlite_url docstring on :memory: ---------------------------------------------- Updated docstring to note both file-backed and in-memory forms are recognized; caller is responsible for ``:memory:`` magic. Bot 🟢#6 — db_path via make_url(...).database --------------------------------------------- Replaced ``database_url.split("///", 1)[1]`` hack with SQLAlchemy's own URL parsing. Naturally handles in-memory (``.database is None`` → falls back to ``""``). Same lazy-import pattern as the existing ``mask_db_password`` to avoid module-import-time cost. Bot 🟢#7 — _to_sync_url unrecognized-driver guard ------------------------------------------------- Pulled ``_KNOWN_ASYNC_DRIVERS = ("aiosqlite", "asyncpg")`` into a module constant. When an unrecognized ``+<driver>`` token survives the strip, emits ``logger.warning`` with the known-supported list. Behavior unchanged for valid URLs. Bot 🟢#8 — get_audit_logs SELECT * → explicit columns ----------------------------------------------------- Replaced ``SELECT *`` with explicit column list. Future schema additions stay out of the dict return. New tests --------- - ``test_close_disposes_engine``: pins the public contract — engine nulled, state reset, second call is a no-op. - ``test_concurrent_initialize_serialized_by_advisory_lock``: spawns 3 concurrent inits against a fresh schema; asserts no "relation already exists" and exactly one ``alembic_version`` row at the end. Without the lock, this reliably fails on the second concurrent task. Docs ---- - ADR-026: new "Concurrent migrations across pods" subsection documents the advisory-lock approach + lock-ID derivation. Verification ------------ - ``uv run pytest tests/unit/`` — 1025 passed. - ``TEST_DATABASE_URL=… uv run pytest tests/integration/test_storage_postgres.py -m postgres`` — 9 passed (was 7). - ``ruff check && ruff format --check && ty check`` — clean. Expected post-push: SQ scan reports 0 OPEN issues (was 4). Tracked on Astrolabe Cloud POC board, card #99. --- _This PR was generated with the help of AI, and reviewed by a Human_ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
51419329b0 |
fix(storage): address PR #798 round-3 review (SonarQube + pool sizing + RETURNING test)
Round-3 fixes. Two threads: - 9 OPEN SonarQube issues caused the "E Security Rating on New Code" gate failure. The bot's diagnosis (sa.text(text_sql) → SQL injection) was a wrong guess; the actual SQ rules firing were different. - Bot's substantive concerns: pool defaults too aggressive, delete_browser_session RETURNING path untested on Postgres, schema_version legacy table created on Postgres, stale module docstring. - User's underlying question on the pool: "isn't 1 connection enough?" Right-sized to 2+5 and documented the concurrency model in ADR-026 so the rationale is durable. SonarQube quality-gate fixes (clears all 9 OPEN issues) ------------------------------------------------------- - BLOCKER S6418: rename `SECRET` constant in test_storage_logging.py to `SENTINEL_PASSWORD_FRAGMENT` + NOSONAR with rationale. - CRITICAL S3776: extract `_build_postgres_engine()` from `initialize()` (was complexity 26 > 15); incidentally creates a clean unit-test seam for engine args. - CRITICAL S4423: `ssl.create_default_context(cafile=...)` is flagged as "weak protocol" — Python 3.10+ already negotiates the strongest available protocol. Explicitly pass `purpose=ssl.Purpose.SERVER_AUTH` and NOSONAR with the Python-version rationale. - MAJOR S3358: split the TLS-mode nested ternary in the engine factory into a `_describe_ssl_arg()` helper. - MAJOR S2068 ×3: bind test app-password literals to local vars and put `# NOSONAR S2068` on the same line as the literal (anchoring requirement) instead of on the closing paren. - MINOR S7503 ×2: `# NOSONAR S7503` on `_Cursor.__aenter__/__aexit__` — they MUST be `async` per the context-manager protocol. Pool sizing right-sized (answers "why so many connections?") ------------------------------------------------------------ - `DATABASE_POOL_SIZE` default 10 → **2**. - `DATABASE_MAX_OVERFLOW` default 20 → **5**. - Per-pod max drops from 30 to 7. With 3 replicas, total = 21 connections (was 90) — well under managed-Postgres `max_connections=100`. - New INFO log at startup: `Postgres engine ready: pool_size=N max_overflow=M (per-pod max K connections)`. Surfaces the active sizing without grepping config. - New ADR-026 § "Concurrency model and pool sizing" explains asyncpg's single-flight connection semantics, the MCP workload shape (read-mostly point lookups), why-not-1 (multi-user serialization), and the tune-up/tune-down recipe. - `docs/configuration.md` table updated with new defaults + homelab-vs-prod tuning guidance, linking the ADR. RETURNING path covered on Postgres ---------------------------------- - New `test_browser_session_delete_returning` exercises the `DELETE … RETURNING user_id` path — the only RETURNING clause in the storage layer and the most dialect-sensitive SQL in this PR. Asserts both present-row (returns True, row gone) and absent-row (returns False) branches. Schema portability polish ------------------------- - `alembic 001`: gate `schema_version` table creation on `op.get_bind().dialect.name == "sqlite"`. The table exists purely to match the fingerprint of pre-Alembic SQLite databases; fresh Postgres installs no longer carry the dead legacy table. Misc polish ----------- - Module docstring: "SQLite-based" → "SQL-backed", with a sentence on the DATABASE_URL opt-in and an ADR-026 link. - Comment on `_wrap_row` noting `row._mapping` is the documented RowMapping accessor in SQLAlchemy 2.x despite the underscore. Skipped (rationale in PR reply) ------------------------------- - `_qmark_to_named` SQL-comment handling: docstring already notes the limitation; no `?` in storage SQL comments today. - Module-level `anyio.Lock()`: established precedent confirmed by the bot itself. - `get_audit_logs` `SELECT *`: pre-existing pattern, out of scope. Verification ------------ - `uv run pytest tests/unit/` — 1025 passed. - `TEST_DATABASE_URL=… uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 7 passed. - `ruff check && ruff format --check && ty check` — clean. - Confirmed `schema_version` absent on fresh Postgres, still present on fresh SQLite. Tracked on Astrolabe Cloud POC board, card #99. --- _This PR was generated with the help of AI, and reviewed by a Human_ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
f2b7bf132f |
fix(storage): address PR #798 review feedback (credentials, asyncpg extra, TLS, pool)
Round-2 fixes after the bot review on PR #798 plus two user follow-ups (self-signed Postgres support; asyncpg should be a PyPI extra). Folded into the same PR rather than a follow-up since the work is still unmerged. Security -------- - Mask database credentials in all 5 log call sites (storage.py × 4, migrations.py × 1) via a new `mask_db_password()` helper in config.py. Uses SQLAlchemy's `make_url(...).render_as_string(hide_password=True)` with a regex fallback so the masking path never raises. - New `tests/unit/test_storage_logging.py` asserts a sentinel password never appears in `caplog` during `RefreshTokenStorage.initialize()`. Distribution ------------ - `asyncpg` moved to `[project.optional-dependencies] postgres` so a vanilla `pip install nextcloud-mcp-server` no longer pulls in the ~5 MB C extension. The Docker image runs `uv sync --extra postgres`, so containerized deployments are unchanged. - When `DATABASE_URL=postgresql+asyncpg://...` is set on a venv missing the extra, `RefreshTokenStorage.initialize()` raises a friendly RuntimeError pointing at `[postgres]` rather than the generic ModuleNotFoundError. TLS for the Postgres backend ---------------------------- - New `DATABASE_VERIFY_SSL` + `DATABASE_CA_BUNDLE` env vars mirror the existing `NEXTCLOUD_VERIFY_SSL` / `NEXTCLOUD_CA_BUNDLE` pattern (validators in Settings.__post_init__, `get_database_ssl()` helper alongside `get_nextcloud_ssl_verify()`). `DATABASE_VERIFY_SSL=false` wins over `DATABASE_CA_BUNDLE` for incident-response convenience. - Default is **None** rather than True — keeps PR #798's behavior intact for cluster-internal Postgres that runs without TLS. Operators opt into verify-full or supply a private CA. ADR-026 records the reasoning vs the Nextcloud HTTPS default. - Engine factory in `storage.py` passes `ssl` via `connect_args` only when `get_database_ssl()` returns non-None; otherwise asyncpg's default (`prefer`) applies. - Storage logs which TLS mode is active at INFO (no secret material). Configurable connection pool ---------------------------- - `DATABASE_POOL_SIZE` (default 10) and `DATABASE_MAX_OVERFLOW` (default 20) replace the hardcoded engine values. With many replicas this can blow past managed-Postgres `max_connections=100`; tune down for large fleets. - gte-1 / gte-0 validators in __post_init__ reject 0/negative pool sizes at startup with the offending value in the error. Consistency polish ------------------ - Migration 006: convert raw `op.execute("ALTER TABLE ... ADD COLUMN")` to `op.batch_alter_table(...).add_column(sa.Column("nonce", sa.Text))` for stylistic consistency with the rewritten 001-005. Downgrade now drops the column instead of being a no-op. - `registered_webhooks.created_at` standardized from `sa.Float` to `sa.BigInteger` (all other `*_at` columns); `store_webhook()` casts `time.time()` → `int`. - `is_sqlite_url()` made case-insensitive. Testing ------- - New `tests/integration/test_storage_postgres.py::test_cleanup_expired_roundtrip` exercises `cleanup_expired_tokens`, `cleanup_expired_sessions`, and `cleanup_expired_browser_sessions` — relies on DELETE rowcount, historically dialect-tricky. - `tests/unit/test_ssl_config.py` extended with `TestDatabaseSSLSettings` + `TestGetDatabaseSSL` classes (9 new tests) mirroring the existing Nextcloud SSL tests one-for-one. Docs ---- - `docs/configuration.md` Centralized-Storage section grew the four new env vars + a homelab example with a private CA. - `docs/ADR-026` grew Distribution, TLS, and `alembic/env.py` async-pattern subsections explaining the non-obvious design choices. Helm chart counterpart in cbcoutinho/helm-charts PR #34 (separate commit on `feat/nextcloud-mcp-server-database-url`). Verification ------------ - `uv run pytest tests/unit/` — 1025 passed. - `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 6 passed (including new cleanup test). - `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean. Tracked on Astrolabe Cloud POC board, card #99. --- _This PR was generated with the help of AI, and reviewed by a Human_ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
292cbb3292 |
feat(storage): pluggable database backend via DATABASE_URL (ADR-026)
Adds a `DATABASE_URL` setting that lets `RefreshTokenStorage` run against
any SQLAlchemy async backend, primarily `postgresql+asyncpg://...` for
HA k8s deployments. Default behavior is unchanged: when `DATABASE_URL` is
unset the server falls back to the existing `TOKEN_STORAGE_DB` path /
ephemeral SQLite tempfile.
Why
---
Today every MCP pod needs its own PVC to hold the SQLite file, which
pins the Deployment to one replica and blocks horizontal scaling. With
this change, operators can point all replicas at a shared Postgres
(CNPG, RDS, etc.) and the pods become stateless. Encryption stays in
Python (Fernet); the database only sees ciphertext.
What changed
------------
- `config.get_database_url()` resolves DATABASE_URL → TOKEN_STORAGE_DB →
ephemeral tempfile in that priority order.
- `RefreshTokenStorage` builds a process-shared `AsyncEngine` in
`initialize()`. SQLite gets NullPool; Postgres gets pool_size=10,
max_overflow=20, pool_pre_ping=True. 30 aiosqlite call sites adapted
via a thin `_DBConn` / `_Cursor` / `_Row` / `_ExecuteCtx` shim so
existing method bodies need no churn beyond the connection
context-manager swap.
- 7 `INSERT OR REPLACE` statements rewritten as portable
`INSERT ... ON CONFLICT (...) DO UPDATE` (SQLite ≥ 3.24, Postgres ≥ 9.5).
- `sqlite_master` legacy-detection lookup replaced with SQLAlchemy
inspector so the path works against either backend.
- File-permission hardening + parent-dir creation gated on
`is_sqlite_url(...)` — centralized backends manage their own filesystem.
- Alembic migrations 001/002/003/005 converted from raw `op.execute(SQL)`
to portable `op.create_table()` / `op.create_index()` with SQLAlchemy
types. All timestamp columns are `sa.BigInteger` so Postgres allocates
BIGINT (unix epochs don't fit in INT4). SQLite treats BIGINT as
INTEGER, so existing deployments at revision 006 see no schema drift.
- `migrations.py` + CLI take URLs; `db {upgrade,downgrade,current,history}`
gain `--database-url / -u` alongside the legacy `--database-path / -d`.
`get_current_revision()` uses SQLAlchemy inspector instead of raw
sqlite3, so the CLI works against Postgres too.
- `docker-compose.yml` adds a `postgres-test` service under the
`postgres` profile (pinned `postgres:16-alpine` digest) for
integration testing.
- Unit storage tests parametrized over backends via shared
`tests/fixtures/storage_backend.py` — every test in
`test_app_password_storage.py` and `test_webhook_storage.py` runs
once per backend that is available. Postgres is opted in by
`TEST_DATABASE_URL`.
- New `tests/integration/test_storage_postgres.py` (5 tests, marked
`postgres` + `integration`) covers refresh-token, app-password,
OAuth-session, webhook, and audit-log paths end-to-end on Postgres.
- New `docs/ADR-026-pluggable-database-backend.md` records the decision;
`docs/configuration.md` documents `DATABASE_URL` with examples.
Out of scope
------------
- No SQLite → Postgres data migration tool (clean cutover; tokens reissue
on next login, webhooks re-register on next sync tick).
- This repo does not provision Postgres. The matching helm chart change
lives in cbcoutinho/helm-charts (database.url / existingSecret values).
Verification
------------
- `uv run pytest tests/unit/` — 1012 passed, SQLite path unchanged.
- `docker compose --profile postgres up -d postgres-test`
- `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres -v`
— 5 passed.
- `TEST_DATABASE_URL=... uv run pytest tests/unit/test_app_password_storage.py
tests/unit/test_webhook_storage.py` — 50 passed (25 per backend).
- `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean.
Tracked on Astrolabe Cloud POC board, card #99.
---
_This PR was generated with the help of AI, and reviewed by a Human_
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
665cb9b1eb |
refactor: convert f-string logging to lazy %-style format (G004)
Sweep all 1676 G004 violations across 112 files, converting
`logger.<level>(f"…{x}…")` to `logger.<level>("…%s…", x)`.
Why: ruff rule G004 was added to pyproject.toml to enforce lazy
%-style logging — defers formatting until the log level is enabled
and lets structured log tooling match the unformatted template.
Conversion preserves rendered output byte-for-byte:
- `{x}` → `%s` + `x`
- `{x!r}` / `{x!s}` / `{x!a}` → `%r` / `%s` / `%a`
- Format specs (`{x:.2f}`, `{x:>10}`) → `%s` + `format(x, 'spec')`
(printf-style specs aren't 1:1 with Python format specs, so we
delegate to `format()` to keep identical output)
- Literal `%` → `%%`
- Concatenated f-strings (`f"a {x} " "b"`) flattened
- Trailing kwargs (`exc_info=True`) preserved
Verified:
- `uv run ruff check --select G004` → 0 violations
- `uv run ty check -- nextcloud_mcp_server` → passes
- `uv run pytest tests/unit/` → 1010 passed
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
8f4f5c0079 |
fix(vector): address PR review round 16 — type-aware index check, comments
Detect pre-existing payload indexes with the wrong schema type in `_ensure_payload_indexes`. The previous "field already in existing_schema → skip" branch silently survived a collection migrated from the int-doc_id era where `doc_id` is indexed as INTEGER, letting `MatchValue(value="123")` searches keep failing with HTTP 400 on Qdrant Cloud strict mode — exactly the production failure this PR was meant to fix. New behaviour: compare `existing_schema[field].data_type` against the declared type; on mismatch log a WARNING and append to `failed_fields` so the consolidated end-of-function summary picks it up. No auto-repair (operator intervention only — see docs/configuration.md recovery procedure). New test exercises the doc_id-INTEGER scenario end-to-end and asserts both the per-field WARNING and the summary line. Clarify the `_verify_news_items` malformed-doc_id rationale: the news API has no per-item endpoint, so a malformed doc_id genuinely cannot be verified against the source of truth. We err toward false-positive (keep) over false-negative (drop) — same conservative posture as `_verify_notes` and `_verify_deck_cards`. The producer-side validation is the real security boundary; the verifier is defence-in-depth. Both the inline comment and the WARNING message now spell this out. Add a TODO in `get_last_indexed_timestamp` flagging the O(N) cost on every incremental sync tick. The previous single-page `limit=10_000` silently bounded the scroll; paginating fixed correctness but made the unbounded cost visible. The follow-up tracker (canonical TODO at `api/visualization.py`) covers migrating the max-`indexed_at` to a sentinel point or collection metadata for O(1) lookup. Consolidate the duplicate non-numeric-doc_type TODOs at `api/visualization.py:508` and `auth/viz_routes.py:570` into a single canonical comment in `visualization.py`; `viz_routes.py` is reduced to a back-reference. Removes the rot risk of "fixed in one place, forgotten in the other." The canonical comment also references the O(1) timestamp follow-up in `scanner.py`. Document the `batch_size = 256` (qdrant_client.py) vs `_DELETION_TRACKING_PAGE_SIZE = 1024` (scanner.py) split with cross-referencing comments at each site: the smaller batch is for the read-write backfill upsert path (Qdrant accepts ~256-point chunks comfortably); the larger page is for read-only deletion-tracking scrolls where no per-page write round-trip applies. Replace `assert qdrant_client is not None` in `scan_user_documents` with `cast(AsyncQdrantClient, qdrant_client)` plus an explanatory comment. `assert` is silently elided under `-O`; `cast` is the conventional zero-cost narrower for branches the type checker can't infer from the surrounding `if not initial_sync` ternary. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
ae23bbe8b8 |
fix(vector): address PR review round 13 — index offset fields + tighten test
- Add chunk_start_offset / chunk_end_offset to _PAYLOAD_INDEX_FIELDS so the legacy offset-based fallback in search/context.py works on Qdrant Cloud strict mode (pre-#75 clients have no chunk_index payload). - Cover chunk_index / chunk_start_offset / chunk_end_offset in the payload-index summary test; refresh the stale field-list comment. - Flag the is_valid_nextcloud_doc_id gate at both chunk-context handler sites with a TODO for future non-numeric doc_types. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
f9ad7dc52e |
fix(vector): address PR review round 12 — bool guard + strict doc_id validation
- _group_int_doc_ids: use type(value) is not int instead of isinstance, since bool is an int subclass and would otherwise stringify to "True"/"False" and corrupt legacy payloads on backfill. - Replace doc_id.isdigit() guards in 5 boundary sites (api/visualization, auth/viz_routes, search/context note/news_item/ deck_card branches) with a shared is_valid_nextcloud_doc_id helper that rejects "0", leading zeros, and Unicode digit classes (superscripts, Arabic-Indic, Devanagari) which pass isdigit() but cannot be valid MySQL AUTO_INCREMENT IDs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
d60348e77b |
fix(api): validate doc_id at chunk-context handler boundary
Add an .isdigit() guard at the top of both chunk-context handlers so a
non-numeric doc_id fails fast with a clear 400 ("doc_id must be numeric,
got 'abc'") rather than silently bottoming out as a 404 from deep inside
get_chunk_with_context. The earlier int(doc_id) coercion was removed when
doc_id became a pure pass-through to Qdrant's keyword payload index, which
also dropped this boundary validation.
Also align test_backfill_emits_progress_log_every_20_batches' scroll stub
with real Qdrant: next_offset is now "next-1" (str) instead of 1 (int),
matching the sibling test_backfill_rewrites_int_doc_ids_to_str. Pure
stub-fidelity fix; production code already treats next_offset as opaque.
Addresses both 🟡 Important items from PR #773 review round 10.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris Coutinho
|
9720a7e4fe | Merge remote-tracking branch 'origin/master' into fix/qdrant-doc-id-keyword-index | ||
|
Chris CoutinhoandClaude Opus 4.7
|
7ef8760d27 |
fix(chunk-context): address PR #767 review — extract bbox helper, fix page_number overwrite
Resolves both 🟡 important issues from the latest review: 1. `page_number` was unconditionally overwritten in `viz_routes.py:696` even when Qdrant's payload lacked the field, clobbering the value resolved from `chunk_context.page_number`. The new helper returns each field independently and both call sites only overwrite via `is not None` guards, matching the existing logic in `visualization.py`. 2. The ~60-line `if chunk_index is not None: ... else: ...` Qdrant scroll block was duplicated between `api/visualization.py` and `auth/viz_routes.py`. Extracted into `get_chunk_bbox_and_page_from_qdrant` in `search/context.py` alongside the existing private `_get_chunk_*_from_qdrant` helpers; both routes now share ~12 lines of caller code. New unit tests at `tests/unit/test_chunk_bbox_helper.py` cover the indexed and offset paths, the `(bbox, None)` regression case, and graceful degradation on Qdrant strict-mode 400 (which also closes nit #4). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
c780f96d2b |
fix(chunk-context): address PR #767 review — drop dead PDF branch, redundant alias, add boundary tests
- Remove unreachable doc_type=="file" branch (and pymupdf/pymupdf4llm imports) from _fetch_document_text in search/context.py — the file path is short-circuited in get_chunk_with_context before reaching it. - Drop the redundant `username = request.user.display_name` alias in auth/viz_routes.py; both Qdrant scroll filters now reference user_id consistently with the rest of the handler. - Add TestAdjacentChunkBoundary in tests/unit/test_chunk_context_offset_gate.py covering chunk_index=0 (before-fetch gate closed) and chunk_index=total_chunks-1 (after-fetch gate closed) — the two off-by-one boundaries previously untested. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
47b0b737b6 |
fix(chunk-context): address PR #767 round-3 review — gate readability + legacy-fallback comment
- search/context.py: rename triple-negation gate condition to `skip_offset_lookup` named boolean for readability; convert new logger.warning to lazy %-style per repo convention. - api/visualization.py, auth/viz_routes.py: add comment on the offset-only Qdrant scroll branch noting it is a legacy path for pre-astrolabe#75 clients and degrades gracefully on Qdrant Cloud strict mode. Reviewer item #2 (extracting the duplicated scroll block into a shared helper) deferred to a follow-up issue per the reviewer's "not blocking" framing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris Coutinho
|
058463ee87 |
Merge remote-tracking branch 'origin/master' into fix/chunk-context-indexed-lookup
# Conflicts: # nextcloud_mcp_server/api/visualization.py # nextcloud_mcp_server/auth/viz_routes.py |
||
|
Chris Coutinho
|
02744a50e0 | Merge remote-tracking branch 'origin/master' into fix/qdrant-doc-id-keyword-index | ||
|
Chris CoutinhoandClaude Opus 4.7
|
51c1d42ea3 |
fix(chunk-context): address PR #767 round-2 review — gate, parity, doc
Latest reviewer comment flagged five items on top of the original PR. This commit addresses every one: 🟡 1. Skip the offset-based Qdrant fallback for `doc_type=file` when `chunk_index` is supplied. Qdrant Cloud's strict mode rejects unindexed filter fields with HTTP 400, which `_get_chunk_from_qdrant` catches and logs at `logger.error` — masking real Qdrant problems in monitoring. Notes/cards keep the offset fallback (cheap, useful for legacy data). 🟡 2. Add a `logger.warning` and clarifying inline comment in the doc-text fallback path when `chunk_index` is None — surfaces the pre-existing "0/N misreport" so callers can detect it. Type-nullability propagation is deferred to a follow-up (out of scope for this hotfix). 🟢 3. Simplify `if chunk_text and doc_id_int is not None:` → `if chunk_text:` with an inner `assert doc_id_int is not None` for `ty` narrowing. The outer second clause was dead. 🟢 4. Add `doc_type` `FieldCondition` to the offset-based image lookup in both `visualization.py` and `viz_routes.py` for parity with the `chunk_index` branches. 🟢 5. Inline the `chunk_filter` local in `visualization.py` directly into the `must=[]` list (matches `viz_routes.py` style). Adds `tests/unit/test_chunk_context_offset_gate.py` with three regression tests covering the gate matrix: (file, with-index → skip offset), (note, with-index → still tries offset), (file, no-index → still tries offset). Lives at top-level rather than `tests/unit/search/` to side-step a pre-existing circular-init issue in `nextcloud_mcp_server.search`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
ee402ea00e |
feat(vector): replace inline page-image payloads with chunk_bbox (Deck #76)
Per-chunk PDF page renders (~150–700 KB base64 PNG each) were the dominant disk consumer in production, repeatedly tripping `No space left on device: WAL buffer size exceeds available disk space` on welcomed-malamute Qdrant. Replace the inline highlighted_page_image / highlighted_page_number / highlight_count fields with a small `chunk_bbox` field: list[(x0, y0, x1, y1)] of normalized [0, 1] floats, ~32 bytes per chunk. Astrolabe (the only known consumer) renders the highlight client-side as a percentage-positioned overlay on top of the existing /api/v1/pdf-preview render-on-demand path (cbcoutinho/astrolabe#76). - pdf_highlighter: new compute_chunk_bboxes_batch() that reuses the existing _find_chunk_bbox text-search path, skipping all pixmap/PIL/PNG work. - processor: store chunk_bbox + chunk_bbox_page in the Qdrant payload, drop highlighted_page_image + friends, drop the base64 import. - visualization /api/v1/chunk-context and auth/viz_routes: read chunk_bbox instead of highlighted_page_image. - vector/__init__: stop eagerly re-exporting `processor`/`scanner` — fixes a pre-existing circular import (search.algorithms -> vector.placeholder -> vector/__init__ -> processor -> scanner -> server.semantic -> search.bm25_hybrid -> search.algorithms partial). Test suite that was broken on master (test_bm25_hybrid.py et al.) now collects and passes. - scripts/purge_page_images.py: ad-hoc, idempotent migration that delete_payload's the legacy keys from existing points. No reindex required; legacy chunks render the page with no overlay. Pairs with cbcoutinho/astrolabe#76. Frontend handles missing chunk_bbox gracefully, so this can land in either order. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
8457c427a5 |
fix(chunk-context): address PR #767 review — doc_type filter parity + tests
- Add `doc_type` FieldCondition to the chunk_index-path highlighted-image Qdrant filter in both `api/visualization.py` and `auth/viz_routes.py`, matching the shape of `_get_chunk_by_index_from_qdrant`. Safe today (the block is guarded by `doc_type == "file"` and Nextcloud file IDs are globally unique) but prevents a latent bug if other doc types start storing highlighted images. - Demote `viz_routes.py` `ValueError` log from `error` to `warning` (lazy %-style) — `_parse_int_param` raises on user-supplied bad input, which is a 400 not a server error and shouldn't pollute error logs. - Hoist `effective_chunk_index` to compute once at the top of `get_chunk_with_context`, removing two duplicate assignments. - Add `test_file_doc_type_qdrant_miss_yields_fast_404` to the management endpoint tests, locking in the proxy-timeout fix contract. - Add `tests/unit/test_viz_routes_chunk_context.py` mirroring management coverage for the OAuth-session route: param forwarding (chunk_index / total_chunks), `doc_type=file` fast 404, and 400 on invalid int params. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
719b3b5034 |
fix(vector): normalize doc_id to str + add Qdrant keyword payload indexes
Production was logging two cascading classes of Qdrant errors against the
welcomed-malamute deployment:
1. HTTP 400 — "Bad request: Index required but not found for \"doc_id\" of
one of the following types: [keyword]". The collection was created via
create_collection() with no payload indexes, so any FieldCondition
filter on doc_id failed at the Qdrant layer (placeholder writes/reads,
eviction, search context lookups).
2. Compounding the missing index, producers wrote a mix of int and str
doc_ids: webhook_parser stringified node_id, scanner stringified note
IDs, news IDs, and deck card IDs — but the file scanner passed the
numeric file_id through unchanged. A keyword index would not have
covered both kinds even if it had existed.
This change:
- Normalizes doc_id to str at every producer site (scanner.py:459,
DocumentTask.doc_id, indexed_*_ids reads from Qdrant).
- Tightens str|int annotations to str across placeholder.py,
eviction.py, search/verification.py, search/context.py,
SearchResult.id, and the auth/api visualization endpoints.
- Defensive str() coercion on doc_id reads in semantic.py /
bm25_hybrid.py / vector/visualization.py for the transition window
before the backfill runs.
- Adds an idempotent startup migration in get_qdrant_client():
- _ensure_keyword_payload_indexes creates KEYWORD indexes for
doc_id, user_id, and doc_type (tolerates "already exists" 400s).
- _backfill_doc_id_to_string scrolls the collection once and rewrites
int doc_ids to str. Skipped after a quick sample shows no legacy
int payloads.
- Public API preserved: SemanticSearchResult.id stays int via explicit
int(r.id) narrowing in server/semantic.py — surfaces a TypeError with
actionable context if a future doc_type ships non-numeric ids.
- Documents the startup migration in docs/configuration.md.
Tests: 11 new unit tests in tests/unit/vector/test_qdrant_client.py
covering happy path / already-exists / unrelated-400 for the index
helpers, and sample-skip / mixed-batch rewrite / payload=None edge cases
for the backfill. 889 unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
53e6dba5a2 |
fix(viz_routes): address PR #767 review — param parity + always-on page_number
- Replace bare int() casts for start/end/context_chars in chunk_context_endpoint with _parse_int_param, matching visualization.py bounds (0–10M for offsets, 0–10K for context_chars), and add the missing end > start guard. - Initialize page_number from chunk_context.page_number so non-file doc_types surface it; include page_number, chunk_index, and total_chunks unconditionally in the response. Only highlighted_page_image stays gated on its own truthiness. - Add a chunk_index forwarding regression test that asserts the new kwargs reach get_chunk_with_context and appear in the response payload. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
a33a365a69 |
fix(viz_routes): validate chunk_index/total_chunks bounds in OAuth route
PR #767 review noted that the OAuth viz route used bare int() parsing for chunk_index and total_chunks while the bearer-token visualization route validates them via _parse_int_param. Mirror the same bounds check so total_chunks=0 and negative chunk_index return 400 instead of silently suppressing adjacent-chunk context. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
90458b6f08 |
fix(chunk-context): use indexed chunk_index lookup, fix close-after-use bug
Two related bugs surfaced in production while viewing chunks from the Astrolabe frontend on the AWS-hosted MCP server: 1. PyMuPDF document closed: in _fetch_document_text the fallback path referenced pdf_doc.page_count after pdf_doc.close(), raising "document closed" and returning None. The slow PDF re-parse already completed but its result was discarded. Capture page_count into a local before close(). 2. Slow/fragile chunk lookup: get_chunk_with_context filtered Qdrant by (chunk_start_offset, chunk_end_offset). Those fields are not part of the always-indexed payload schema, and with strict_mode enabled they yield 400 errors. Even with manually-added indexes the filter is fragile if a doc is re-chunked. Switch to chunk_index (always indexed) as the primary lookup key, falling back to offset-based lookup when callers don't supply it. Plumb chunk_index/total_chunks through both the management API (api/visualization.py) and the OAuth viz route (auth/viz_routes.py). Apply the same change to the highlighted-image lookup so all four chunk-context Qdrant queries prefer the indexed field. Skip the slow PDF re-parse fallback entirely for files: when both the chunk_index and offset Qdrant lookups miss, re-downloading and re-parsing the source PDF won't find the chunk either, and routinely exceeds 30s on large documents - which is the proxy timeout in Astrolabe. Notes/cards keep the document-fetch fallback (cheap). Removes dead code (_get_file_path_from_qdrant) that was only used by the now-unreachable file fallback path. Companion change in the Astrolabe app passes chunk_index from search results through to the new endpoint params. --- _This PR was generated with the help of AI, and reviewed by a Human_ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
b875eaf069 |
fix(auth): address PR #758 round-7 medium/minor review
- Gate browser session creation on a successful refresh token. When the IdP returns no refresh token, SessionAuthBackend would silently reject every subsequent request and bounce the user back to /oauth/login in a loop. The callback now bails with a 400 + correlation ID + actionable hint about offline_access *before* writing browser_sessions or setting the cookie. Pinned by a new end-to-end unit test. - Evict orphaned browser_sessions rows in SessionAuthBackend when the associated refresh token is gone, instead of letting them accumulate until TTL cleanup. Best-effort; deletion errors stay non-fatal. - Demote identity-bearing logs in the Flow 2 OAuth callback (user_id, scopes, audience, expires_at) from INFO to DEBUG so they don't leak into multi-tenant log aggregation on every provision. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
27fcf05d3a |
fix(auth): address PR #758 round-7 important review
- Coerce refresh_expires_in to int before arithmetic in both callback paths so IdPs that serialize the field as a JSON string (e.g. AWS Cognito) don't trigger an unhandled TypeError 500. - Drop the orphaned oauth_session row written by _check_logged_in. The canonical Flow 2 row is created by generate_oauth_url_for_flow2 keyed by `state`, which is what the unified callback looks up; the flow2_<hex> session_id was never matched and just churned the table for 10 minutes per call. - Match delete_cookie attributes (httponly, secure, samesite) to the set_cookie call on logout so browsers reliably evict the cookie even on implementations that consider security flags during deletion. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
ec9b9b2a75 |
fix(auth): address PR #758 round-6 medium/low review
Five findings from the latest review on #758 (2 medium, 3 nit): Medium: - browser_oauth_routes.oauth_login_callback + oauth_routes.oauth_callback_nextcloud: fail closed with 400 when the oauth_session row is unknown/expired. Previously both callbacks fell through with code_verifier="" and expected_nonce=None, silently bypassing the PKCE + nonce protections introduced in earlier rounds. Symmetric unit tests pin both contracts. - token_utils.verify_id_token: use secrets.compare_digest for the nonce check instead of short-circuit !=. Mirrors the sibling PKCE verifier comparison; closes the last secret-equality timing-side-channel surface in the auth path. Nit: - Tighten the comment at all 4 mcp_authorization_code/code_verifier store + retrieve sites so a future refactor sees the field reuse immediately (renaming the column requires a schema migration). - _should_use_secure_cookies: explicit string normalisation instead of bool(settings.cookie_secure). Dynaconf normally coerces but tests / direct settings.set calls can leave the raw string in place — bool("false") is True. New parametrized unit tests cover the coercion matrix + http/https fallback. - oauth_routes.py:591 f-string log converted to lazy %s formatting (folded into the Flow 2 callback rewrite). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
e2955e8246 |
fix(auth): address PR #758 round-5 medium/low review
Three findings from the latest review on #758 (1 medium, 2 low): Medium: - browser_oauth_routes.oauth_logout: move delete_browser_session into a finally block so an error from delete_refresh_token can no longer leave an orphan browser_sessions row. The orphan was not exploitable (SessionAuthBackend rejects sessions without a live refresh token), but it lingered until the hourly cleanup cron — a correctness gap. New regression test pins the fix. Low: - oauth_callback_nextcloud: drop redundant ``or None`` from ``expected_nonce=nonce``. ``nonce`` is already ``str | None`` and ``secrets.token_urlsafe`` never produces an empty string, so the coercion was a no-op that could mislead future readers into thinking empty-string was a valid skip-the-check path. - storage.RefreshTokenStorage.initialize: fail fast at startup when SQLite < 3.35, since ``DELETE ... RETURNING`` (used in ``delete_browser_session``) needs that minimum. Ubuntu 20.04 ships 3.31 and would otherwise hit OperationalError on every logout. Prerequisite also documented in docs/installation.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
b696541918 |
fix(auth): address PR #758 round-4 review
Seven findings from the latest review on #758 (3 medium, 4 low/nit): Medium: - storage.py: replace 5 ``assert self.cipher is not None`` sites with explicit ``RuntimeError`` so missing TOKEN_ENCRYPTION_KEY can't silently become an AttributeError under ``python -O`` - session_backend.py: document the silent-invalidation invariant — refresh-token TTL expiry without explicit logout deliberately makes the browser session unusable; future readers must not relax it - server/oauth_tools.py: drop user_id from the Flow 2 session_id identifier — use ``flow2_{secrets.token_hex(16)}`` so audit logs and DB rows don't carry user_id in the session_id field Low / nit: - token_utils.py: drop _fetch_locks dict entry in finally so a probed deployment can't grow the lock dict without bound; coalescing test now pins the invariant with len(_fetch_locks) == 0 - browser_oauth_routes.py: strip trailing slash from settings.nextcloud_host before constructing the well-known URL so a host configured as ``https://cloud.example.com/`` doesn't produce a double-slash - browser_oauth_routes.py: add comment explaining the three-layer CSRF policy on the mcp_session cookie set (SameSite=Lax + POST-only logout + Origin/Referer check) - oauth_routes.py: convert all 23 f-string log calls to lazy %-style per the CLAUDE.md / memory feedback_lazy_logging convention Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
3a4fa8adc8 |
fix(auth): address PR #758 round-3 final review
Seven findings from the latest review on #758, plus a regression test catching the substance of the cache-stampede fix: - verify_id_token: widen id_token annotation to str | None to match callers passing nc_token_response.get("id_token") - extract_user_id_from_token: use JSON-RPC reserved error code -32001 instead of -1 - _get_cached: per-URL anyio.Lock dict + meta-lock coalesces concurrent cache misses into a single IdP fetch (mirrors token_broker.py idiom) - delete_browser_session: collapse SELECT+DELETE into atomic DELETE ... RETURNING user_id (SQLite >= 3.35) - new test_origin_normalise.py: parametrized port/scheme/host equivalence cases for the CSRF Origin guard - browser_oauth_routes: correct misleading "PR #758 finding 5" cross- references (finding 5 was Fernet-key hardening, not CSRF) - ASProxySession.nonce: make required, drop spurious "legacy session" default; reword the in-flight `or None` comment to reflect that ASProxySession is purely in-memory - new test_get_cached_coalesces_concurrent_misses: pins the cache-stampede protection — fires 10 concurrent _get_cached calls and asserts exactly one HTTP fetch Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
9d0e7dcebe |
fix(auth): address PR #758 round-3 review
- Flow 2 (oauth_authorize_nextcloud) now generates a nonce, stores it on the oauth_session row, forwards it to the IdP, and verifies it via expected_nonce in oauth_callback_nextcloud — closes the last replay- protection gap (round-3 finding 1). - _origin_matches_self fails closed when mcp_server_url is missing instead of allowing the logout, and the diagnostic log is promoted from warning to error so the misconfiguration is monitorable (round-3 finding 2). New regression test pins the new behaviour. - The five user_id-accepting helpers in oauth_tools.py (get_provisioning_status, provision_nextcloud_access, revoke_nextcloud_access, check_provisioning_status, check_logged_in) are renamed with leading underscores to make the trust boundary structural rather than documentary (round-3 finding 3). - create_browser_session and delete_browser_session now emit audit_log rows so session establishment / teardown match the pattern used by the rest of the security-relevant storage operations (round-3 nit 5). delete_browser_session selects user_id before delete so the audit row is attributable. - oauth_login_callback no longer reflects raw IdP-error text or exception strings into the HTML failure page; users see a generic "internal error occurred" message + a correlation ID, with the detail logged server-side keyed by the same ID (round-3 nit 6). The XSS regression test is updated to pin the stricter contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
c33d52ea91 |
fix(auth): address PR #758 round-2 review
- oauth_login_callback's integrated-mode token-exchange branch now reuses the shared discovery cache via get_oidc_discovery (round-2 finding 1). - AS proxy flow now generates an OIDC nonce in oauth_authorize, stores it on ASProxySession, forwards it to the IdP, and passes it as expected_nonce to verify_id_token in _oauth_callback_as_proxy (round-2 finding 2). - Consolidate the two parallel discovery caches: oauth_routes' local _discovery_cache and _get_cached_discovery are removed; all callers now go through token_utils.get_oidc_discovery, which acquires the follow_redirects=True knob it needs for Nextcloud installs without pretty URLs (round-2 finding 3). - Demote per-user INFO logs in oauth_tools.py (check_logged_in, get_provisioning_status) to DEBUG; the elicitation auth URL is no longer logged because it contains a sensitive state token (round-2 finding 4). Also pin nonce binding behaviour with a new unit test that asserts _oauth_callback_as_proxy forwards session.nonce to verify_id_token, and update test mocks to track the cache consolidation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
4c84d82984 |
fix(auth): address PR #758 auto-review (id-token verify, nonce, CI key)
Blocking: - AS proxy callback now calls verify_id_token before caching the proxy code so a tampered IdP response can't smuggle identity claims. Important: - Browser OAuth flow generates and verifies an OIDC nonce; new alembic migration 006 adds the nonce column to oauth_sessions. - _origin_matches_self logs a warning when CSRF check is bypassed. - oauth_tools.py uses get_shared_storage instead of fresh handles. Nits: - New token_utils.get_oidc_discovery shares the 5-minute cache with verify_id_token; oauth_login (integrated) and _revoke_refresh_token_at_idp now use it instead of issuing fresh discovery fetches. - Drop typing.Optional from oauth_tools.py in favour of X | None. CI: - test.yml generates an ephemeral Fernet TOKEN_ENCRYPTION_KEY per run with openssl, removing the dependency on a missing repo secret. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
2ef4bfc4af |
fix(auth): fail closed on missing sub claim, delete Flow 2 callback session
Addresses the two remaining 🟡 findings from the PR #758 follow-up review: 1. extract_user_id_from_token previously fell back to "default_user" when the verified access token had no sub claim. In a multi-tenant deployment a malformed IdP token could have bucketed every request under a single sentinel user, risking cross-tenant data exposure. The function now raises McpError on that branch; the BasicAuth no-token sentinel path is preserved. 2. oauth_callback_nextcloud (Flow 2) read the PKCE code_verifier from oauth_sessions but never deleted the row, leaving the verifier valid for the full 10-minute TTL. The row is now deleted eagerly inside the same branch, mirroring oauth_login_callback in browser_oauth_routes. Also wires TOKEN_ENCRYPTION_KEY through the docker-compose step in the CI test workflow so the integration matrix can boot — every job had been failing fast on the ${TOKEN_ENCRYPTION_KEY:?...} interpolation guard added in PR #758 finding 5. Tests pin both fixes (test_token_utils_user_id.py, test_oauth_callback_session_cleanup.py). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
2d340a5a6b |
fix(auth): address PR #758 follow-up review
Six findings from the latest claude-bot review on PR #758: - JWKS cache had no kid-miss refresh path (Medium): on IdP key rotation every login failed for up to _OIDC_CACHE_TTL. Evict and refetch once before raising, per OIDC core §10.1.1. - _should_use_secure_cookies fell back to nextcloud_host scheme, but the cookie is issued by the MCP server. Switch to settings.nextcloud_mcp_server_url so split-scheme deployments get the right Secure flag. - _origin_matches_self compared raw netloc strings, which include the port. Browsers omit default ports per RFC 6454 §6.2; an mcp_server_url like :443 falsely 403'd every legitimate logout. Normalise (scheme, host, port) tuples with default ports stripped. - delete_oauth_session exists in storage.py — drop the stale "we don't have this method" comment and call it eagerly so replays can't be processed and the table doesn't accumulate completed-but-not-yet-expired browser-login rows. - extract_user_id_from_token's unused ctx param renamed to _ctx to signal "intentionally unused" at the signature level. - provisioning_decorator instantiated RefreshTokenStorage per call. Switch to get_shared_storage() for the lock-protected process-wide singleton. Plus pre-push self-review catch: lazy-logging on the unchanged except arm in session_backend.py. Adds 5 regression tests: - JWKS rotation: success on refetch - JWKS rotation: still-missing-kid surfaces original error - JWKS rotation: network error during refresh wrapped as IdTokenVerificationError - default-port CSRF: explicit :443 in config + portless Origin - default-port CSRF: portless config + explicit :443 in Origin - scheme-mismatch CSRF: same host, different scheme rejected Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
af25c281bf |
fix(auth): use Settings for OIDC env vars in token revocation helper
After merging master, _should_use_secure_cookies was refactored to read from Settings instead of os.getenv, which dropped `import os` from browser_oauth_routes.py — leaving _revoke_refresh_token_at_idp's four remaining os.getenv() calls undefined (CI ruff F821). Migrate the helper to the same Settings-based pattern: - oidc_discovery_url → settings.oidc_discovery_url - OIDC_CLIENT_ID → settings.oidc_client_id - OIDC_CLIENT_SECRET → settings.oidc_client_secret - NEXTCLOUD_HOST → settings.nextcloud_host Drive-by: the previous fallback read OIDC_CLIENT_ID, but the canonical env var per env.sample / docker-compose is NEXTCLOUD_OIDC_CLIENT_ID. The Settings layer handles this mapping via dynaconf, so the corrected name is now used automatically. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris Coutinho
|
50a97ffcb3 | Merge remote-tracking branch 'origin/master' into security/oauth-session-hardening-626 | ||
|
Chris CoutinhoandClaude Opus 4.7
|
931ee602eb |
fix(auth): address PR #758 review — XSS, CSRF, open redirect, JWKS cache
Addresses all 9 findings from the review on PR #758: Blocking: - _revoke_refresh_token_at_idp now reads config from oauth_ctx["config"] (the production-shaped nested dict). Previously read flat keys, causing IdP revocation to silently no-op in production. Test fixtures rebuilt to the realistic nested shape so the bug can't regress unnoticed. - HTML error responses in oauth_login_callback now wrap IdP-controlled error_body, str(e), and the attacker-controlled error/error_description query params in html_escape. New test_browser_oauth_xss.py pins this. Important: - New _safe_next_url helper validates the ?next= query param at write time (oauth_login), in oauth_logout, and on read from the session row in oauth_login_callback. Blocks https://, // (protocol-relative), and CRLF/whitespace injection. - verify_id_token now caches discovery + JWKS (5-min TTL) using the same pattern as oauth_routes._get_cached_discovery. New caching regression test pins to one fetch per URL across multiple calls. - /oauth/logout is now POST-only at the route layer (defeats passive CSRF via <img src>). oauth_logout also validates Origin/Referer against the configured mcp_server_url. Logout UI in user_info.html converted from <a href> to <form method="post">. - New storage.cleanup_expired_browser_sessions() called from the hourly cleanup loop in app.py — previously these rows accumulated for users who never explicitly logged out. Nits: - Demoted INFO logs that leaked oauth_config.keys() / client_id / token-storage state to DEBUG. Operator-relevant outcome lines (login successful, refresh token stored, logged out) stay INFO. - verify_id_token algorithms widened to RS256, PS256, ES256 — covers Azure AD (PS256) and Cognito/some Keycloak realms (ES256). Symmetric and "none" remain off the allowlist. - Migrated all Optional[X] usages in auth/storage.py to X | None per CLAUDE.md. Breaking change: GET /oauth/logout now returns 405. The in-tree logout UI was migrated to a POST form; any external bookmark or curl-based caller that relied on GET will need to switch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
ce80a36877 |
fix(auth): address PR #757 round-3 review feedback
Three review items from the third-round review on PR #757:
- scope_authorization: split the combined logger.warning(error_msg) in
the require_scopes decorator's missing-app-password branch into two
lazy %-style logger calls (one per branch), keeping the f-string
error_msg for the exception only. The else branch also logs the
elicit_result for diagnostics. Bypassing lazy %-interpolation in
security-sensitive code formatted the message regardless of log level
and matched the repo-wide lazy-logging preference; the new code now
conforms.
- config + browser_oauth_routes: wire COOKIE_SECURE through Settings
(cookie_secure: bool | None = None) so _should_use_secure_cookies()
reads it via get_settings() rather than os.getenv. Completes the
consolidation pass that touched this file in commit
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
15dbb26349 |
fix(auth): harden OAuth/session for hosted multi-tenant deployment (#626)
Pre-launch hardening for the hosted Astrolabe Cloud offering. Addresses all five findings raised in #626 (Tim Kaufmann, code review of v0.65.0). Re-verified against master before fixing. Finding 3 (LLM-controllable user_id) — drop user_id from the public signatures of provision_nextcloud_access, revoke_nextcloud_access, check_provisioning_status, check_logged_in. Tool wrappers now always derive identity from the verified AccessToken; user_id is no longer accepted as MCP input. Adds parameterized CI-guard test that locks the schema. Finding 2 (predictable session cookie) — replace mcp_session=<user_id> cookie with a cryptographically random session_id mapped server-side (new browser_sessions table, alembic 005). Cookie value is opaque, expires, revocable. SessionAuthBackend looks up user_id via the new mapping and additionally requires a refresh token to fail closed. Finding 4 (logout doesn't revoke refresh token) — oauth_logout now calls the IdP revocation_endpoint (RFC 7009) when advertised, deletes the stored refresh token regardless, and clears the browser_sessions row. Cleanup is best-effort: logout always 302s. Finding 1 (unverified ID token decodes) — verify_id_token helper does JWKS signature + issuer + audience + exp + nonce checks per OIDC core 3.1.3.7. Used by both OAuth callback handlers (browser + MCP). Removes the four "verify_signature: False" decodes that previously trusted IdP claims unconditionally. Drops dead-code _validate_token_audience in token_broker. Refactors token_utils + provisioning_decorator to read user_id from the verified AccessToken instead of re-decoding the JWT. Finding 5 (hardcoded Fernet keys in docker-compose.yml) — replace the three inline TOKEN_ENCRYPTION_KEY values with required env var interpolation; document in env.sample. Test coverage: 4 new unit test modules (signature pinning, browser sessions, ID-token verification, logout + revoke + session backend). 693 unit tests pass; ruff/format/ty clean. Migration note: existing browser admin-UI sessions become invalid on rollout (cookies are looked up against the new browser_sessions table, which starts empty). Users re-login. MCP API access is unaffected. Tracked on Astrolabe Cloud POC board card #37. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
f31d0544b7 |
fix(auth): invalidate scope cache on web/REST provisioning paths
The elicitation flow points users to the Astrolabe web route or the BasicAuth REST endpoint to provision their app password. Both paths stored the password without clearing the in-process scope cache, so a user who provisioned through them would keep hitting ProvisioningRequiredError for up to _SCOPE_CACHE_TTL (5 min) afterwards. Add invalidate_scope_cache(user_id) to both write-paths (matching the existing pattern in nc_auth_check_status), correct the now-misleading comment in scope_authorization.py to name all three invalidation paths, and add a one-line hint above the first elicitation patch in the test file so future authors don't "fix" the patch target to the wrong module. Addresses PR #757 round-3 review feedback. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |