Commit Graph
100 Commits
Author SHA1 Message Date
Chris CoutinhoandClaude Opus 4.7 5f01312cc4 docs: correct OAuth enum value and split boot-log block by mode
Reviewer found two accuracy issues in the rewritten "Check Deployment
Mode" section:

- The AuthMode.OAUTH_SINGLE_AUDIENCE enum value is `oauth_single`, not
  `oauth_single_audience` (config_validators.py:28). A user grepping
  their container logs would have found nothing.
- The "Configuring MCP server for <mode> mode" line was presented as a
  uniform <mode> substitution, but app.py:1170 hardcodes the literal
  string `OAuth mode` for OAuth, while app.py:1239 uses the enum value
  for the two BasicAuth modes.

Split the boot-time block into per-mode subsections so each one shows
the actual literal text users will see, and add a one-line note
calling out the OAuth string difference.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 07:43:05 +02:00
Chris CoutinhoandClaude Opus 4.7 e78e191818 docs: fix /health → /health/live, refresh stale mode-detection log examples (#766)
Issue #766 reported that the running.md quick-start tells users to
`curl http://localhost:8000/health`, which returns 404 — the server
only registers `/health/live` and `/health/ready` (K8s-style probes).
The same section also listed BasicAuth and OAuth startup log lines
(`BasicAuth mode detected …`, `OAuth mode detected …`) that no longer
exist anywhere in the codebase.

Update running.md and troubleshooting.md to point at the real
endpoints, explain liveness vs readiness, and replace the fictional
log examples with messages the server actually emits today. Also
clarify that the per-session BasicAuth messages only appear after the
first MCP client connects, which is the second symptom the reporter
hit.

Docs-only change; code paths and endpoint surface unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 07:35:49 +02:00
Chris CoutinhoandClaude Opus 4.7 9d5ac01f24 fix(calendar): preserve floating/TZID semantics across CalDAV roundtrip (#782)
The CalDAV REPORT in `_search_events_by_date` unconditionally requested
server-side `<C:expand>`. Per RFC 4791 §9.6.5 the server then normalizes
every expanded DTSTART/DTEND to UTC `Z`, which destroyed two pieces of
information on the read path:

- RFC 5545 floating local times came back as fake-UTC (a `+00:00` suffix
  that did not match the stored value), so a 2:30 PM floating event was
  indistinguishable from a 14:30 UTC event in the MCP response.
- TZID-bound events lost their IANA TZID context — a "10am America/New_York"
  event came back as `14:00:00+00:00`, making it impossible for callers to
  reconstruct DST-aware recurrence semantics.

Replace `<C:expand>` with client-side recurrence expansion via the
`recurring-ical-events` library (promoted from transitive to direct dep),
so the wire response retains its original DTSTART format. Surface the
TZID parameter as new `start_tz`/`end_tz` fields on `CalendarEventSummary`.

Add an optional `timezone` (IANA name) parameter to `nc_calendar_create_event`
and `nc_calendar_update_event` so callers can pin a TZID for naive input;
the helper attaches `ZoneInfo(...)` and emits a paired `VTIMEZONE`
component. Naive input without `timezone` continues to store as RFC 5545
floating local time (with a warning logged). Offset-aware input continues
to store as UTC `Z`.

Drive-by: switch the update path's DTSTART/DTEND assignment from raw
`datetime` to `vDDDTypes(dt)` wrappers — the previous code produced invalid
iCal like `DTSTART:2026-05-14 10:00:00+00:00` for any TZ-aware update.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 02:41:10 +02:00
Chris Coutinho 9072559d26 chore: Address reviewers feedback 2026-05-11 00:22:38 +02:00
Chris CoutinhoandClaude Opus 4.7 271904c227 fix(deck): address review — notesPath key, scopes, modernize types
PR #781 review round 1:

- 🔴 Fix notesPath key: the Notes API returns the folder under camelCase
  ``notesPath`` (see models/notes.py:43), but `deck_attach_note` was
  looking up snake_case ``notes_path`` and silently falling back to
  ``"Notes"``. Users with a non-default notes folder would have produced
  shares pointing at non-existent files (404 on click in Deck UI).

- 🔴 Add wire-through unit test that would have caught the above:
  extract `_resolve_note_attach_path(client, note_id)` as a testable
  helper that encapsulates the camelCase-key lookup. Three new tests:
  custom notesPath honored, missing key falls back to default, null
  category handled.

- 🟡 Modernize new fields on `DeckAttachmentExtendedData` to PEP 604
  (`X | None`) per CLAUDE.md.

- 🟡 Drop unnecessary string forward reference on
  `ListAttachmentsResponse.results` — DeckAttachment is defined earlier
  in the same module.

- 🟢 Move `pytestmark = pytest.mark.unit` to module level in
  test_sharing_client.py to match the convention in test_deck_server.py.

Per user request: `deck_attach_file` is now scoped `deck.write` +
``files.read`` (was just `deck.write`) so the generic file-share
permission story is consistent — only `deck_attach_note` keeps
`notes.read` since it specifically reads from the Notes app. Docstring
updated to emphasise the tool is generic over the user's Files
(PDFs/images/etc., not just markdown).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 23:28:37 +02:00
Chris CoutinhoandClaude Opus 4.7 c0a974c498 feat(deck): add file/note attachment MCP tools
Adds four tools that expose Deck card attachments via the MCP surface:
deck_attach_file, deck_attach_note, deck_list_attachments, and
deck_delete_attachment. The attach* variants share an existing Files
entry (or Notes-app note) with the card via OCS shareType=12 — same
mechanism the Deck UI's "Share from Files" picker uses, no file copy.

This replaces the prior workaround of appending bulky activity content
as Deck card comments: per-PR/per-event narrative now lives in NC Notes
and surfaces on the tracking card as a clickable attachment that opens
the original note in place.

Implementation reuses existing client methods (SharingClient.create_share,
DeckClient.get/delete_attachment, NotesClient.get_settings/get_note);
no new client code. _SHARE_TYPE_DECK is centralised with a CI-guard test
to prevent silent drift, and SharingClient.create_share's wire format is
pinned to what the Deck Vue source sends.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 23:17:20 +02:00
Chris CoutinhoandClaude Opus 4.7 9cb0b33ec1 fix(health): forward api-key to Qdrant /readyz so Cloud probes work
The readiness handler called the configured `qdrant_url/readyz` with a
bare httpx.AsyncClient — no headers. That works against a self-hosted
Qdrant (where /readyz is anonymous), but Qdrant Cloud's auth gateway
returns 403 for any unauthenticated request, including /readyz, /livez
and /healthz. Result: every probe against a Cloud cluster fell into
the "status 403" branch, the handler returned 503, and the Pod never
went Ready — even when the configured `AsyncQdrantClient` itself was
authenticating fine for actual collection traffic.

Forward `settings.qdrant_api_key` as the `api-key` header (mirroring
what `vector/qdrant_client.py:540` already does for the real client).
When the key is unset (self-hosted, anonymous case) we send no header,
so existing self-hosted deployments are unchanged.

Verified end-to-end against Qdrant Cloud:
- Without header: GET /readyz -> 403 {"error":"forbidden"}
- With api-key:  same request shape returns 200 (matches what
  AsyncQdrantClient.wait() relies on internally).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 20:15:05 +02:00
Chris Coutinho f3a66cf6bb chore: ruff format 2026-05-10 18:37:27 +02:00
Chris Coutinho 189024f1ca Merge remote-tracking branch 'origin/master' into fix/qdrant-doc-id-keyword-index 2026-05-10 18:37:07 +02:00
Chris CoutinhoandClaude Opus 4.7 8246d9a088 fix(vector): address PR review round 17 + local-mode collection-creation regression
Round 17 reviewer (🟡 Important):

1. docs/configuration.md degraded-migration runbook said `doc_id backfill
   failed on …` but the actual log line in qdrant_client.py:415 is
   `doc_id backfill scroll failed on …`. Operators grepping the runbook
   string would have missed it. Insert the `scroll` qualifier.

2. _create_one_payload_index returned True on the 400 schema-conflict
   path, so a wrong-type index discovered at create time skipped the
   consolidated `Payload index creation incomplete` summary — but a
   wrong-type index discovered via the existing-schema check at line
   195-206 did fire it. Tenants whose payload_schema is hidden from
   their JWT (Qdrant Cloud collection-scoped tokens) only ever observe
   the create-time path, so they never saw the operator-level summary.
   Return False so the summary fires in both cases.

3. docs/configuration.md said the upgrade-time delay was `proportional to
   point count while writes are issued` — overstating the cost. Writes
   are proportional to int-typed points only; the scroll itself is
   proportional to total point count. Reword.

Local-mode collection-creation regression (root-cause of failing
single-user / login-flow / multi-user-basic CI jobs):

PR #779 changed the existence probe in get_qdrant_client from
collection_exists() (returned bool in both modes) to get_collection()
+ except UnexpectedResponse(status_code=404). The HTTP-mode client
raises UnexpectedResponse with a 404 body, but the local/in-memory
client raises ValueError(f"Collection {name} not found") — see
qdrant_client/local/async_qdrant_local.py. The narrow except clause
let the ValueError propagate, app.py's lifespan re-raised as
RuntimeError, and the mcp container crashed on first start. Catch
ValueError too, with a `not found` substring guard so genuine
programming bugs (bad collection_name, etc.) still surface.

Tests: extend the existing 400-path test to assert the new
failed_fields contract; add two get_qdrant_client unit tests pinning
the local-mode VE catch (positive case + propagation case).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 18:33:51 +02:00
Chris Coutinho 363c2a2624 Merge remote-tracking branch 'origin/master' into fix/qdrant-doc-id-keyword-index
# Conflicts:
#	nextcloud_mcp_server/vector/qdrant_client.py
2026-05-10 17:55:53 +02:00
Chris CoutinhoandClaude Opus 4.7 8f4f5c0079 fix(vector): address PR review round 16 — type-aware index check, comments
Detect pre-existing payload indexes with the wrong schema type in
`_ensure_payload_indexes`. The previous "field already in
existing_schema → skip" branch silently survived a collection migrated
from the int-doc_id era where `doc_id` is indexed as INTEGER, letting
`MatchValue(value="123")` searches keep failing with HTTP 400 on Qdrant
Cloud strict mode — exactly the production failure this PR was meant to
fix. New behaviour: compare `existing_schema[field].data_type` against
the declared type; on mismatch log a WARNING and append to
`failed_fields` so the consolidated end-of-function summary picks it up.
No auto-repair (operator intervention only — see docs/configuration.md
recovery procedure). New test exercises the doc_id-INTEGER scenario
end-to-end and asserts both the per-field WARNING and the summary line.

Clarify the `_verify_news_items` malformed-doc_id rationale: the news
API has no per-item endpoint, so a malformed doc_id genuinely cannot be
verified against the source of truth. We err toward false-positive
(keep) over false-negative (drop) — same conservative posture as
`_verify_notes` and `_verify_deck_cards`. The producer-side validation
is the real security boundary; the verifier is defence-in-depth. Both
the inline comment and the WARNING message now spell this out.

Add a TODO in `get_last_indexed_timestamp` flagging the O(N) cost on
every incremental sync tick. The previous single-page `limit=10_000`
silently bounded the scroll; paginating fixed correctness but made the
unbounded cost visible. The follow-up tracker (canonical TODO at
`api/visualization.py`) covers migrating the max-`indexed_at` to a
sentinel point or collection metadata for O(1) lookup.

Consolidate the duplicate non-numeric-doc_type TODOs at
`api/visualization.py:508` and `auth/viz_routes.py:570` into a single
canonical comment in `visualization.py`; `viz_routes.py` is reduced to
a back-reference. Removes the rot risk of "fixed in one place,
forgotten in the other." The canonical comment also references the
O(1) timestamp follow-up in `scanner.py`.

Document the `batch_size = 256` (qdrant_client.py) vs
`_DELETION_TRACKING_PAGE_SIZE = 1024` (scanner.py) split with
cross-referencing comments at each site: the smaller batch is for the
read-write backfill upsert path (Qdrant accepts ~256-point chunks
comfortably); the larger page is for read-only deletion-tracking
scrolls where no per-page write round-trip applies.

Replace `assert qdrant_client is not None` in `scan_user_documents`
with `cast(AsyncQdrantClient, qdrant_client)` plus an explanatory
comment. `assert` is silently elided under `-O`; `cast` is the
conventional zero-cost narrower for branches the type checker can't
infer from the surrounding `if not initial_sync` ternary.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 17:50:23 +02:00
Chris CoutinhoandClaude Opus 4.7 04bc2325f2 fix(qdrant): use get_collection for startup probe (multi-tenant safe, take 2)
Follow-up to PR #778 — `collection_exists()` is also denied by Qdrant
Cloud on a collection-scoped JWT, so the multi-tenant fix needs to go
one step further: use `get_collection(name)` (the underlying GET
`/collections/{name}` call) and treat a 404 `UnexpectedResponse` as
the "doesn't exist" signal. That endpoint is the only existence-probe
Qdrant permits on a collection-scoped JWT — listing or probing
collection metadata cluster-wide is a tenant-isolation boundary by
design.

Hit during Astrolabe Cloud smoke17 with the post-#778 image:

    qdrant_client.http.exceptions.UnexpectedResponse: 403 (Forbidden)
    raw response: {"error":"forbidden"}
    File "qdrant_client.py", line 84, in get_qdrant_client
        collection_present = await _qdrant_client.collection_exists(...)

Folds the existence check into the same `get_collection()` call that
already runs immediately afterward for dimension validation, so the
new path is also one fewer round-trip on the happy path.

Cold-start (collection genuinely missing) behavior is unchanged: 404
→ `collection_info` is None → fall through to `create_collection()`.
Whether `create_collection` succeeds is an orthogonal concern (managed
multi-tenant setups pre-provision collections externally; admin-key
single-tenant setups can create on the fly).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 14:19:11 +02:00
Chris CoutinhoandClaude Opus 4.7 f2d4982b2f fix(qdrant): use collection_exists for startup probe (multi-tenant safe)
The startup path in `get_qdrant_client()` calls `get_collections()` to
check whether the configured collection already exists. That's a
cluster-wide list operation; in managed multi-tenant Qdrant Cloud
deployments where each tenant's JWT is scoped to a single collection
(by design — `access: [{"collection": "tenant_<id>", "access": "rw"}]`),
the call returns `403 Forbidden` and the FastAPI lifespan crashes:

    qdrant_client.http.exceptions.UnexpectedResponse: 403 (Forbidden)
    raw response: {"error":"forbidden"}
    RuntimeError: Cannot start vector sync - Qdrant initialization failed

Switching to `collection_exists(collection_name)` (per-collection
HEAD-style probe) only requires access to the named collection, which
the tenant JWT has. Single-tenant deployments using an admin/master
key are unaffected — they had access to both forms; this picks the
narrower one.

Doesn't change creation semantics: when the collection isn't present
the code path still calls `create_collection`. In a managed setup
where the collection is pre-provisioned by an external admin (e.g.,
the Astrolabe Cloud control plane's create-tenant workflow), that
branch never fires for an existing tenant; cold-start tenants get
their collection created by the workflow before the Pod boots.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 14:03:48 +02:00
Chris CoutinhoandClaude Opus 4.7 68506f96c5 fix(vector): address PR review round 15 — concurrency, pagination, stale coercion
Defer publication of `_qdrant_client` until after the in-lock backfill +
payload-index migration awaits complete. The fast-path check at the top of
`get_qdrant_client` reads the singleton without holding the init lock, so
publishing the constructed-but-unmigrated client let concurrent fast-path
callers fire filtered searches before `_ensure_payload_indexes` ran —
producing HTTP 400 ("Index required but not found") on Qdrant Cloud strict
mode. Local `provisional` is now used for every await inside the lock; the
global is assigned exactly once, last.

Replace the five hand-rolled `scroll(..., limit=10000)` calls in
`vector/scanner.py` (notes / files / news / deck-cards deletion tracking,
plus the timestamp scroll) with a single paginated `_scroll_all_points`
helper. The previous single-page cap silently dropped deletion-tracking
points beyond the first 10 k for any user past that threshold. Pagination
follows Qdrant's documented contract (loop until `next_page_offset is
None`) with a fixed per-page `_DELETION_TRACKING_PAGE_SIZE = 1024`.

Extract `_create_one_payload_index` from `_ensure_payload_indexes` to drop
its cognitive complexity below the SonarQube limit (17 → ≤ 15) without
losing the per-field error-containment rationale; every comment is
preserved verbatim on the helper.

Drop the stale `SearchResult.id` `int | str` comment and the redundant
`str(d)` coercion in `_verify_news_items` — the contract has been
str-only since the producer-side stringification landed earlier in this
PR.

Fix eight `doc_id=<int>` test calls in `test_chunk_context_offset_gate.py`
that violated the `doc_id: str` signature of `get_chunk_with_context`,
plus align `_make_result` in `test_verification.py` to coerce `id=str(...)`
matching the production contract — and update 30+ assertions from int
sets (`{1, 2, 3}`) to str sets (`{"1", "2", "3"}`) so the tests now model
the post-PR `SearchResult.id: str` reality end-to-end. Previously these
were masked by the `str(d)` coercion now removed from production.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 13:49:17 +02:00
Chris CoutinhoandClaude Opus 4.7 c5020d9629 fix(vector): address PR review round 14 — accurate offset-skip comment + news_item doc_id guard
Round-14 review surfaced one blocking and one important issue.

context.py: the comment justifying `skip_offset_lookup` claimed
chunk_start/end_offset weren't in _PAYLOAD_INDEX_FIELDS — round 13
indexed both as INTEGER, so the comment now actively misleads. Replace
with the real reason: an indexed chunk_index miss is canonical (both
paths hit the same Qdrant collection), and skipping the offset filter
avoids a redundant round-trip.

verification.py: hoist an is_valid_nextcloud_doc_id guard before the
`int(d)` cast in _verify_news_items, mirroring the boundary-validation
pattern already in _fetch_document_text. Coerce via `str(d)` because
SearchResult.id is `int | str` (D1 forward-compat widening). Malformed
ids now surface as a logger.warning rather than a generic debug line;
fail-open semantics are preserved.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 11:50:46 +02:00
Chris CoutinhoandClaude Opus 4.7 ae23bbe8b8 fix(vector): address PR review round 13 — index offset fields + tighten test
- Add chunk_start_offset / chunk_end_offset to _PAYLOAD_INDEX_FIELDS so
  the legacy offset-based fallback in search/context.py works on Qdrant
  Cloud strict mode (pre-#75 clients have no chunk_index payload).
- Cover chunk_index / chunk_start_offset / chunk_end_offset in the
  payload-index summary test; refresh the stale field-list comment.
- Flag the is_valid_nextcloud_doc_id gate at both chunk-context handler
  sites with a TODO for future non-numeric doc_types.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 10:27:57 +02:00
Chris CoutinhoandClaude Opus 4.7 f9ad7dc52e fix(vector): address PR review round 12 — bool guard + strict doc_id validation
- _group_int_doc_ids: use type(value) is not int instead of isinstance,
  since bool is an int subclass and would otherwise stringify to
  "True"/"False" and corrupt legacy payloads on backfill.
- Replace doc_id.isdigit() guards in 5 boundary sites
  (api/visualization, auth/viz_routes, search/context note/news_item/
  deck_card branches) with a shared is_valid_nextcloud_doc_id helper
  that rejects "0", leading zeros, and Unicode digit classes
  (superscripts, Arabic-Indic, Devanagari) which pass isdigit() but
  cannot be valid MySQL AUTO_INCREMENT IDs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 20:28:47 +02:00
Chris CoutinhoandClaude Opus 4.7 f3ce46da0f fix(vector): address PR review round 11 — broaden offset-skip gate, clarify ordering
- search/context.py: drop the doc_type=='file' guard on skip_offset_lookup
  so notes / deck cards / news items also bypass the unindexed offset
  fallback when chunk_index is available. Legacy chunk_index=None data
  still uses the offset path.
- vector/qdrant_client.py: clarify the backfill/_ensure_payload_indexes
  ordering invariant (backfill rewrites payload values only, never schema
  or indexes). Acknowledge OSS-vs-Cloud uncertainty in the 400-branch
  comment and the new-collection call-site comment.
- vector/scanner.py: hoist qdrant_client to function scope so the
  file-scroll block doesn't depend on a name bound inside the
  notes-scroll block.
- tests/unit/test_chunk_context_offset_gate.py: flip the note-with-
  chunk_index test to assert the offset fallback is skipped.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 17:59:41 +02:00
Chris CoutinhoandClaude Opus 4.7 47c531969f fix(vector): address PR review round 10 — index chunk_index, harden index loop, lazy-init lock
Three coordinated fixes flagged as Important in the round-10 review of
PR #773:

1. Index chunk_index. The chunk-context fast path in
   _get_chunk_by_index_from_qdrant and get_chunk_bbox_and_page_from_qdrant
   filters on chunk_index, but the field was absent from
   _PAYLOAD_INDEX_FIELDS. On Qdrant Cloud strict mode every chunk-context
   lookup via chunk_index would 400 and silently fall back to the
   document re-fetch path — the exact failure mode the chunk_index
   shortcut exists to avoid. Added as INTEGER schema.

2. Catch raw network errors in _ensure_payload_indexes. The
   create_payload_index loop only caught UnexpectedResponse, so an
   httpx.ConnectError or asyncio.TimeoutError mid-loop would propagate
   uncaught — leaving _qdrant_client assigned and silently skipping all
   remaining fields. Added a broad Exception catch with the same
   per-field containment as the 5xx path: log at ERROR with exc_info,
   append to failed_fields, continue. New test covers the path.

3. Lazy-initialise _qdrant_init_lock. Constructing anyio.Lock() at
   module import time works for the asyncio backend but anyio's docs
   advise instantiating synchronization primitives within an async
   context, and pyproject.toml's anyio_mode = "auto" means tests can
   run under trio. Moved the construction into get_qdrant_client; safe
   under cooperative multitasking because there is no await between the
   None-check and the assignment.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 17:07:15 +02:00
Chris CoutinhoandClaude Opus 4.7 d60348e77b fix(api): validate doc_id at chunk-context handler boundary
Add an .isdigit() guard at the top of both chunk-context handlers so a
non-numeric doc_id fails fast with a clear 400 ("doc_id must be numeric,
got 'abc'") rather than silently bottoming out as a 404 from deep inside
get_chunk_with_context. The earlier int(doc_id) coercion was removed when
doc_id became a pure pass-through to Qdrant's keyword payload index, which
also dropped this boundary validation.

Also align test_backfill_emits_progress_log_every_20_batches' scroll stub
with real Qdrant: next_offset is now "next-1" (str) instead of 1 (int),
matching the sibling test_backfill_rewrites_int_doc_ids_to_str. Pure
stub-fidelity fix; production code already treats next_offset as opaque.

Addresses both 🟡 Important items from PR #773 review round 10.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 16:54:06 +02:00
Chris CoutinhoandClaude Opus 4.7 fec1596784 fix(vector): address PR review round 9 — drop redundant guard, add init lock, test float doc_id path
Addresses the four 🟡 important findings from claude-bot review on PR #773:

str (non-Optional) and the guard would silently skip the Qdrant lookup
for an empty string. Removing the guard matches the type signature.

(`all([…, doc_id, …])` rejects None and empty string, plus
`assert doc_id is not None`). No code change needed.

`get_qdrant_client()` with a module-level `anyio.Lock`. Double-checked
locking keeps the steady-state hot path lock-free. Without this,
parallel cold-start callers could all enter the init block and run
`_backfill_doc_id_to_string` + `_ensure_payload_indexes` redundantly
(idempotent, but noisy). Pattern matches `auth/storage.py:2071`.

behavior with three tests covering the float-warning path (the gap
called out in the review), the str/None silent-skip paths, and the
int-grouping happy path.

Verification:
- ruff check / format: clean
- ty check -- nextcloud_mcp_server: clean
- uv run pytest tests/unit/: 969 passed

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 15:53:33 +02:00
Chris CoutinhoandClaude Opus 4.7 0c14501a2b test: align CI assertions with documented contracts
Two unrelated CI failures on this branch, one fix each:

- tests/integration/test_deck_vector_search.py: pass str(card.id) to
  get_chunk_with_context. The function's contract is doc_id: str
  (keyword-indexed in Qdrant), and real callers (viz_routes.py URL
  path, server/semantic.py via str(result.id)) all stringify. The
  test was the only int caller, hitting the .isdigit() guard added
  earlier on this branch.

- tests/server/login_flow/test_login_flow_integration.py:
  test_check_status_provisioned now accepts scopes=None as valid.
  Per ProvisionStatusResponse in models/auth.py, None is the
  documented sentinel for "all scopes granted" — and the web
  provisioning path (provision_routes.py, used by Astrolabe's
  "Enable Semantic Search" flow exercised by the new regression test
  added on this branch) stores exactly that. The previous
  is-not-None assertion hid behind test order until that flow ran.

- Replace anyio.sleep(0) with anyio.lowlevel.checkpoint()

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 15:28:15 +02:00
Chris CoutinhoandClaude Opus 4.7 64f0842977 fix(vector): guard _group_int_doc_ids against non-int doc_id values
Skip and warn instead of stringifying floats / unexpected types in the
backfill helper. A stray doc_id=3.0 would otherwise be rewritten to
"3.0", which producers (str(int)) and the keyword index would never
match, and which int() on the verification side would reject. Also add
a doc_id=0 case to the backfill test to guard against a future
falsy-skip regression.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 14:53:12 +02:00
Chris Coutinho fc9786c3a9 test: Raise on missing NEXTCLOUD_PASSWORD 2026-05-09 14:39:55 +02:00
Chris CoutinhoandClaude Opus 4.7 22a2a24941 test(integration): add login-flow Astrolabe provisioning regression test
Add the missing end-to-end coverage for the seam that PR #773's
`ALLOWED_MGMT_CLIENT` ↔ `astrolabeMcpClientOAuth00000000000` drift
bug exposed: browser → Astrolabe (NC PHP app) → MCP server's
management API on the `mcp-login-flow` profile. Every existing
`tests/integration/test_astrolabe_*.py` is marked `multi_user_basic`
and exercises the BasicAuth flow, not Login Flow v2 / OAuth.

The new test mirrors the production-shaped flow exactly:

1. Log in as admin via Playwright.
2. Navigate to `/settings/user/astrolabe`.
3. Click the "Enable Semantic Search" OAuth link rendered by
   `oauth-required.php`. (Same selector Astrolabe's own e2e helper
   uses — `third_party/astrolabe/tests/e2e/helpers/authorize.ts`.)
4. Click "Allow" on the Nextcloud OIDC consent screen.
5. Wait for the redirect back to the Astrolabe settings page.
6. Assert the "Enable Semantic Search" link is no longer visible.

Step 6 is the canary for the drift class: if Astrolabe's management
API call to `/api/v1/users/{id}/session` is rejected (HTTP 401, the
original bug), the session lookup falls back to "no token" and the
same `oauth-required.php` template re-renders with the link still
present — so the test fails loudly with a message naming the
likely cause.

Reuses `login_to_nextcloud` and `navigate_to_astrolabe_settings`
helpers from `tests/integration/test_astrolabe_multi_user_background_sync.py`
(already pattern-imported by the Plotly viz test). No fixture-level
OIDC client creation: `app-hooks/before-starting/26-configure-astrolabe-oauth.sh`
already provisions `astrolabeMcpClientOAuth00000000000` with the
correct redirect URI and scopes when `MCP_SERVER_URL` is set in the
shell that runs `docker compose --profile login-flow up`.

The test skips cleanly when admin is already authorized (typical
state on a re-run against a long-lived dev stack), so it's safe to
run repeatedly without manual reset.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 14:32:54 +02:00
Chris CoutinhoandClaude Opus 4.7 a27f738dbf fix(vector): tighten get_chunk_bbox_and_page_from_qdrant doc_id to str
🔴 Blocking finding from PR #773 latest review:
`get_chunk_bbox_and_page_from_qdrant` (`search/context.py:199`) still
declared `doc_id: int | str` and passed the raw value into
`MatchValue(value=doc_id)` at lines 239 and 256 without `str()`
coercion. After this branch's startup backfill normalises every Qdrant
`doc_id` payload to a string, an `int` filter would silently match zero
points — the function would return `(None, None)` instead of the
chunk bbox / page, and PDF highlight overlays would fail in production.

Take option 2 from the reviewer's two suggestions (annotation
tightening over inline coercion): the producer side of this PR has
already narrowed every other `doc_id` annotation to `str`, so this
function is the last hold-out. Pushing the contract into the type
system means `ty` will catch any future regression at the call site.

Production callers in `api/visualization.py` and `auth/viz_routes.py`
already pass `doc_id` (str) verbatim after the recent merge with
master's chunk_index-first refactor, so no caller-side changes needed.

Update the 9 calls in `tests/unit/test_chunk_bbox_helper.py` to use
string literals (`"42"` / `"99"` / `"1"`) instead of integers. The
mock doesn't validate `MatchValue` value types, so the tests passed
with stale int doc_ids today — but they were exercising a path
production no longer takes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 14:32:33 +02:00
Chris CoutinhoandClaude Opus 4.7 d83c32a9dd test(login-flow): use Astrolabe's client id for management API tests
The previous commit moved `mcp-login-flow`'s `ALLOWED_MGMT_CLIENT` to
`astrolabeMcpClientOAuth00000000000` so production-shaped Astrolabe
traffic actually validates. Update the management API test fixture to
match: the static OIDC client created in
`tests/server/login_flow/conftest.py:login_flow_static_client_credentials`
now uses the same id `app-hooks/before-starting/26-configure-astrolabe-oauth.sh`
provisions in real deployments, so the test path exercises the same
code as production rather than a substituted fixture-only id.

`mcp-multi-user-basic`'s allowlist is unchanged
(`nextcloudMcpServerUIPublicClient`) and the shared
`configure_astrolabe_for_mcp_server` fixture in `tests/conftest.py`
keeps that as its default, so multi-user-basic tests are unaffected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 14:06:02 +02:00
Chris CoutinhoandClaude Opus 4.7 fd8c037eea fix(login-flow): allow Astrolabe's OAuth client on the management API
The `mcp-login-flow` profile's `ALLOWED_MGMT_CLIENT` was set to the
test-fixture id `nextcloudMcpServerUIPublicClient` only, but the actual
Astrolabe app provisions its OIDC client as
`astrolabeMcpClientOAuth00000000000` (see
`app-hooks/before-starting/26-configure-astrolabe-oauth.sh:39`). All
tokens issued through the "Enable Semantic Search" flow were rejected
with HTTP 401 by `unified_verifier.py:222-227`'s allowlist check, and
the Astrolabe UI's retry loop subsequently exhausted the
`api/passwords.py` 5/hr rate limit (HTTP 429).

Switch the `mcp-login-flow` allowlist to Astrolabe's client id so
production-shaped traffic actually validates. The `mcp-multi-user-basic`
profile keeps `nextcloudMcpServerUIPublicClient` for the
`configure_astrolabe_for_mcp_server` test fixture.

Also bump `third_party/astrolabe` 0.13.12 → 0.14.0 to pull in the
chunk-context indexed-lookup fix (#75) and the PDF bbox highlight
overlay (#76) that match the master-side changes already merged on
this branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 14:03:42 +02:00
Chris CoutinhoandClaude Opus 4.7 d390b3a4b8 fix(vector): address PR review round 8 — anyio convention + cosine-safe sentinel + dedup get_collection
Reviewer findings (1 blocking + 2 important):

- 🔴 Replace `import asyncio` / `await asyncio.sleep(0)` with
  `import anyio` / `await anyio.sleep(0)` in the four async side-effect
  helpers (_scroll_raises, _upsert_raises, _get_collection_raises,
  _create_index). CLAUDE.md mandates anyio for all async operations;
  conftest pins the backend to asyncio so the asyncio.sleep call worked
  today, but the inconsistency would surface the moment that pin moves.
- 🟡 Replace the sentinel's zero dense vector with a single non-zero
  element (`[1e-9] + [0.0] * (dimension - 1)`). Cosine distance is
  mathematically undefined for the zero vector and Qdrant Cloud strict
  mode rejects zero-vector upserts. The exact value doesn't matter
  (sentinel never participates in a search — no user_id/doc_id/doc_type
  payload) but the upsert itself must be valid.
- 🟡 Avoid the duplicate `get_collection` round-trip on every restart.
  `_ensure_payload_indexes` now accepts an optional
  `existing_schema: dict | None` parameter; when None it fetches
  collection_info itself (and the get_collection-failure swallow still
  applies), but `get_qdrant_client` already fetches collection_info
  for dimension validation in the existing-collection branch — pass
  `collection_info.payload_schema or {}` through to skip the second
  call. The new-collection branch passes `existing_schema={}`
  explicitly since a freshly created collection has no payload schema.

The 🟡 deck_card iteration-fallback finding doesn't apply: the
`isdigit()` guard at context.py:612 returns early before either the
fast-path or the iteration fallback runs, so non-numeric doc_ids
cannot reach the inner `c.id == int(doc_id)` comparison.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 13:48:54 +02:00
Chris Coutinho 9720a7e4fe Merge remote-tracking branch 'origin/master' into fix/qdrant-doc-id-keyword-index 2026-05-09 13:48:21 +02:00
Chris CoutinhoandClaude Opus 4.7 7ef8760d27 fix(chunk-context): address PR #767 review — extract bbox helper, fix page_number overwrite
Resolves both 🟡 important issues from the latest review:

1. `page_number` was unconditionally overwritten in `viz_routes.py:696` even
   when Qdrant's payload lacked the field, clobbering the value resolved
   from `chunk_context.page_number`. The new helper returns each field
   independently and both call sites only overwrite via `is not None`
   guards, matching the existing logic in `visualization.py`.

2. The ~60-line `if chunk_index is not None: ... else: ...` Qdrant scroll
   block was duplicated between `api/visualization.py` and
   `auth/viz_routes.py`. Extracted into `get_chunk_bbox_and_page_from_qdrant`
   in `search/context.py` alongside the existing private `_get_chunk_*_from_qdrant`
   helpers; both routes now share ~12 lines of caller code.

New unit tests at `tests/unit/test_chunk_bbox_helper.py` cover the indexed
and offset paths, the `(bbox, None)` regression case, and graceful
degradation on Qdrant strict-mode 400 (which also closes nit #4).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 13:29:22 +02:00
Chris CoutinhoandClaude Opus 4.7 d00779ce79 fix(vector): add BOOL index for is_placeholder + correct wait=True docstring
Reviewer feedback (2 items):

- Add a BOOL payload index for `is_placeholder` alongside the three
  KEYWORD fields. Strict-mode index-required filtering on Qdrant Cloud
  enforces a payload index on any field used in a `FieldCondition`
  regardless of value type, so `get_placeholder_filter` and
  `delete_placeholder_point` would have produced HTTP 400 on Cloud
  instances even after this PR's KEYWORD fix.

  Implementation: replace `_KEYWORD_PAYLOAD_FIELDS: tuple` with
  `_PAYLOAD_INDEX_FIELDS: dict[str, PayloadSchemaType]` so each
  field carries its own schema type. Rename
  `_ensure_keyword_payload_indexes` to `_ensure_payload_indexes` since
  the function now creates more than just KEYWORD indexes. The
  per-field log line now includes the schema type
  ("Created KEYWORD payload index on 'doc_id'", "Created BOOL payload
  index on 'is_placeholder'") so operators can tell which type was
  created without checking the source.

- Correct the misleading `wait=True` docstring in
  `_apply_backfill_writes`. The previous wording said
  `_ensure_payload_indexes` runs "immediately after this function",
  but `_apply_backfill_writes` is called in a loop inside
  `_backfill_doc_id_to_string` — the index creation runs after the
  backfill function *returns*, not after each write. Rewrote the
  docstring to capture both load-bearing reasons:
  (1) per-batch commit ordering for crash-recovery safety, and
  (2) ensuring the keyword index built later covers committed
  payloads only.

Adds `test_ensure_payload_indexes_includes_is_placeholder_as_bool`
asserting the schema type is BOOL specifically. Existing tests
updated to use the new dict-based registry (side_effect lists now
extend to all four entries; field-set assertions derive from the
registry instead of hardcoding 3 KEYWORD names).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 13:01:05 +02:00
Chris CoutinhoandClaude Opus 4.7 928b973eb8 fix(webdav): decode percent-encoded names in PROPFIND/SEARCH responses
`<d:href>` is required by RFC 3986 to be percent-encoded, so non-ASCII
filenames (e.g. Chinese, Cyrillic) were leaking through `list_directory`
and the SEARCH-based tools (`find_by_name`, `find_by_type`,
`list_favorites`, `search_files`) as their URL-encoded form. Decode with
the already-imported `urllib.parse.unquote` before exposing to callers.

Fixes #776

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 12:57:58 +02:00
Chris CoutinhoandClaude Opus 4.7 60a9882c92 fix(vector): address PR review round 6 + SonarCloud findings
Reviewer feedback (3 important + 3 nits):

- Wrap _ensure_keyword_payload_indexes' get_collection() call in
  try/except. The qdrant_client singleton is already assigned by the
  time this function runs, so a transient timeout/DNS failure
  propagating out left the process holding a usable client with the
  migration silently skipped on every subsequent call. Now logs ERROR
  with exc_info and returns; next process restart retries.
- Add `and "doc_id" in point.payload` guard to the four set
  comprehensions in scanner.py (indexed_doc_ids, indexed_file_ids,
  indexed_item_ids, indexed_card_ids). Previously a payload missing
  the doc_id key would raise KeyError and crash the entire scan.
- Tighten test_ensure_keyword_payload_indexes_logs_400_as_warning to
  match the per-field warning prefix exactly (`startswith("Schema
  conflict on payload index")`), so a future change adding 400s to
  the partial-failure summary surfaces here as a count mismatch.
- Add new-collection vs existing-collection context to the
  _backfill_doc_id_to_string docstring's `dimension` parameter.
- Replace the misleading "rewrote 0/N from int to str" wording when
  no rewriting was needed with "N points scanned, none required
  rewriting (collection already in str form)".
- Add test_ensure_keyword_payload_indexes_logs_and_returns_when_
  get_collection_raises mirroring the scroll-failure test.

SonarCloud (1 CRITICAL + 1 MINOR):

- Refactor _backfill_doc_id_to_string to bring cognitive complexity
  under 15 (was 19). Extracted two pure helpers: _group_int_doc_ids
  (group point IDs by stringified doc_id) and _apply_backfill_writes
  (apply set_payload calls and return rewritten count). The main
  function's scroll/loop/sentinel structure is unchanged.
- Add `await asyncio.sleep(0)` to the three async test side_effect
  helpers (_scroll_raises, _upsert_raises, _create_index) so they use
  an actual async feature (S7503). The async-callable shape is still
  required to avoid the AsyncMock unawaited-coroutine warning when
  side_effect raises.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 00:30:29 +02:00
Chris CoutinhoandClaude Opus 4.7 c27556c332 fix(vector): address PR review round 5 — progress logging, summary visibility, sentinel split
Addresses three important findings from the latest reviewer comment:

- Add progress INFO log every 20 scroll batches (≈5120 points at
  batch_size=256) in _backfill_doc_id_to_string so a long-running
  migration on a large collection (50k+ points) doesn't look like a
  startup hang. The line carries collection name, scanned count, and
  rewritten count so it doubles as a heartbeat.
- Track non-400 failures in _ensure_keyword_payload_indexes and emit
  a WARNING summary line listing every field that failed to get an
  index. Per-field ERROR lines are easy to miss in startup noise; the
  summary makes the partial-failure state visible at a glance.
- Split the sentinel upsert out of the data-scroll try/except in
  _backfill_doc_id_to_string. A scroll-time failure still logs ERROR
  with the new "scroll failed" wording (data is incomplete). A
  sentinel-write failure now logs WARNING with "data succeeded but
  sentinel write failed" wording — data is correct, only the
  short-circuit marker is missing, and the next restart re-scrolls
  an already-clean collection (idempotent zero-write) before retrying
  the upsert.

Also fix the RuntimeWarning emitted by
test_backfill_logs_and_returns_when_scroll_raises: replace the bare
`RuntimeError` side_effect with an async-callable side_effect so
AsyncMock awaits the coroutine before the exception propagates.

Three new unit tests cover the new branches:
test_backfill_emits_progress_log_every_20_batches,
test_backfill_logs_warning_when_sentinel_upsert_fails,
test_ensure_keyword_payload_indexes_summarises_failed_fields.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 00:15:56 +02:00
Chris CoutinhoandClaude Opus 4.7 c780f96d2b fix(chunk-context): address PR #767 review — drop dead PDF branch, redundant alias, add boundary tests
- Remove unreachable doc_type=="file" branch (and pymupdf/pymupdf4llm
  imports) from _fetch_document_text in search/context.py — the file
  path is short-circuited in get_chunk_with_context before reaching it.
- Drop the redundant `username = request.user.display_name` alias in
  auth/viz_routes.py; both Qdrant scroll filters now reference user_id
  consistently with the rest of the handler.
- Add TestAdjacentChunkBoundary in tests/unit/test_chunk_context_offset_gate.py
  covering chunk_index=0 (before-fetch gate closed) and
  chunk_index=total_chunks-1 (after-fetch gate closed) — the two
  off-by-one boundaries previously untested.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 00:07:51 +02:00
Chris CoutinhoandClaude Opus 4.7 b97ac23228 fix(vector): address PR review round 4 — backfill resilience + degraded-mode docs
- Remove three stale `# Use numeric file ID` / `# Pass file path` comments
  in scanner.py. file_id is already normalized to str() above each call
  site, so the inline comments mislead readers.
- Wrap `_backfill_doc_id_to_string` scroll loop + sentinel upsert in
  try/except Exception. The qdrant_client singleton is assigned before
  this migration runs, so a transient scroll failure was leaving the
  process holding a usable client with int payloads permanently
  unbackfilled until the next restart. Catch broadly, log ERROR with
  exc_info, and return without writing the sentinel — next process
  restart retries from scratch.
- Note `:memory:` mode behavior near the sentinel constants so future
  readers don't read the every-start scroll as a bug.
- Document the two degraded-migration ERROR log signals in
  docs/configuration.md so operators know when a clean restart is
  required to recover indexing.
- Add unit test asserting scroll-time exceptions are logged and swallowed
  without writing the sentinel.

Closes round-4 review feedback on PR #773.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 23:39:37 +02:00
Chris CoutinhoandClaude Opus 4.7 47b0b737b6 fix(chunk-context): address PR #767 round-3 review — gate readability + legacy-fallback comment
- search/context.py: rename triple-negation gate condition to
  `skip_offset_lookup` named boolean for readability; convert new
  logger.warning to lazy %-style per repo convention.
- api/visualization.py, auth/viz_routes.py: add comment on the offset-only
  Qdrant scroll branch noting it is a legacy path for pre-astrolabe#75
  clients and degrades gracefully on Qdrant Cloud strict mode.

Reviewer item #2 (extracting the duplicated scroll block into a shared
helper) deferred to a follow-up issue per the reviewer's "not blocking"
framing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 23:21:30 +02:00
Chris Coutinho 058463ee87 Merge remote-tracking branch 'origin/master' into fix/chunk-context-indexed-lookup
# Conflicts:
#	nextcloud_mcp_server/api/visualization.py
#	nextcloud_mcp_server/auth/viz_routes.py
2026-05-08 23:10:09 +02:00
Chris Coutinho 02744a50e0 Merge remote-tracking branch 'origin/master' into fix/qdrant-doc-id-keyword-index 2026-05-08 23:08:39 +02:00
Chris CoutinhoandClaude Opus 4.7 0b004f54bd refactor(vector): address PR #775 review round 3 — fix unused var, harden boundary lookup, rename trace span
- pdf_highlighter.compute_chunk_bboxes_batch: drop unused chunk_text
  destructure (SonarQube finding), and replace positional
  page_boundaries[page_num - 1] with a key-based next() match so
  reordered or non-1-indexed boundaries can't silently shift the bbox.
  Convert touched f-string log to lazy %s formatting.
- vector/processor: rename the trace_operation span from
  "vector_sync.generate_highlights" to "vector_sync.compute_chunk_bboxes"
  to match what the function actually does.
- Add test_compute_chunk_bboxes_handles_unordered_page_boundaries —
  reverses the boundaries list and asserts identical results to the
  in-order case, guarding the boundary-lookup regression class.
- Pin pre-push-review skill to sonnet model.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 23:05:31 +02:00
Chris CoutinhoandClaude Opus 4.7 92b2d50cd7 fix(vector): address PR review round 3 — sentinel guard, skip indexed fields, narrow types
- Add a fixed-UUID sentinel point written after a successful doc_id
  backfill so subsequent restarts retrieve it and short-circuit the
  O(N) scroll. Sentinel has no user_id/doc_id/doc_type payload so
  production search filters never see it.
- Pre-fetch payload_schema in _ensure_keyword_payload_indexes and
  silently skip fields that are already indexed; the "Created KEYWORD
  payload index" INFO log fires only on actual creation.
- Narrow stale `int | str` doc_id annotations to `str` across
  search/verification.py (BatchVerifier return type, per-verifier
  accessible sets, by_type / accessible_by_type / inaccessible
  collections); drop the now-redundant `type(d).__name__` prefix in
  the dropped-docs log.
- Align the backfill log message with the PR description's
  "Running doc_id backfill" promise; add a caller cross-reference to
  the wait=True comment.
- Fix _get_file_path_from_qdrant docstring (file_id is str, not numeric).
- Convert legacy `id=1` to `id="1"` in test_search_result.py to match
  the SearchResult.id: str annotation.

Three new unit tests cover sentinel-found, sentinel-written, and
skip-existing-index branches; existing backfill tests pass dimension
and explicit retrieve.return_value=[] for the no-sentinel path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 22:59:46 +02:00
Chris CoutinhoandClaude Opus 4.7 8bc87ed37d refactor(vector): address PR #775 review round 2 — drop dead page field, add omission tests
- chunk_bboxes is now dict[int, list[tuple[...]]] holding the bbox list
  directly, not {"bbox": ..., "page": ...}. The page from text-search was
  stored but never read; page_number from offset-based assignment is
  authoritative for the Qdrant payload.
- Add two unit tests for the documented omission contract: chunks whose
  offsets fall outside every page boundary, and chunks whose text cannot
  be located on the rendered page, are silently dropped from the result.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 22:16:50 +02:00
Chris CoutinhoandClaude Opus 4.7 45da876cf5 fix(chunk-context): propagate chunk_index=None through ChunkContext
Addresses the deferred half of PR #767 review issue 2: instead of just
documenting the "0/N misreport" with a logger.warning, propagate the
caller's None for chunk_index through the dataclass, position markers,
and response builders so callers can distinguish "unknown position"
from "actually chunk 0".

Changes:
- ChunkContext.chunk_index: int → int | None
- _insert_position_markers: chunk_index parameter is int | None; when
  None, renders "Chunk ?/N" instead of "Chunk 1 of N"
- get_chunk_with_context: drops the effective_chunk_index local entirely.
  Passes chunk_index (may be None) directly into both ChunkContext and
  _insert_position_markers, in both the Qdrant fast path and the
  doc-text fallback.
- Fast path: when chunk_index is None and chunk_text was retrieved via
  the offset lookup (notes/cards), skip the adjacent-chunk fetch.
  Index arithmetic from a default 0 would query the chunks at positions
  -1 and 1 even when the actual chunk is, say, 5/20 — silently producing
  wrong "before"/"after" text. Mark both sides as truncated instead.
- Drop the now-redundant logger.warning in the doc-text fallback (the
  response correctly communicates the unknown state via chunk_index=None).

Both existing response builders (`api/visualization.py:657` and
`auth/viz_routes.py:717`) already serialise `chunk_context.chunk_index`
unconditionally; `None` becomes JSON `null`. No route changes needed.

Adds 5 regression tests:
- ChunkContext.chunk_index propagates as None in fast path
- ChunkContext.chunk_index propagates as None in doc-text fallback
- Fast path with chunk_index renders "Chunk N of M" correctly
- _insert_position_markers renders "?/N" for None chunk_index
- _insert_position_markers renders explicit index when supplied

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 22:16:00 +02:00
Chris CoutinhoandClaude Opus 4.7 b5b4025bb4 fix(vector): address PR review round 2 — status branching, doc_id guard, doc restore
- _ensure_keyword_payload_indexes: distinguish 400 (schema conflict, warning)
  from other status codes (5xx/network, error) so a transient outage doesn't
  silently leave the collection unindexed.
- build_search_result_from_point: use .get("doc_id") + return None on missing
  instead of KeyError-crashing the search; reverse metadata merge order so
  payload-derived chunk_index/total_chunks win over caller-supplied extras.
- docs/configuration.md: restore the OpenAI/Mistral/Bedrock/Simple provider
  sections + reference-table rows that were dropped in the rebase. Reword
  the "Startup migrations" bullet to describe what the code actually does
  (no sampling — full scroll, zero writes when clean). Add operator note
  about the SemanticSearchResult.id TypeError path.
- tests: pytest.approx for float equality (Sonar python:S1244); coverage
  for non-400 → ERROR, payload={doc_id: None}, and missing doc_id key.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 21:37:10 +02:00
Chris CoutinhoandClaude Opus 4.7 80b27b1bc6 refactor(vector): address PR #775 review — drop unused payload key, fix resource leaks
- Drop chunk_bbox_page from Qdrant payload — viz endpoints never read it
  (page_number is the canonical PDF page field).
- Bump upsert BATCH_SIZE 10 → 100 now that payloads no longer carry PNGs.
- compute_chunk_bboxes_batch: move doc.close() into finally, replace
  unused stored_page_num with _.
- purge_page_images.py: switch to anyio.run() per project convention,
  and wrap AsyncQdrantClient in try/finally so the aiohttp session is
  always closed (the class doesn't implement async-context-manager).
- Decorate new bbox unit tests with @pytest.mark.unit so they run under
  the fast-feedback selector.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 21:32:12 +02:00
Chris CoutinhoandClaude Opus 4.7 51c1d42ea3 fix(chunk-context): address PR #767 round-2 review — gate, parity, doc
Latest reviewer comment flagged five items on top of the original PR. This
commit addresses every one:

🟡 1. Skip the offset-based Qdrant fallback for `doc_type=file` when
   `chunk_index` is supplied. Qdrant Cloud's strict mode rejects unindexed
   filter fields with HTTP 400, which `_get_chunk_from_qdrant` catches and
   logs at `logger.error` — masking real Qdrant problems in monitoring.
   Notes/cards keep the offset fallback (cheap, useful for legacy data).

🟡 2. Add a `logger.warning` and clarifying inline comment in the doc-text
   fallback path when `chunk_index` is None — surfaces the pre-existing
   "0/N misreport" so callers can detect it. Type-nullability propagation
   is deferred to a follow-up (out of scope for this hotfix).

🟢 3. Simplify `if chunk_text and doc_id_int is not None:` →
   `if chunk_text:` with an inner `assert doc_id_int is not None` for
   `ty` narrowing. The outer second clause was dead.

🟢 4. Add `doc_type` `FieldCondition` to the offset-based image lookup in
   both `visualization.py` and `viz_routes.py` for parity with the
   `chunk_index` branches.

🟢 5. Inline the `chunk_filter` local in `visualization.py` directly into
   the `must=[]` list (matches `viz_routes.py` style).

Adds `tests/unit/test_chunk_context_offset_gate.py` with three regression
tests covering the gate matrix: (file, with-index → skip offset),
(note, with-index → still tries offset), (file, no-index → still tries
offset). Lives at top-level rather than `tests/unit/search/` to side-step
a pre-existing circular-init issue in `nextcloud_mcp_server.search`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 21:28:50 +02:00
Chris CoutinhoandClaude Opus 4.7 ee402ea00e feat(vector): replace inline page-image payloads with chunk_bbox (Deck #76)
Per-chunk PDF page renders (~150–700 KB base64 PNG each) were the dominant
disk consumer in production, repeatedly tripping `No space left on device:
WAL buffer size exceeds available disk space` on welcomed-malamute Qdrant.

Replace the inline highlighted_page_image / highlighted_page_number /
highlight_count fields with a small `chunk_bbox` field:
list[(x0, y0, x1, y1)] of normalized [0, 1] floats, ~32 bytes per chunk.
Astrolabe (the only known consumer) renders the highlight client-side as
a percentage-positioned overlay on top of the existing /api/v1/pdf-preview
render-on-demand path (cbcoutinho/astrolabe#76).

- pdf_highlighter: new compute_chunk_bboxes_batch() that reuses the
  existing _find_chunk_bbox text-search path, skipping all pixmap/PIL/PNG
  work.
- processor: store chunk_bbox + chunk_bbox_page in the Qdrant payload,
  drop highlighted_page_image + friends, drop the base64 import.
- visualization /api/v1/chunk-context and auth/viz_routes: read
  chunk_bbox instead of highlighted_page_image.
- vector/__init__: stop eagerly re-exporting `processor`/`scanner` —
  fixes a pre-existing circular import (search.algorithms ->
  vector.placeholder -> vector/__init__ -> processor -> scanner ->
  server.semantic -> search.bm25_hybrid -> search.algorithms partial).
  Test suite that was broken on master (test_bm25_hybrid.py et al.) now
  collects and passes.
- scripts/purge_page_images.py: ad-hoc, idempotent migration that
  delete_payload's the legacy keys from existing points. No reindex
  required; legacy chunks render the page with no overlay.

Pairs with cbcoutinho/astrolabe#76. Frontend handles missing chunk_bbox
gracefully, so this can land in either order.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 21:16:00 +02:00
Chris CoutinhoandClaude Opus 4.7 6aba589a6e fix(vector): address PR review — wait=True backfill, batched writes, search helper
Addresses reviewer feedback on PR #773:

- Backfill set_payload now uses wait=True to avoid a race where
  _ensure_keyword_payload_indexes builds the KEYWORD index before
  fire-and-forget writes have committed, leaving int payloads
  invisible to filters.
- Batch points sharing the same int doc_id into a single set_payload
  call (one document → many chunks → one round-trip instead of N).
- Drop _has_int_doc_id_sample short-circuit. The sample's false-negative
  window (clean first 256 results, ints further in) is gone; full scroll
  is the dominant cost on first run anyway.
- Simplify _ensure_keyword_payload_indexes: the "already exists" 400
  branch was dead code (Qdrant returns 200 on identical re-create); any
  400 now logs a warning and continues.
- search/context.py: comment the broadened file-type guard. Add explicit
  not doc_id.isdigit() checks at the top of note/news_item/deck_card
  branches in _fetch_document_text so malformed payloads surface as
  warnings instead of being swallowed by the broad except.

Also extracts build_search_result_from_point into search/algorithms.py
to deduplicate the 71-line payload-extraction loop shared by
SemanticSearchAlgorithm and BM25HybridSearchAlgorithm. This fixes
SonarQube's quality-gate failure (4.0% new-code duplication, max 3%).

Test coverage:
- 7 new unit tests for build_search_result_from_point covering missing
  payload, note/file/deck_card metadata, int doc_id coercion, and
  metadata_extras merging.
- Replace _has_int_doc_id_sample tests with clean-collection no-op and
  per-batch grouping tests.
- Update set_payload assertions from wait=False to wait=True.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 21:14:28 +02:00
Chris CoutinhoandClaude Opus 4.7 8457c427a5 fix(chunk-context): address PR #767 review — doc_type filter parity + tests
- Add `doc_type` FieldCondition to the chunk_index-path highlighted-image
  Qdrant filter in both `api/visualization.py` and `auth/viz_routes.py`,
  matching the shape of `_get_chunk_by_index_from_qdrant`. Safe today (the
  block is guarded by `doc_type == "file"` and Nextcloud file IDs are
  globally unique) but prevents a latent bug if other doc types start
  storing highlighted images.
- Demote `viz_routes.py` `ValueError` log from `error` to `warning` (lazy
  %-style) — `_parse_int_param` raises on user-supplied bad input, which
  is a 400 not a server error and shouldn't pollute error logs.
- Hoist `effective_chunk_index` to compute once at the top of
  `get_chunk_with_context`, removing two duplicate assignments.
- Add `test_file_doc_type_qdrant_miss_yields_fast_404` to the management
  endpoint tests, locking in the proxy-timeout fix contract.
- Add `tests/unit/test_viz_routes_chunk_context.py` mirroring management
  coverage for the OAuth-session route: param forwarding (chunk_index /
  total_chunks), `doc_type=file` fast 404, and 400 on invalid int params.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 20:50:27 +02:00
Chris CoutinhoandClaude Opus 4.7 719b3b5034 fix(vector): normalize doc_id to str + add Qdrant keyword payload indexes
Production was logging two cascading classes of Qdrant errors against the
welcomed-malamute deployment:

1. HTTP 400 — "Bad request: Index required but not found for \"doc_id\" of
   one of the following types: [keyword]". The collection was created via
   create_collection() with no payload indexes, so any FieldCondition
   filter on doc_id failed at the Qdrant layer (placeholder writes/reads,
   eviction, search context lookups).

2. Compounding the missing index, producers wrote a mix of int and str
   doc_ids: webhook_parser stringified node_id, scanner stringified note
   IDs, news IDs, and deck card IDs — but the file scanner passed the
   numeric file_id through unchanged. A keyword index would not have
   covered both kinds even if it had existed.

This change:

- Normalizes doc_id to str at every producer site (scanner.py:459,
  DocumentTask.doc_id, indexed_*_ids reads from Qdrant).
- Tightens str|int annotations to str across placeholder.py,
  eviction.py, search/verification.py, search/context.py,
  SearchResult.id, and the auth/api visualization endpoints.
- Defensive str() coercion on doc_id reads in semantic.py /
  bm25_hybrid.py / vector/visualization.py for the transition window
  before the backfill runs.
- Adds an idempotent startup migration in get_qdrant_client():
  - _ensure_keyword_payload_indexes creates KEYWORD indexes for
    doc_id, user_id, and doc_type (tolerates "already exists" 400s).
  - _backfill_doc_id_to_string scrolls the collection once and rewrites
    int doc_ids to str. Skipped after a quick sample shows no legacy
    int payloads.
- Public API preserved: SemanticSearchResult.id stays int via explicit
  int(r.id) narrowing in server/semantic.py — surfaces a TypeError with
  actionable context if a future doc_type ships non-numeric ids.
- Documents the startup migration in docs/configuration.md.

Tests: 11 new unit tests in tests/unit/vector/test_qdrant_client.py
covering happy path / already-exists / unrelated-400 for the index
helpers, and sample-skip / mixed-batch rewrite / payload=None edge cases
for the backfill. 889 unit tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 19:30:51 +02:00
Chris CoutinhoandClaude Opus 4.7 53e6dba5a2 fix(viz_routes): address PR #767 review — param parity + always-on page_number
- Replace bare int() casts for start/end/context_chars in chunk_context_endpoint
  with _parse_int_param, matching visualization.py bounds (0–10M for offsets,
  0–10K for context_chars), and add the missing end > start guard.
- Initialize page_number from chunk_context.page_number so non-file doc_types
  surface it; include page_number, chunk_index, and total_chunks unconditionally
  in the response. Only highlighted_page_image stays gated on its own truthiness.
- Add a chunk_index forwarding regression test that asserts the new kwargs reach
  get_chunk_with_context and appear in the response payload.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 19:14:43 +02:00
Chris CoutinhoandClaude Opus 4.7 a33a365a69 fix(viz_routes): validate chunk_index/total_chunks bounds in OAuth route
PR #767 review noted that the OAuth viz route used bare int() parsing for
chunk_index and total_chunks while the bearer-token visualization route
validates them via _parse_int_param. Mirror the same bounds check so
total_chunks=0 and negative chunk_index return 400 instead of silently
suppressing adjacent-chunk context.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 18:35:20 +02:00
Chris CoutinhoandClaude Opus 4.7 adcf13f082 refactor(providers): address PR #772 review round 3 — hermetic test, lazy logging, defensive-guard tests
- test_registry.py: stub `mistralai.client.Mistral` in
  `test_registry_mistral_wins_over_ollama`, mirroring the sibling
  picker test, so the test doesn't depend on the SDK accepting
  arbitrary keys.
- openai.py: convert remaining f-string `logger.info(...)` calls to
  lazy `%s` formatting, aligning with the pattern in mistral.py and
  the repo's logging convention.
- test_mistral.py: add four tests covering the defensive RuntimeError
  guards in `embed()` and `_embed_batch_request()` — empty
  response.data, single null embedding, batch null embedding, and
  count-mismatch.
- docs/configuration.md: add `AWS_ACCESS_KEY_ID` and
  `AWS_SECRET_ACCESS_KEY` rows to the env-var reference table; they
  were already mentioned in prose but missing from the table.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 18:28:04 +02:00
Chris CoutinhoandClaude Opus 4.7 20f1770794 refactor(providers): address PR #772 review round 2 — guard, naming, docs, tests
- _retry.py: replace `assert last_error is not None` with explicit
  `if last_error is None: raise RuntimeError(...)` so the original
  rate-limit error is preserved under `python -O`.
- openai.py: drop the `_retry_factory` alias chain; rename the bound
  decorator to `_retry_429` to match the pattern in mistral.py.
- mistral.py: comment the imports so future reviewers understand why
  `from mistralai.client import …` is the canonical path on 2.x (no
  top-level `__init__.py`; no `mistralai.models` subpackage either).
- docs/configuration.md: add `OPENAI_GENERATION_MODEL` and
  `OLLAMA_GENERATION_MODEL` rows to the env-var reference table.
- test_mistral.py: add direct unit test for the `_is_rate_limit`
  predicate (429 → True, 500 → False, missing-attr → False).
- test_registry.py: stub `mistralai.client.Mistral` in the registry
  picker test, mirroring the Ollama sibling, so the test doesn't
  depend on the SDK accepting arbitrary keys.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 18:11:57 +02:00
Chris CoutinhoandClaude Opus 4.7 e360a7782b refactor(providers): address PR #772 review — shared retry, cleaner imports, no-op close
Addresses the Claude Code review on PR #772 plus the SonarCloud S1192 finding:

- Extract `retry_on_rate_limit` into `nextcloud_mcp_server/providers/_retry.py`
  as a parametric decorator. OpenAI and Mistral now share the same backoff
  loop; future providers can reuse it without copy-paste.
- New `tests/unit/providers/test_retry.py` covers the decorator: 429 retry +
  success, non-429 immediate re-raise, MAX_RETRIES exhaustion, default
  predicate, and unrelated exception passthrough.
- Tighten Mistral SDK import to `from mistralai.client.errors import SDKError`
  (the canonical sub-path; the reviewer's `from mistralai.models import
  SDKError` does not exist in mistralai 2.4.5).
- Replace `MistralProvider.close()`'s direct `__aexit__` call with a no-op +
  comment — the Speakeasy-generated client has no public close hook and the
  underlying httpx client is closed by GC.
- Extract the duplicated "Embedding not supported" message to a module-level
  constant (SonarCloud S1192).
- Align `Settings.get_embedding_model_name()` Bedrock check with the registry
  by also considering `bedrock_generation_model`.
- Add the `mock_mistral_client` fixture to
  `test_mistral_no_embeddings_disabled` for parity with the rest of the file.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 17:46:32 +02:00
Chris Coutinho cf59663d9e Merge remote-tracking branch 'origin/master' into feat/mistral-embedding-provider 2026-05-08 17:40:02 +02:00
Chris Coutinho 0690378915 build: Add sonar settings/hooks 2026-05-08 17:39:45 +02:00
Chris CoutinhoandClaude Opus 4.7 3268a13d11 feat(providers): add Mistral embedding provider, route registry through dynaconf
Adds a hosted Mistral embedding option (mistral-embed, 1024-dim) alongside
the existing Bedrock / OpenAI / Ollama / Simple providers. Implementation
mirrors OpenAIProvider: lazy dimension detection with a known-models lookup,
chunked batch requests, defensive index sort, and a 429-aware retry decorator.

In the same change, ProviderRegistry switches from os.getenv to the
dynaconf-backed Settings dataclass so all five providers share a single
configuration path. config.py gains the previously-uncovered Bedrock keys,
the new Mistral keys, the missing OPENAI_GENERATION_MODEL /
OLLAMA_GENERATION_MODEL, and SIMPLE_EMBEDDING_DIMENSION.

Auto-detection priority: Bedrock → OpenAI → Mistral → Ollama → Simple.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 17:25:24 +02:00
Chris CoutinhoandClaude Opus 4.7 90458b6f08 fix(chunk-context): use indexed chunk_index lookup, fix close-after-use bug
Two related bugs surfaced in production while viewing chunks from the
Astrolabe frontend on the AWS-hosted MCP server:

1. PyMuPDF document closed: in _fetch_document_text the fallback path
   referenced pdf_doc.page_count after pdf_doc.close(), raising
   "document closed" and returning None. The slow PDF re-parse already
   completed but its result was discarded. Capture page_count into a
   local before close().

2. Slow/fragile chunk lookup: get_chunk_with_context filtered Qdrant by
   (chunk_start_offset, chunk_end_offset). Those fields are not part of
   the always-indexed payload schema, and with strict_mode enabled they
   yield 400 errors. Even with manually-added indexes the filter is
   fragile if a doc is re-chunked. Switch to chunk_index (always
   indexed) as the primary lookup key, falling back to offset-based
   lookup when callers don't supply it.

Plumb chunk_index/total_chunks through both the management API
(api/visualization.py) and the OAuth viz route (auth/viz_routes.py).
Apply the same change to the highlighted-image lookup so all four
chunk-context Qdrant queries prefer the indexed field.

Skip the slow PDF re-parse fallback entirely for files: when both the
chunk_index and offset Qdrant lookups miss, re-downloading and
re-parsing the source PDF won't find the chunk either, and routinely
exceeds 30s on large documents - which is the proxy timeout in
Astrolabe. Notes/cards keep the document-fetch fallback (cheap).

Removes dead code (_get_file_path_from_qdrant) that was only used by
the now-unreachable file fallback path.

Companion change in the Astrolabe app passes chunk_index from search
results through to the new endpoint params.

---

_This PR was generated with the help of AI, and reviewed by a Human_

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 10:58:48 +02:00
Chris CoutinhoandClaude Opus 4.7 e9e6bcc60a fix(webdav): include fileid in find_by_type SEARCH + address PR #765 review
The default property set in `search_files` omits `<oc:fileid>`, so
`find_by_type` returned descendant dicts with no `file_id` — which
`NextcloudClient.find_files_by_tag` then silently dropped via its
dedup-by-id guard. Net effect: tag-on-folder produced zero expanded
descendants in CI (single-user / nc31, nc32). Mirrors the explicit
property list already used in `WebDAVClient.find_by_tag`.

Also addresses three nits from the PR #765 bot review:
- trim multi-paragraph docstring on `_normalise_search_result`
- trim multi-line docstring on `find_files_by_tag`
- match `is not None` ID-extraction pattern in the descendant loop
- assert positional `mime_type` arg in the unit test

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 01:41:35 +02:00
Chris CoutinhoandClaude Opus 4.7 43c6788555 feat(vector): expand tagged directories for include + apply EXCLUDED_TAGS in scanner
NextcloudClient.find_files_by_tag now mirrors the directory semantics
already used by the exclusion path (issue #710): when a tagged item is
a folder, walk its descendants via WebDAV SEARCH (Depth: infinity) and
include any files matching the MIME filter. Without this, tagging the
root of a corpus with `vector-index` indexed nothing because the tag
applies to the directory only, not to its children.

The vector scanner additionally consults EXCLUDED_TAGS now, so a folder
marked off-limits is skipped even if it (or an ancestor) carries the
include tag — defense-in-depth, matching the "exclusion wins" contract
already enforced by the MCP file tools.

Also addressed a recurring memory-style nit: pre-existing f-string log
lines in find_files_by_tag were converted to lazy %-style.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 00:29:37 +02:00
Chris CoutinhoandClaude Opus 4.7 81c190c9c5 fix(webdav): finish lazy-logging conversion in get_tag_by_name
The two debug calls in get_tag_by_name were left as f-strings when the
method was migrated to _make_request in round 1. Convert to lazy
%-style formatting per repo convention (PR #764 review round 5).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 23:40:53 +02:00
Chris CoutinhoandClaude Opus 4.7 56f01b3499 fix(webdav): address PR #764 review round 4
- Guard against malformed PROPFIND responses where tag["id"] is None
  before calling get_files_by_tag (prevents
  <oc:systemtag>None</oc:systemtag> dispatch).
- Add OCS-APIRequest: true header to get_tag_by_name and
  get_files_by_tag to match every other PROPFIND/REPORT in the file —
  fixes a latent reverse-proxy compatibility hazard.
- Add test_copy_resource_blocks_excluded_source to mirror the
  existing move-source coverage; closes the asymmetric test gap.
- Add test_skips_tag_with_missing_id covering the new fail-open
  branch in _resolve_one_tag.
- Reword _resolve_one_tag docstring: "distinct slot" was misleading
  (tasks append rather than pre-allocate).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 20:24:11 +02:00
Chris CoutinhoandClaude Opus 4.7 2ee4d03e3f fix(webdav): address PR #764 review round 3
Three important issues raised by the latest review on the
get_tag_by_name and get_files_by_tag methods:

1. Add explicit response.raise_for_status() after _make_request in both
   methods. _make_request already raises HTTPStatusError on non-2xx so
   the calls are redundant in practice, but keeping them visible at the
   call site makes the contract self-documenting and prevents a future
   refactor from silently feeding an error body into ET.fromstring.

2. Replace href_path.replace(webdav_prefix, "/") with a startswith +
   slice. str.replace strips every occurrence of the prefix; while no
   real Nextcloud path embeds the prefix mid-string, the fix removes
   the theoretical exposure and matches the pattern used elsewhere in
   the file.

3. Add Content-Type: text/xml to the systemtags PROPFIND headers.
   Other PROPFIND-with-body calls in this file (list_directory line
   240, list_attachments line 1041) include it; the systemtags PROPFIND
   was the only outlier. Same header added to the systemtag REPORT for
   symmetry.

No test changes — the existing get_files_by_tag mock test continues to
pass (the mock response yields valid XML so raise_for_status is a
no-op, and the user-relative path comparison is unaffected by the
prefix-strip swap on a non-adversarial path).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 19:53:07 +02:00
Chris CoutinhoandClaude Opus 4.7 35abfb2e3a fix(webdav): drop anyio.Lock and add integration tests for tag exclusion
Addresses two points from the latest PR #764 review:

1. The anyio.Lock in get_excluded_file_paths bought nothing under
   anyio's cooperative multitasking model (single-threaded between
   awaits, raw set mutations are already safe). _resolve_one_tag now
   builds a local set of paths and appends it to a shared list — list
   append between awaits is safe without a lock — and the caller
   merges via set().union(*results) after the task group completes.
   This removes the cognitive overhead the reviewer flagged without
   changing the public API.

2. Adds tests/integration/test_tag_exclusion.py exercising the
   resolution pipeline end-to-end against a real Nextcloud instance:
   creates a system tag, tags a real file and a real directory,
   verifies get_excluded_file_paths resolves both via real PROPFIND +
   REPORT calls, and verifies is_path_excluded correctly classifies
   exact matches, descendants of tagged directories, and unrelated
   paths. Includes the disabled-feature short-circuit case.

Cleanup runs in reverse order (untag, delete files); per-run uuid
suffix avoids cross-run interference.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 19:33:19 +02:00
Chris CoutinhoandClaude Opus 4.7 d179ca8c8b fix(webdav): address PR #764 review round 2
Addresses the four points raised in the automated review on PR #764:

1. Scope guards on the four search tools (search_files, find_by_name,
   find_by_type, list_favorites) so an excluded `scope` raises ToolError
   instead of silently returning an empty result. Previously an LLM
   could probe the asymmetry between list_directory (raises) and the
   search tools (silent) to infer that an excluded directory exists.
   The 4 search tools now mirror the early-guard pattern from
   list_directory and avoid an unnecessary upstream query for known-
   excluded scopes.

2. Concurrent per-tag resolution in get_excluded_file_paths via
   anyio.create_task_group(). Previously the 2N network calls (1
   PROPFIND + 1 REPORT per tag) ran serially. Per-tag fail-open
   behaviour is preserved by extracting _resolve_one_tag, which
   swallows its own exceptions so a single tag failure does not abort
   the surrounding task group.

3. WebDAVClient.get_tag_by_name and get_files_by_tag now route through
   _make_request, inheriting the @retry_on_429 decorator. Previously
   they bypassed it; with tag exclusion invoked on every WebDAV tool
   call, a transient 429 from the systemtags endpoint was hitting the
   fail-open path instead of being transparently retried.

4. Test coverage: 6 new tests in test_webdav_tools_exclusion.py (4
   scope-guard, 2 missing filter tests for find_by_type and
   list_favorites) and 2 new tests in test_tag_exclusion.py (a
   concurrency proof using an event-barrier that would deadlock under
   sequential execution, and a fail-open-under-task-group test with
   order-independent side_effect callables).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 19:15:39 +02:00
Chris CoutinhoandClaude Opus 4.7 a6c188abbb fix(webdav): address PR #764 review
Six findings raised in the PR review:

🔴 Blocking
- Fail-open on tag-resolution errors. get_excluded_file_paths now
  wraps each tag's get_tag_by_name and get_files_by_tag call in
  try/except; failures log a warning and the tag is skipped, rather
  than propagating to the caller and disabling all WebDAV tools when
  the systemtags endpoint is degraded. Documented in the docstring as
  the intended fail-open behaviour (threat model is preventing
  accidental exfiltration, not surviving server compromise).

🟡 Important
- nc_webdav_list_directory now raises ToolError when the listed path
  itself is tagged, instead of silently returning an empty listing
  after a wasted PROPFIND. Behaviour now mirrors the mutating tools.
- Destination error messages in move/copy/create_directory said "is
  inside" but is_path_excluded matches exact paths too. Reworded to
  "is or is inside".

🟢 Nits
- get_excluded_file_paths log message clarified: N counts
  directly-tagged paths, not total descendants.
- Test isolation: tests/unit/conftest.py already has an autouse
  _reload_dynaconf_after_test fixture that handles teardown. Removed
  the redundant module-local fixture I had drafted; documented the
  reliance in the module docstring instead.
- Added tests/unit/test_webdav_tools_exclusion.py: 12 server-layer
  tests that register the WebDAV tools on a fresh FastMCP and invoke
  each tool's underlying function with a mocked excluded set, asserting
  ToolError is raised / results filtered as expected. Catches future
  guard-integration regressions (e.g. wrong argument order).

Also added two unit tests for the new fail-open behaviour.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 12:26:25 +02:00
Chris CoutinhoandClaude Opus 4.7 22ed9e99a0 feat(webdav): add tag-based file exclusion (#710)
Hide sensitive files/folders from the WebDAV MCP tool surface by
tagging them with a configured Nextcloud system tag. Defence-in-depth
control for users who connect LLMs to accounts holding contracts,
medical records, credentials, etc.

A new EXCLUDED_TAGS env var (comma-separated tag names, empty by
default) gates an exclusion layer that runs at the start of every
WebDAV tool call: tag names are resolved to tag IDs, those IDs are
expanded to the set of tagged paths, then listings/searches are
filtered and read/write/delete/move/copy operations on excluded paths
raise ToolError. Tagged folders exclude their descendants via prefix
match. Empty EXCLUDED_TAGS disables the feature entirely.

The threat model is preventing accidental data exfiltration via the
LLM tool surface — not hiding files from a determined operator. The
docs explicitly recommend creating exclusion tags with
user_assignable=false so the credentials the MCP server uses cannot
remove the tag.

Implementation:

- config.py: add `excluded_tags` to _DEFAULTS, Settings, and the
  _field_map alongside other comma-separated env vars.
- client/webdav.py: get_files_by_tag now requests <d:resourcetype/>
  and surfaces is_directory so tagged directories can recursively
  exclude descendants.
- server/tag_exclusion.py (new): get_excluded_tag_names,
  get_excluded_file_paths, is_path_excluded.
- server/webdav.py: exclusion guards in all 11 WebDAV tools;
  read/write/create/delete/move/copy raise ToolError, list/search
  tools silently filter excluded entries. Existing f-string log
  calls converted to lazy %-style.
- tests: 17 new unit tests covering path-matching edge cases
  (shared-prefix non-match, descendants of excluded dirs), tag-name
  parsing, and get_excluded_file_paths with mocked WebDAV; 1 new
  client test asserting <d:resourcetype/> -> is_directory parsing.
- docs/configuration.md: new "Tag-Based File Exclusion" section with
  per-tool effect table, security guidance, and per-call cost note.
- README.md: feature mention under Key Features.

Closes #710.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 12:12:54 +02:00
Chris CoutinhoandClaude Opus 4.7 b3a7587f1a fix(webhooks): use HTTP 428 instead of 412 for unprovisioned users
428 (Precondition Required, RFC 6585) is the correct semantic — the
request requires the client to complete a prerequisite step (Login Flow
v2 provisioning) before retrying. 412 (Precondition Failed) is for
header-based preconditions like ETags / If-Match.

No behavior change beyond the status code; same JSON payload.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 00:15:55 +02:00
Chris CoutinhoandClaude Opus 4.7 a0e484d95b fix(webhooks): use app-password basic auth for NC API calls
The webhook API endpoints in api/webhooks.py forwarded the inbound MCP
OAuth bearer token directly to Nextcloud as the Authorization header.
Per ADR-022 / docs/login-flow-v2.md the data leg from MCP server to
Nextcloud must use HTTP Basic Auth with the user's stored Login Flow v2
app password — bearer-forwarding requires upstream user_oidc patches that
were never merged and is incompatible with admin endpoints gated by
@PasswordConfirmationRequired (e.g. webhook_listeners/api/v1/webhooks,
which 401s).

PR #760 papered over the symptom for /api/v1/apps by switching to the
permissive /cloud/capabilities endpoint, but the same architectural
mistake remained on list_webhooks / create_webhook / delete_webhook,
which still 500'd on the astrolabe admin UI's preset page.

Changes:
- New helper api/_auth.py:get_basic_auth_for_user(user_id) reads the
  user's app password from encrypted storage and returns
  (username, app_password). Mirrors context.py:_get_client_from_login_flow
  but is callable from Starlette routes (no MCP Context required).
- All four endpoints in api/webhooks.py now use httpx.BasicAuth instead
  of forwarding the OAuth bearer; ProvisioningRequiredError is mapped to
  HTTP 412 so callers can render a "complete provisioning" CTA rather
  than receiving an opaque 500.
- Outbound NC requests now identify the user by the username recorded at
  Login Flow v2 provisioning time (which may differ from the IdP-issued
  user_id) — flowed into WebhooksClient and used for logging.

Tests:
- tests/unit/test_management_apps_endpoint.py: assertions updated to
  verify outbound NC request uses BasicAuth and carries no Authorization
  header. Replaced "missing-Authorization → 500" test with a
  ProvisioningRequiredError → 412 case.
- tests/unit/test_webhooks_api_auth.py (new): cross-endpoint coverage
  for list_webhooks, create_webhook, delete_webhook and the new helper —
  including 412 symmetry for all four endpoints.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 23:50:59 +02:00
Chris CoutinhoandClaude Opus 4.7 285f5174bd fix(test): retry consent handling in login_flow_static_client_token
The oidc app does a JS-driven re-authorize chain after login
(/apps/oidc/redirect → /apps/oidc/authorize → /apps/oidc/consent).
wait_for_load_state("networkidle") can fire during the brief gap before
the consent page renders, so a single _handle_oauth_consent_screen call
right after login often misses the consent div and the OAuth flow
deadlocks waiting for a callback that never arrives.

Move consent handling inside the callback-wait loop and poll for either
the consent page or the callback hit. Loop bound bumped to 60s to give
the JS-driven re-auth headroom.

Confirmed locally: integration test now passes against docker compose
--profile login-flow with the static OIDC client.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 22:52:45 +02:00
Chris Coutinho 079188e16a test: mark management-api integration test with login_flow
`tests/server/login_flow/test_management_api.py` had
`[pytest.mark.integration, pytest.mark.oauth]` while every other test in
`tests/server/login_flow/` uses `[pytest.mark.integration,
pytest.mark.login_flow]`. The single-user CI matrix filter is
`(integration and not keycloak and not login_flow and not
multi_user_basic)`, so the missing `login_flow` mark let this test
collect and run under single-user mode against `localhost:8004` (which
isn't up there, hitting the bug being reported), even though it's
specifically driving the login-flow MCP server.

Also `oauth` isn't a registered marker (see `[tool.pytest.ini_options]`
in pyproject.toml), so it was emitting an unregistered-marker warning.

Replacing the marker aligns this file with its siblings: single-user /
multi-user-basic / keycloak filters all deselect it now, and the
login-flow filter still picks it up.

Verified: `pytest --collect-only -m "<single-user filter>"` reports 2
deselected; `-m login_flow` collects both tests.
2026-05-03 22:44:58 +02:00
Chris CoutinhoandClaude Opus 4.7 148ab8c117 fix(webhooks): use OCS v2 capabilities for /api/v1/apps
The Astrolabe webhooks UI hits /api/v1/apps on the MCP server, which
forwarded the OAuth bearer token to /ocs/v1.php/cloud/apps?filter=enabled.
That OCS endpoint is admin-only AND @PasswordConfirmationRequired —
neither requirement is satisfiable via an OAuth bearer token, so even an
admin user's token returns a silent 401 (no entry in nextcloud.log).

Switch to /ocs/v2.php/cloud/capabilities, which has no admin or password-
confirmation gate, accepts the existing bearer token, and returns a
capabilities map keyed by app id (notes, files, tables, forms, etc.).
This is sufficient for the webhook presets UI to gate available presets
against the running Nextcloud instance's enabled apps.

Bearer is preserved on the outbound call because anonymous capabilities
omits notes/tables/forms — only authenticated capabilities exposes them.

Tests:
- New unit test covers the regression (asserts /ocs/v2.php/cloud/capabilities
  is hit, NOT /cloud/apps), response parsing, sanitized error messages,
  and missing-config paths.
- New integration test under tests/server/login_flow/ drives a real
  OAuth flow against mcp-login-flow with a static OIDC client
  (nextcloudMcpServerUIPublicClient) and asserts /api/v1/apps returns 200
  with core/files in the response.

docker-compose.yml: aligns mcp-login-flow's ALLOWED_MGMT_CLIENT with
mcp-multi-user-basic so the same static-client test fixture works for both.

Follow-up to homelab-argocd #1608, which set ALLOWED_MGMT_CLIENT in
production but didn't unblock the webhooks flow.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 22:26:35 +02:00
Chris CoutinhoandClaude Opus 4.7 b875eaf069 fix(auth): address PR #758 round-7 medium/minor review
- Gate browser session creation on a successful refresh token. When the
  IdP returns no refresh token, SessionAuthBackend would silently reject
  every subsequent request and bounce the user back to /oauth/login in a
  loop. The callback now bails with a 400 + correlation ID + actionable
  hint about offline_access *before* writing browser_sessions or setting
  the cookie. Pinned by a new end-to-end unit test.
- Evict orphaned browser_sessions rows in SessionAuthBackend when the
  associated refresh token is gone, instead of letting them accumulate
  until TTL cleanup. Best-effort; deletion errors stay non-fatal.
- Demote identity-bearing logs in the Flow 2 OAuth callback (user_id,
  scopes, audience, expires_at) from INFO to DEBUG so they don't leak
  into multi-tenant log aggregation on every provision.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:21:12 +02:00
Chris CoutinhoandClaude Opus 4.7 27fcf05d3a fix(auth): address PR #758 round-7 important review
- Coerce refresh_expires_in to int before arithmetic in both callback
  paths so IdPs that serialize the field as a JSON string (e.g. AWS
  Cognito) don't trigger an unhandled TypeError 500.
- Drop the orphaned oauth_session row written by _check_logged_in. The
  canonical Flow 2 row is created by generate_oauth_url_for_flow2 keyed
  by `state`, which is what the unified callback looks up; the
  flow2_<hex> session_id was never matched and just churned the table
  for 10 minutes per call.
- Match delete_cookie attributes (httponly, secure, samesite) to the
  set_cookie call on logout so browsers reliably evict the cookie even
  on implementations that consider security flags during deletion.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 13:36:36 +02:00
Chris CoutinhoandClaude Opus 4.7 ec9b9b2a75 fix(auth): address PR #758 round-6 medium/low review
Five findings from the latest review on #758 (2 medium, 3 nit):

Medium:
- browser_oauth_routes.oauth_login_callback + oauth_routes.oauth_callback_nextcloud:
  fail closed with 400 when the oauth_session row is unknown/expired. Previously
  both callbacks fell through with code_verifier="" and expected_nonce=None,
  silently bypassing the PKCE + nonce protections introduced in earlier rounds.
  Symmetric unit tests pin both contracts.
- token_utils.verify_id_token: use secrets.compare_digest for the nonce check
  instead of short-circuit !=. Mirrors the sibling PKCE verifier comparison;
  closes the last secret-equality timing-side-channel surface in the auth path.

Nit:
- Tighten the comment at all 4 mcp_authorization_code/code_verifier store +
  retrieve sites so a future refactor sees the field reuse immediately
  (renaming the column requires a schema migration).
- _should_use_secure_cookies: explicit string normalisation instead of
  bool(settings.cookie_secure). Dynaconf normally coerces but tests / direct
  settings.set calls can leave the raw string in place — bool("false") is True.
  New parametrized unit tests cover the coercion matrix + http/https fallback.
- oauth_routes.py:591 f-string log converted to lazy %s formatting (folded into
  the Flow 2 callback rewrite).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 13:03:49 +02:00
Chris CoutinhoandClaude Opus 4.7 e2955e8246 fix(auth): address PR #758 round-5 medium/low review
Three findings from the latest review on #758 (1 medium, 2 low):

Medium:
- browser_oauth_routes.oauth_logout: move delete_browser_session into a
  finally block so an error from delete_refresh_token can no longer leave
  an orphan browser_sessions row. The orphan was not exploitable
  (SessionAuthBackend rejects sessions without a live refresh token), but
  it lingered until the hourly cleanup cron — a correctness gap. New
  regression test pins the fix.

Low:
- oauth_callback_nextcloud: drop redundant ``or None`` from
  ``expected_nonce=nonce``. ``nonce`` is already ``str | None`` and
  ``secrets.token_urlsafe`` never produces an empty string, so the
  coercion was a no-op that could mislead future readers into thinking
  empty-string was a valid skip-the-check path.
- storage.RefreshTokenStorage.initialize: fail fast at startup when
  SQLite < 3.35, since ``DELETE ... RETURNING`` (used in
  ``delete_browser_session``) needs that minimum. Ubuntu 20.04 ships
  3.31 and would otherwise hit OperationalError on every logout.
  Prerequisite also documented in docs/installation.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 01:37:38 +02:00
Chris CoutinhoandClaude Opus 4.7 a995155bd4 fix(deck): address PR #759 round-3 review feedback
- Drop "(in-place)" from filter-helper docstrings; callers should
  consume the return value, mutation is an implementation detail.
- Document that deck_get_archived_stacks always returns cards (an
  archived stack without its cards has no audit value); point to
  description_max_length for size control.
- Document that deck_get_cards applies filtering client-side, so it
  is network-equivalent to deck_get_stack(include_cards=True).
- Pin the empty-list contract: a stack with all-archived cards and
  include_archived_cards=False yields cards == [] (loaded but empty),
  not cards is None (explicitly suppressed).
- Add explicit one-character-over-limit truncation test alongside the
  existing exact-boundary test.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 01:09:54 +02:00
Chris CoutinhoandClaude Opus 4.7 b696541918 fix(auth): address PR #758 round-4 review
Seven findings from the latest review on #758 (3 medium, 4 low/nit):

Medium:
- storage.py: replace 5 ``assert self.cipher is not None`` sites with
  explicit ``RuntimeError`` so missing TOKEN_ENCRYPTION_KEY can't silently
  become an AttributeError under ``python -O``
- session_backend.py: document the silent-invalidation invariant —
  refresh-token TTL expiry without explicit logout deliberately makes
  the browser session unusable; future readers must not relax it
- server/oauth_tools.py: drop user_id from the Flow 2 session_id
  identifier — use ``flow2_{secrets.token_hex(16)}`` so audit logs and
  DB rows don't carry user_id in the session_id field

Low / nit:
- token_utils.py: drop _fetch_locks dict entry in finally so a probed
  deployment can't grow the lock dict without bound; coalescing test
  now pins the invariant with len(_fetch_locks) == 0
- browser_oauth_routes.py: strip trailing slash from settings.nextcloud_host
  before constructing the well-known URL so a host configured as
  ``https://cloud.example.com/`` doesn't produce a double-slash
- browser_oauth_routes.py: add comment explaining the three-layer CSRF
  policy on the mcp_session cookie set (SameSite=Lax + POST-only logout
  + Origin/Referer check)
- oauth_routes.py: convert all 23 f-string log calls to lazy %-style
  per the CLAUDE.md / memory feedback_lazy_logging convention

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 00:57:12 +02:00
Chris CoutinhoandClaude Opus 4.7 b7805c2180 fix(deck): address PR #759 round-2 review feedback
- Move description_max_length validation to tool layer
  (_validate_description_max_length), matching the existing
  _validate_comment_message pattern; helper now trusts callers per
  CLAUDE.md ("validate at system boundaries only").
- Fix mutation/return inconsistency: deck_get_stacks now uses a list
  comprehension to capture _apply_stack_filters' return, matching
  deck_get_stack / deck_get_archived_stacks.
- Rename include_archived -> include_archived_cards on deck_get_cards
  and _apply_card_filters for consistency with deck_get_stacks.
- Route deck_get_archived_stacks through _apply_stack_filters so
  future filters apply uniformly to active + archived paths.
- Trim _truncate_card_descriptions docstring to one line; add inline
  comment in _apply_stack_filters explaining the breaking-change
  default (mirrors Deck UI archived-card filtering).
- Replace fragile call_args[0][1] with call_args.args[1] in the
  archived-stacks client test.
- Modernize Optional[X] -> X | None throughout deck.py (adjacent
  cleanup called out in the review).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 00:50:56 +02:00
Chris CoutinhoandClaude Opus 4.7 7d633a945d fix(deck): address PR #759 review feedback
- Validate description_max_length is positive (raises ValueError on 0
  or negative); the prior code would have wiped descriptions to a
  single ellipsis character on description_max_length=0.
- Extract filter logic into testable module-level helpers
  (_apply_board_filters, _apply_stack_filters, _apply_card_filters)
  and replace the dense `continue`-based loop in deck_get_stacks with
  the reviewer's elif form.
- Document the truncation length quirk in the helper docstring (result
  is description_max_length + 1 chars when truncation fires).
- Add 13 new unit tests covering the include/exclude flags on board,
  stacks, and flat card lists, plus the new validation paths and an
  explicit "description fits within limit, no ellipsis" case.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 00:41:14 +02:00
Chris CoutinhoandClaude Opus 4.7 d8cd073e66 feat(deck): add response filters and archived stacks tool
Add filtering options to deck read tools to keep responses compact on
boards with accumulated cards/comments, and expose archived stacks so
agents can audit completed work that has been archived off the active
board.

- deck_get_board: include_acl, include_users, include_labels
- deck_get_stacks/deck_get_stack: include_cards, include_archived_cards,
  description_max_length
- deck_get_cards: include_archived, description_max_length
- New deck_get_archived_stacks tool wrapping the existing client method

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 00:00:09 +02:00
Chris CoutinhoandClaude Opus 4.7 3a4fa8adc8 fix(auth): address PR #758 round-3 final review
Seven findings from the latest review on #758, plus a regression test
catching the substance of the cache-stampede fix:

- verify_id_token: widen id_token annotation to str | None to match
  callers passing nc_token_response.get("id_token")
- extract_user_id_from_token: use JSON-RPC reserved error code -32001
  instead of -1
- _get_cached: per-URL anyio.Lock dict + meta-lock coalesces concurrent
  cache misses into a single IdP fetch (mirrors token_broker.py idiom)
- delete_browser_session: collapse SELECT+DELETE into atomic
  DELETE ... RETURNING user_id (SQLite >= 3.35)
- new test_origin_normalise.py: parametrized port/scheme/host equivalence
  cases for the CSRF Origin guard
- browser_oauth_routes: correct misleading "PR #758 finding 5" cross-
  references (finding 5 was Fernet-key hardening, not CSRF)
- ASProxySession.nonce: make required, drop spurious "legacy session"
  default; reword the in-flight `or None` comment to reflect that
  ASProxySession is purely in-memory
- new test_get_cached_coalesces_concurrent_misses: pins the
  cache-stampede protection — fires 10 concurrent _get_cached calls and
  asserts exactly one HTTP fetch

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 23:14:07 +02:00
Chris CoutinhoandClaude Opus 4.7 9d0e7dcebe fix(auth): address PR #758 round-3 review
- Flow 2 (oauth_authorize_nextcloud) now generates a nonce, stores it on
  the oauth_session row, forwards it to the IdP, and verifies it via
  expected_nonce in oauth_callback_nextcloud — closes the last replay-
  protection gap (round-3 finding 1).
- _origin_matches_self fails closed when mcp_server_url is missing
  instead of allowing the logout, and the diagnostic log is promoted
  from warning to error so the misconfiguration is monitorable
  (round-3 finding 2). New regression test pins the new behaviour.
- The five user_id-accepting helpers in oauth_tools.py (get_provisioning_status,
  provision_nextcloud_access, revoke_nextcloud_access, check_provisioning_status,
  check_logged_in) are renamed with leading underscores to make the
  trust boundary structural rather than documentary
  (round-3 finding 3).
- create_browser_session and delete_browser_session now emit audit_log
  rows so session establishment / teardown match the pattern used by
  the rest of the security-relevant storage operations
  (round-3 nit 5). delete_browser_session selects user_id before delete
  so the audit row is attributable.
- oauth_login_callback no longer reflects raw IdP-error text or
  exception strings into the HTML failure page; users see a generic
  "internal error occurred" message + a correlation ID, with the
  detail logged server-side keyed by the same ID (round-3 nit 6).
  The XSS regression test is updated to pin the stricter contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 22:40:06 +02:00
Chris CoutinhoandClaude Opus 4.7 c33d52ea91 fix(auth): address PR #758 round-2 review
- oauth_login_callback's integrated-mode token-exchange branch now reuses
  the shared discovery cache via get_oidc_discovery (round-2 finding 1).
- AS proxy flow now generates an OIDC nonce in oauth_authorize, stores it
  on ASProxySession, forwards it to the IdP, and passes it as
  expected_nonce to verify_id_token in _oauth_callback_as_proxy
  (round-2 finding 2).
- Consolidate the two parallel discovery caches: oauth_routes' local
  _discovery_cache and _get_cached_discovery are removed; all callers
  now go through token_utils.get_oidc_discovery, which acquires the
  follow_redirects=True knob it needs for Nextcloud installs without
  pretty URLs (round-2 finding 3).
- Demote per-user INFO logs in oauth_tools.py (check_logged_in,
  get_provisioning_status) to DEBUG; the elicitation auth URL is no
  longer logged because it contains a sensitive state token
  (round-2 finding 4).

Also pin nonce binding behaviour with a new unit test that asserts
_oauth_callback_as_proxy forwards session.nonce to verify_id_token, and
update test mocks to track the cache consolidation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 21:56:08 +02:00
Chris CoutinhoandClaude Opus 4.7 4c84d82984 fix(auth): address PR #758 auto-review (id-token verify, nonce, CI key)
Blocking:
- AS proxy callback now calls verify_id_token before caching the proxy
  code so a tampered IdP response can't smuggle identity claims.

Important:
- Browser OAuth flow generates and verifies an OIDC nonce; new alembic
  migration 006 adds the nonce column to oauth_sessions.
- _origin_matches_self logs a warning when CSRF check is bypassed.
- oauth_tools.py uses get_shared_storage instead of fresh handles.

Nits:
- New token_utils.get_oidc_discovery shares the 5-minute cache with
  verify_id_token; oauth_login (integrated) and _revoke_refresh_token_at_idp
  now use it instead of issuing fresh discovery fetches.
- Drop typing.Optional from oauth_tools.py in favour of X | None.

CI:
- test.yml generates an ephemeral Fernet TOKEN_ENCRYPTION_KEY per run
  with openssl, removing the dependency on a missing repo secret.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 20:48:25 +02:00
Chris CoutinhoandClaude Opus 4.7 2ef4bfc4af fix(auth): fail closed on missing sub claim, delete Flow 2 callback session
Addresses the two remaining 🟡 findings from the PR #758 follow-up review:

  1. extract_user_id_from_token previously fell back to "default_user" when
     the verified access token had no sub claim. In a multi-tenant deployment
     a malformed IdP token could have bucketed every request under a single
     sentinel user, risking cross-tenant data exposure. The function now
     raises McpError on that branch; the BasicAuth no-token sentinel path is
     preserved.

  2. oauth_callback_nextcloud (Flow 2) read the PKCE code_verifier from
     oauth_sessions but never deleted the row, leaving the verifier valid for
     the full 10-minute TTL. The row is now deleted eagerly inside the same
     branch, mirroring oauth_login_callback in browser_oauth_routes.

Also wires TOKEN_ENCRYPTION_KEY through the docker-compose step in the CI
test workflow so the integration matrix can boot — every job had been
failing fast on the ${TOKEN_ENCRYPTION_KEY:?...} interpolation guard added
in PR #758 finding 5.

Tests pin both fixes (test_token_utils_user_id.py,
test_oauth_callback_session_cleanup.py).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 19:46:36 +02:00
Chris CoutinhoandClaude Opus 4.7 2d340a5a6b fix(auth): address PR #758 follow-up review
Six findings from the latest claude-bot review on PR #758:

- JWKS cache had no kid-miss refresh path (Medium): on IdP key
  rotation every login failed for up to _OIDC_CACHE_TTL. Evict
  and refetch once before raising, per OIDC core §10.1.1.
- _should_use_secure_cookies fell back to nextcloud_host scheme,
  but the cookie is issued by the MCP server. Switch to
  settings.nextcloud_mcp_server_url so split-scheme deployments
  get the right Secure flag.
- _origin_matches_self compared raw netloc strings, which include
  the port. Browsers omit default ports per RFC 6454 §6.2; an
  mcp_server_url like :443 falsely 403'd every legitimate logout.
  Normalise (scheme, host, port) tuples with default ports stripped.
- delete_oauth_session exists in storage.py — drop the stale
  "we don't have this method" comment and call it eagerly so
  replays can't be processed and the table doesn't accumulate
  completed-but-not-yet-expired browser-login rows.
- extract_user_id_from_token's unused ctx param renamed to _ctx
  to signal "intentionally unused" at the signature level.
- provisioning_decorator instantiated RefreshTokenStorage per
  call. Switch to get_shared_storage() for the lock-protected
  process-wide singleton.

Plus pre-push self-review catch: lazy-logging on the unchanged
except arm in session_backend.py.

Adds 5 regression tests:
  - JWKS rotation: success on refetch
  - JWKS rotation: still-missing-kid surfaces original error
  - JWKS rotation: network error during refresh wrapped as
    IdTokenVerificationError
  - default-port CSRF: explicit :443 in config + portless Origin
  - default-port CSRF: portless config + explicit :443 in Origin
  - scheme-mismatch CSRF: same host, different scheme rejected

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 18:59:34 +02:00
Chris CoutinhoandClaude Opus 4.7 af25c281bf fix(auth): use Settings for OIDC env vars in token revocation helper
After merging master, _should_use_secure_cookies was refactored to read
from Settings instead of os.getenv, which dropped `import os` from
browser_oauth_routes.py — leaving _revoke_refresh_token_at_idp's four
remaining os.getenv() calls undefined (CI ruff F821).

Migrate the helper to the same Settings-based pattern:
  - oidc_discovery_url     → settings.oidc_discovery_url
  - OIDC_CLIENT_ID         → settings.oidc_client_id
  - OIDC_CLIENT_SECRET     → settings.oidc_client_secret
  - NEXTCLOUD_HOST         → settings.nextcloud_host

Drive-by: the previous fallback read OIDC_CLIENT_ID, but the canonical
env var per env.sample / docker-compose is NEXTCLOUD_OIDC_CLIENT_ID.
The Settings layer handles this mapping via dynaconf, so the corrected
name is now used automatically.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 18:32:19 +02:00
Chris Coutinho 50a97ffcb3 Merge remote-tracking branch 'origin/master' into security/oauth-session-hardening-626 2026-05-02 18:29:53 +02:00
Chris CoutinhoandClaude Opus 4.7 931ee602eb fix(auth): address PR #758 review — XSS, CSRF, open redirect, JWKS cache
Addresses all 9 findings from the review on PR #758:

Blocking:
- _revoke_refresh_token_at_idp now reads config from oauth_ctx["config"]
  (the production-shaped nested dict). Previously read flat keys, causing
  IdP revocation to silently no-op in production. Test fixtures rebuilt
  to the realistic nested shape so the bug can't regress unnoticed.
- HTML error responses in oauth_login_callback now wrap IdP-controlled
  error_body, str(e), and the attacker-controlled error/error_description
  query params in html_escape. New test_browser_oauth_xss.py pins this.

Important:
- New _safe_next_url helper validates the ?next= query param at write
  time (oauth_login), in oauth_logout, and on read from the session row
  in oauth_login_callback. Blocks https://, // (protocol-relative), and
  CRLF/whitespace injection.
- verify_id_token now caches discovery + JWKS (5-min TTL) using the
  same pattern as oauth_routes._get_cached_discovery. New caching
  regression test pins to one fetch per URL across multiple calls.
- /oauth/logout is now POST-only at the route layer (defeats passive
  CSRF via <img src>). oauth_logout also validates Origin/Referer
  against the configured mcp_server_url. Logout UI in user_info.html
  converted from <a href> to <form method="post">.
- New storage.cleanup_expired_browser_sessions() called from the hourly
  cleanup loop in app.py — previously these rows accumulated for users
  who never explicitly logged out.

Nits:
- Demoted INFO logs that leaked oauth_config.keys() / client_id /
  token-storage state to DEBUG. Operator-relevant outcome lines
  (login successful, refresh token stored, logged out) stay INFO.
- verify_id_token algorithms widened to RS256, PS256, ES256 — covers
  Azure AD (PS256) and Cognito/some Keycloak realms (ES256). Symmetric
  and "none" remain off the allowlist.
- Migrated all Optional[X] usages in auth/storage.py to X | None per
  CLAUDE.md.

Breaking change: GET /oauth/logout now returns 405. The in-tree logout
UI was migrated to a POST form; any external bookmark or curl-based
caller that relied on GET will need to switch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 18:26:39 +02:00
Chris CoutinhoandClaude Opus 4.7 ce80a36877 fix(auth): address PR #757 round-3 review feedback
Three review items from the third-round review on PR #757:

- scope_authorization: split the combined logger.warning(error_msg) in
  the require_scopes decorator's missing-app-password branch into two
  lazy %-style logger calls (one per branch), keeping the f-string
  error_msg for the exception only. The else branch also logs the
  elicit_result for diagnostics. Bypassing lazy %-interpolation in
  security-sensitive code formatted the message regardless of log level
  and matched the repo-wide lazy-logging preference; the new code now
  conforms.
- config + browser_oauth_routes: wire COOKIE_SECURE through Settings
  (cookie_secure: bool | None = None) so _should_use_secure_cookies()
  reads it via get_settings() rather than os.getenv. Completes the
  consolidation pass that touched this file in commit 7464340 and
  removes the last raw os.getenv from browser_oauth_routes.py
  (import os dropped). Dynaconf auto-coerces "true"/"false" → bool;
  "1"/"0" arrive as int and are normalised by an explicit bool() at
  the consumer.
- elicitation: clarify the _astrolabe_settings_url docstring to call
  out that the empty-string case is also a None-return path (matches
  the existing `if not base:` guard).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 17:24:50 +02:00
Chris CoutinhoandClaude Opus 4.7 15dbb26349 fix(auth): harden OAuth/session for hosted multi-tenant deployment (#626)
Pre-launch hardening for the hosted Astrolabe Cloud offering. Addresses
all five findings raised in #626 (Tim Kaufmann, code review of v0.65.0).
Re-verified against master before fixing.

Finding 3 (LLM-controllable user_id) — drop user_id from the public
signatures of provision_nextcloud_access, revoke_nextcloud_access,
check_provisioning_status, check_logged_in. Tool wrappers now always
derive identity from the verified AccessToken; user_id is no longer
accepted as MCP input. Adds parameterized CI-guard test that locks the
schema.

Finding 2 (predictable session cookie) — replace mcp_session=<user_id>
cookie with a cryptographically random session_id mapped server-side
(new browser_sessions table, alembic 005). Cookie value is opaque,
expires, revocable. SessionAuthBackend looks up user_id via the new
mapping and additionally requires a refresh token to fail closed.

Finding 4 (logout doesn't revoke refresh token) — oauth_logout now
calls the IdP revocation_endpoint (RFC 7009) when advertised, deletes
the stored refresh token regardless, and clears the browser_sessions
row. Cleanup is best-effort: logout always 302s.

Finding 1 (unverified ID token decodes) — verify_id_token helper does
JWKS signature + issuer + audience + exp + nonce checks per OIDC core
3.1.3.7. Used by both OAuth callback handlers (browser + MCP). Removes
the four "verify_signature: False" decodes that previously trusted IdP
claims unconditionally. Drops dead-code _validate_token_audience in
token_broker. Refactors token_utils + provisioning_decorator to read
user_id from the verified AccessToken instead of re-decoding the JWT.

Finding 5 (hardcoded Fernet keys in docker-compose.yml) — replace the
three inline TOKEN_ENCRYPTION_KEY values with required env var
interpolation; document in env.sample.

Test coverage: 4 new unit test modules (signature pinning, browser
sessions, ID-token verification, logout + revoke + session backend).
693 unit tests pass; ruff/format/ty clean.

Migration note: existing browser admin-UI sessions become invalid on
rollout (cookies are looked up against the new browser_sessions table,
which starts empty). Users re-login. MCP API access is unaffected.

Tracked on Astrolabe Cloud POC board card #37.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 17:03:57 +02:00
Chris CoutinhoandClaude Opus 4.7 f31d0544b7 fix(auth): invalidate scope cache on web/REST provisioning paths
The elicitation flow points users to the Astrolabe web route or the
BasicAuth REST endpoint to provision their app password. Both paths
stored the password without clearing the in-process scope cache, so a
user who provisioned through them would keep hitting
ProvisioningRequiredError for up to _SCOPE_CACHE_TTL (5 min) afterwards.

Add invalidate_scope_cache(user_id) to both write-paths (matching the
existing pattern in nc_auth_check_status), correct the now-misleading
comment in scope_authorization.py to name all three invalidation paths,
and add a one-line hint above the first elicitation patch in the test
file so future authors don't "fix" the patch target to the wrong module.

Addresses PR #757 round-3 review feedback.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 16:59:53 +02:00
Chris CoutinhoandClaude Opus 4.7 7464340763 fix(auth): address PR #757 round-2 review feedback
Four review items from the second-round review on PR #757:

- scope_authorization: broaden the post-elicit retry message to acknowledge
  the 5-minute scope-cache TTL — if the LFv2 poller is still in-flight at
  acknowledge-time, the immediate retry can still hit a stale cache.
- elicitation: extract a shared `_run_elicit(ctx, message, schema, *,
  log_label)` helper so `present_login_url` and
  `present_provisioning_required` no longer duplicate the
  hasattr-guard / try-NotImplementedError / try-Exception fallback block.
  The data-acknowledged warning specific to login-flow stays in
  `present_login_url` so behaviour is preserved exactly.
- elicitation: detect missing http:// / https:// scheme in
  `_astrolabe_settings_url`, log a warning, and return None — caller
  renders the safe tool-only fallback instead of producing a broken link.
  New unit test locks this in.
- browser_oauth_routes: replace the stray
  `os.getenv(\"NEXTCLOUD_HOST\")` in `_should_use_secure_cookies` with
  `get_settings().nextcloud_host` for consistency with the rest of the
  file (PR #757 review nit).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 16:33:43 +02:00
Chris CoutinhoandClaude Opus 4.7 f3256e515e refactor(config): consolidate NEXTCLOUD_PUBLIC_ISSUER_URL through Settings
Lift NEXTCLOUD_PUBLIC_ISSUER_URL out of raw os.getenv reads into
Settings.nextcloud_public_issuer_url across all 8 production call sites
(app.py x2, oauth_routes.py x2, browser_oauth_routes.py,
provision_routes.py, userinfo_routes.py, elicitation.py). cli.py
remains the env-write source so the existing config-by-flag pipeline
still works.

Also addresses remaining PR #757 review nits:
- elicitation.py: align URL-present/absent wording on "open in your
  browser" so users don't try clicking in the terminal
- test_scope_authorization_stored.py: lock in the deliberately-shared
  fall-through branch with explicit declined/cancelled decorator tests
- test_elicitation.py: switch from monkeypatch.setenv to
  patch(get_settings) since Settings is now the canonical surface

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 16:16:19 +02:00
Chris CoutinhoandClaude Opus 4.7 822a8fe2ed fix(auth): address PR #757 review feedback
- Branch the ProvisioningRequiredError message on the elicit result so a
  user who acknowledged the prompt isn't told to call
  nc_auth_provision_access (which would loop an LLM that just confirmed
  via elicitation). Other paths keep the existing instruction.
- Convert present_login_url's f-string logger.warning to lazy %s, matching
  present_provisioning_required and the repo's lazy-logging preference.
- Add a test for NEXTCLOUD_PUBLIC_ISSUER_URL trailing-slash normalization.
- Strengthen the decorator-elicits test: split into the "accepted" and
  "message_only" branches so the error-message change is regression-tested.

Refs: cbcoutinho/nextcloud-mcp-server#757#issuecomment-4363552487

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 15:34:56 +02:00
Chris CoutinhoandClaude Opus 4.7 da60322597 docs(login-flow): add external-IdP setup section
Calls out the apps-to-install matrix (user_oidc required, oidc skip,
astrolabe optional), the OIDC clients to register and what each is for,
the per-app scope advertisement requirement on the IdP side, and the
"OAuth succeeded but Nextcloud returns 401" diagnosis path.

Mined from the cbcoutinho/nextcloud-mcp-server#752 thread.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 12:02:26 +02:00