- pdf_highlighter.compute_chunk_bboxes_batch: drop unused chunk_text
destructure (SonarQube finding), and replace positional
page_boundaries[page_num - 1] with a key-based next() match so
reordered or non-1-indexed boundaries can't silently shift the bbox.
Convert touched f-string log to lazy %s formatting.
- vector/processor: rename the trace_operation span from
"vector_sync.generate_highlights" to "vector_sync.compute_chunk_bboxes"
to match what the function actually does.
- Add test_compute_chunk_bboxes_handles_unordered_page_boundaries —
reverses the boundaries list and asserts identical results to the
in-order case, guarding the boundary-lookup regression class.
- Pin pre-push-review skill to sonnet model.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add a fixed-UUID sentinel point written after a successful doc_id
backfill so subsequent restarts retrieve it and short-circuit the
O(N) scroll. Sentinel has no user_id/doc_id/doc_type payload so
production search filters never see it.
- Pre-fetch payload_schema in _ensure_keyword_payload_indexes and
silently skip fields that are already indexed; the "Created KEYWORD
payload index" INFO log fires only on actual creation.
- Narrow stale `int | str` doc_id annotations to `str` across
search/verification.py (BatchVerifier return type, per-verifier
accessible sets, by_type / accessible_by_type / inaccessible
collections); drop the now-redundant `type(d).__name__` prefix in
the dropped-docs log.
- Align the backfill log message with the PR description's
"Running doc_id backfill" promise; add a caller cross-reference to
the wait=True comment.
- Fix _get_file_path_from_qdrant docstring (file_id is str, not numeric).
- Convert legacy `id=1` to `id="1"` in test_search_result.py to match
the SearchResult.id: str annotation.
Three new unit tests cover sentinel-found, sentinel-written, and
skip-existing-index branches; existing backfill tests pass dimension
and explicit retrieve.return_value=[] for the no-sentinel path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- chunk_bboxes is now dict[int, list[tuple[...]]] holding the bbox list
directly, not {"bbox": ..., "page": ...}. The page from text-search was
stored but never read; page_number from offset-based assignment is
authoritative for the Qdrant payload.
- Add two unit tests for the documented omission contract: chunks whose
offsets fall outside every page boundary, and chunks whose text cannot
be located on the rendered page, are silently dropped from the result.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses the deferred half of PR #767 review issue 2: instead of just
documenting the "0/N misreport" with a logger.warning, propagate the
caller's None for chunk_index through the dataclass, position markers,
and response builders so callers can distinguish "unknown position"
from "actually chunk 0".
Changes:
- ChunkContext.chunk_index: int → int | None
- _insert_position_markers: chunk_index parameter is int | None; when
None, renders "Chunk ?/N" instead of "Chunk 1 of N"
- get_chunk_with_context: drops the effective_chunk_index local entirely.
Passes chunk_index (may be None) directly into both ChunkContext and
_insert_position_markers, in both the Qdrant fast path and the
doc-text fallback.
- Fast path: when chunk_index is None and chunk_text was retrieved via
the offset lookup (notes/cards), skip the adjacent-chunk fetch.
Index arithmetic from a default 0 would query the chunks at positions
-1 and 1 even when the actual chunk is, say, 5/20 — silently producing
wrong "before"/"after" text. Mark both sides as truncated instead.
- Drop the now-redundant logger.warning in the doc-text fallback (the
response correctly communicates the unknown state via chunk_index=None).
Both existing response builders (`api/visualization.py:657` and
`auth/viz_routes.py:717`) already serialise `chunk_context.chunk_index`
unconditionally; `None` becomes JSON `null`. No route changes needed.
Adds 5 regression tests:
- ChunkContext.chunk_index propagates as None in fast path
- ChunkContext.chunk_index propagates as None in doc-text fallback
- Fast path with chunk_index renders "Chunk N of M" correctly
- _insert_position_markers renders "?/N" for None chunk_index
- _insert_position_markers renders explicit index when supplied
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- _ensure_keyword_payload_indexes: distinguish 400 (schema conflict, warning)
from other status codes (5xx/network, error) so a transient outage doesn't
silently leave the collection unindexed.
- build_search_result_from_point: use .get("doc_id") + return None on missing
instead of KeyError-crashing the search; reverse metadata merge order so
payload-derived chunk_index/total_chunks win over caller-supplied extras.
- docs/configuration.md: restore the OpenAI/Mistral/Bedrock/Simple provider
sections + reference-table rows that were dropped in the rebase. Reword
the "Startup migrations" bullet to describe what the code actually does
(no sampling — full scroll, zero writes when clean). Add operator note
about the SemanticSearchResult.id TypeError path.
- tests: pytest.approx for float equality (Sonar python:S1244); coverage
for non-400 → ERROR, payload={doc_id: None}, and missing doc_id key.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Drop chunk_bbox_page from Qdrant payload — viz endpoints never read it
(page_number is the canonical PDF page field).
- Bump upsert BATCH_SIZE 10 → 100 now that payloads no longer carry PNGs.
- compute_chunk_bboxes_batch: move doc.close() into finally, replace
unused stored_page_num with _.
- purge_page_images.py: switch to anyio.run() per project convention,
and wrap AsyncQdrantClient in try/finally so the aiohttp session is
always closed (the class doesn't implement async-context-manager).
- Decorate new bbox unit tests with @pytest.mark.unit so they run under
the fast-feedback selector.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Latest reviewer comment flagged five items on top of the original PR. This
commit addresses every one:
🟡 1. Skip the offset-based Qdrant fallback for `doc_type=file` when
`chunk_index` is supplied. Qdrant Cloud's strict mode rejects unindexed
filter fields with HTTP 400, which `_get_chunk_from_qdrant` catches and
logs at `logger.error` — masking real Qdrant problems in monitoring.
Notes/cards keep the offset fallback (cheap, useful for legacy data).
🟡 2. Add a `logger.warning` and clarifying inline comment in the doc-text
fallback path when `chunk_index` is None — surfaces the pre-existing
"0/N misreport" so callers can detect it. Type-nullability propagation
is deferred to a follow-up (out of scope for this hotfix).
🟢 3. Simplify `if chunk_text and doc_id_int is not None:` →
`if chunk_text:` with an inner `assert doc_id_int is not None` for
`ty` narrowing. The outer second clause was dead.
🟢 4. Add `doc_type` `FieldCondition` to the offset-based image lookup in
both `visualization.py` and `viz_routes.py` for parity with the
`chunk_index` branches.
🟢 5. Inline the `chunk_filter` local in `visualization.py` directly into
the `must=[]` list (matches `viz_routes.py` style).
Adds `tests/unit/test_chunk_context_offset_gate.py` with three regression
tests covering the gate matrix: (file, with-index → skip offset),
(note, with-index → still tries offset), (file, no-index → still tries
offset). Lives at top-level rather than `tests/unit/search/` to side-step
a pre-existing circular-init issue in `nextcloud_mcp_server.search`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-chunk PDF page renders (~150–700 KB base64 PNG each) were the dominant
disk consumer in production, repeatedly tripping `No space left on device:
WAL buffer size exceeds available disk space` on welcomed-malamute Qdrant.
Replace the inline highlighted_page_image / highlighted_page_number /
highlight_count fields with a small `chunk_bbox` field:
list[(x0, y0, x1, y1)] of normalized [0, 1] floats, ~32 bytes per chunk.
Astrolabe (the only known consumer) renders the highlight client-side as
a percentage-positioned overlay on top of the existing /api/v1/pdf-preview
render-on-demand path (cbcoutinho/astrolabe#76).
- pdf_highlighter: new compute_chunk_bboxes_batch() that reuses the
existing _find_chunk_bbox text-search path, skipping all pixmap/PIL/PNG
work.
- processor: store chunk_bbox + chunk_bbox_page in the Qdrant payload,
drop highlighted_page_image + friends, drop the base64 import.
- visualization /api/v1/chunk-context and auth/viz_routes: read
chunk_bbox instead of highlighted_page_image.
- vector/__init__: stop eagerly re-exporting `processor`/`scanner` —
fixes a pre-existing circular import (search.algorithms ->
vector.placeholder -> vector/__init__ -> processor -> scanner ->
server.semantic -> search.bm25_hybrid -> search.algorithms partial).
Test suite that was broken on master (test_bm25_hybrid.py et al.) now
collects and passes.
- scripts/purge_page_images.py: ad-hoc, idempotent migration that
delete_payload's the legacy keys from existing points. No reindex
required; legacy chunks render the page with no overlay.
Pairs with cbcoutinho/astrolabe#76. Frontend handles missing chunk_bbox
gracefully, so this can land in either order.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses reviewer feedback on PR #773:
- Backfill set_payload now uses wait=True to avoid a race where
_ensure_keyword_payload_indexes builds the KEYWORD index before
fire-and-forget writes have committed, leaving int payloads
invisible to filters.
- Batch points sharing the same int doc_id into a single set_payload
call (one document → many chunks → one round-trip instead of N).
- Drop _has_int_doc_id_sample short-circuit. The sample's false-negative
window (clean first 256 results, ints further in) is gone; full scroll
is the dominant cost on first run anyway.
- Simplify _ensure_keyword_payload_indexes: the "already exists" 400
branch was dead code (Qdrant returns 200 on identical re-create); any
400 now logs a warning and continues.
- search/context.py: comment the broadened file-type guard. Add explicit
not doc_id.isdigit() checks at the top of note/news_item/deck_card
branches in _fetch_document_text so malformed payloads surface as
warnings instead of being swallowed by the broad except.
Also extracts build_search_result_from_point into search/algorithms.py
to deduplicate the 71-line payload-extraction loop shared by
SemanticSearchAlgorithm and BM25HybridSearchAlgorithm. This fixes
SonarQube's quality-gate failure (4.0% new-code duplication, max 3%).
Test coverage:
- 7 new unit tests for build_search_result_from_point covering missing
payload, note/file/deck_card metadata, int doc_id coercion, and
metadata_extras merging.
- Replace _has_int_doc_id_sample tests with clean-collection no-op and
per-batch grouping tests.
- Update set_payload assertions from wait=False to wait=True.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add `doc_type` FieldCondition to the chunk_index-path highlighted-image
Qdrant filter in both `api/visualization.py` and `auth/viz_routes.py`,
matching the shape of `_get_chunk_by_index_from_qdrant`. Safe today (the
block is guarded by `doc_type == "file"` and Nextcloud file IDs are
globally unique) but prevents a latent bug if other doc types start
storing highlighted images.
- Demote `viz_routes.py` `ValueError` log from `error` to `warning` (lazy
%-style) — `_parse_int_param` raises on user-supplied bad input, which
is a 400 not a server error and shouldn't pollute error logs.
- Hoist `effective_chunk_index` to compute once at the top of
`get_chunk_with_context`, removing two duplicate assignments.
- Add `test_file_doc_type_qdrant_miss_yields_fast_404` to the management
endpoint tests, locking in the proxy-timeout fix contract.
- Add `tests/unit/test_viz_routes_chunk_context.py` mirroring management
coverage for the OAuth-session route: param forwarding (chunk_index /
total_chunks), `doc_type=file` fast 404, and 400 on invalid int params.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Production was logging two cascading classes of Qdrant errors against the
welcomed-malamute deployment:
1. HTTP 400 — "Bad request: Index required but not found for \"doc_id\" of
one of the following types: [keyword]". The collection was created via
create_collection() with no payload indexes, so any FieldCondition
filter on doc_id failed at the Qdrant layer (placeholder writes/reads,
eviction, search context lookups).
2. Compounding the missing index, producers wrote a mix of int and str
doc_ids: webhook_parser stringified node_id, scanner stringified note
IDs, news IDs, and deck card IDs — but the file scanner passed the
numeric file_id through unchanged. A keyword index would not have
covered both kinds even if it had existed.
This change:
- Normalizes doc_id to str at every producer site (scanner.py:459,
DocumentTask.doc_id, indexed_*_ids reads from Qdrant).
- Tightens str|int annotations to str across placeholder.py,
eviction.py, search/verification.py, search/context.py,
SearchResult.id, and the auth/api visualization endpoints.
- Defensive str() coercion on doc_id reads in semantic.py /
bm25_hybrid.py / vector/visualization.py for the transition window
before the backfill runs.
- Adds an idempotent startup migration in get_qdrant_client():
- _ensure_keyword_payload_indexes creates KEYWORD indexes for
doc_id, user_id, and doc_type (tolerates "already exists" 400s).
- _backfill_doc_id_to_string scrolls the collection once and rewrites
int doc_ids to str. Skipped after a quick sample shows no legacy
int payloads.
- Public API preserved: SemanticSearchResult.id stays int via explicit
int(r.id) narrowing in server/semantic.py — surfaces a TypeError with
actionable context if a future doc_type ships non-numeric ids.
- Documents the startup migration in docs/configuration.md.
Tests: 11 new unit tests in tests/unit/vector/test_qdrant_client.py
covering happy path / already-exists / unrelated-400 for the index
helpers, and sample-skip / mixed-batch rewrite / payload=None edge cases
for the backfill. 889 unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Replace bare int() casts for start/end/context_chars in chunk_context_endpoint
with _parse_int_param, matching visualization.py bounds (0–10M for offsets,
0–10K for context_chars), and add the missing end > start guard.
- Initialize page_number from chunk_context.page_number so non-file doc_types
surface it; include page_number, chunk_index, and total_chunks unconditionally
in the response. Only highlighted_page_image stays gated on its own truthiness.
- Add a chunk_index forwarding regression test that asserts the new kwargs reach
get_chunk_with_context and appear in the response payload.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #767 review noted that the OAuth viz route used bare int() parsing for
chunk_index and total_chunks while the bearer-token visualization route
validates them via _parse_int_param. Mirror the same bounds check so
total_chunks=0 and negative chunk_index return 400 instead of silently
suppressing adjacent-chunk context.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- test_registry.py: stub `mistralai.client.Mistral` in
`test_registry_mistral_wins_over_ollama`, mirroring the sibling
picker test, so the test doesn't depend on the SDK accepting
arbitrary keys.
- openai.py: convert remaining f-string `logger.info(...)` calls to
lazy `%s` formatting, aligning with the pattern in mistral.py and
the repo's logging convention.
- test_mistral.py: add four tests covering the defensive RuntimeError
guards in `embed()` and `_embed_batch_request()` — empty
response.data, single null embedding, batch null embedding, and
count-mismatch.
- docs/configuration.md: add `AWS_ACCESS_KEY_ID` and
`AWS_SECRET_ACCESS_KEY` rows to the env-var reference table; they
were already mentioned in prose but missing from the table.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- _retry.py: replace `assert last_error is not None` with explicit
`if last_error is None: raise RuntimeError(...)` so the original
rate-limit error is preserved under `python -O`.
- openai.py: drop the `_retry_factory` alias chain; rename the bound
decorator to `_retry_429` to match the pattern in mistral.py.
- mistral.py: comment the imports so future reviewers understand why
`from mistralai.client import …` is the canonical path on 2.x (no
top-level `__init__.py`; no `mistralai.models` subpackage either).
- docs/configuration.md: add `OPENAI_GENERATION_MODEL` and
`OLLAMA_GENERATION_MODEL` rows to the env-var reference table.
- test_mistral.py: add direct unit test for the `_is_rate_limit`
predicate (429 → True, 500 → False, missing-attr → False).
- test_registry.py: stub `mistralai.client.Mistral` in the registry
picker test, mirroring the Ollama sibling, so the test doesn't
depend on the SDK accepting arbitrary keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses the Claude Code review on PR #772 plus the SonarCloud S1192 finding:
- Extract `retry_on_rate_limit` into `nextcloud_mcp_server/providers/_retry.py`
as a parametric decorator. OpenAI and Mistral now share the same backoff
loop; future providers can reuse it without copy-paste.
- New `tests/unit/providers/test_retry.py` covers the decorator: 429 retry +
success, non-429 immediate re-raise, MAX_RETRIES exhaustion, default
predicate, and unrelated exception passthrough.
- Tighten Mistral SDK import to `from mistralai.client.errors import SDKError`
(the canonical sub-path; the reviewer's `from mistralai.models import
SDKError` does not exist in mistralai 2.4.5).
- Replace `MistralProvider.close()`'s direct `__aexit__` call with a no-op +
comment — the Speakeasy-generated client has no public close hook and the
underlying httpx client is closed by GC.
- Extract the duplicated "Embedding not supported" message to a module-level
constant (SonarCloud S1192).
- Align `Settings.get_embedding_model_name()` Bedrock check with the registry
by also considering `bedrock_generation_model`.
- Add the `mock_mistral_client` fixture to
`test_mistral_no_embeddings_disabled` for parity with the rest of the file.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a hosted Mistral embedding option (mistral-embed, 1024-dim) alongside
the existing Bedrock / OpenAI / Ollama / Simple providers. Implementation
mirrors OpenAIProvider: lazy dimension detection with a known-models lookup,
chunked batch requests, defensive index sort, and a 429-aware retry decorator.
In the same change, ProviderRegistry switches from os.getenv to the
dynaconf-backed Settings dataclass so all five providers share a single
configuration path. config.py gains the previously-uncovered Bedrock keys,
the new Mistral keys, the missing OPENAI_GENERATION_MODEL /
OLLAMA_GENERATION_MODEL, and SIMPLE_EMBEDDING_DIMENSION.
Auto-detection priority: Bedrock → OpenAI → Mistral → Ollama → Simple.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a search tool that finds contacts by free-text substring match
across the four fields users actually look at:
- full name (FN)
- nickname
- any email address
- any phone number (compared as digits-only so "+1 234-567-890"
matches a search for "2345678")
Without this tool, an MCP client that wants to "find John's email"
has to pull every contact via list_contacts and filter client-side,
which costs a CardDAV REPORT per addressbook even when the user only
has a single hit. Doing the filter server-side (still cheap — it
streams the full vcards but discards non-matches before serialising
the response) keeps the tool surface symmetric with the rest of the
contacts API: list, get, create, update, delete, *search*.
When ``addressbook`` is omitted the search spans every addressbook
the authenticated user can read.
License: AGPL-3.0, matching the project.
Two related bugs surfaced in production while viewing chunks from the
Astrolabe frontend on the AWS-hosted MCP server:
1. PyMuPDF document closed: in _fetch_document_text the fallback path
referenced pdf_doc.page_count after pdf_doc.close(), raising
"document closed" and returning None. The slow PDF re-parse already
completed but its result was discarded. Capture page_count into a
local before close().
2. Slow/fragile chunk lookup: get_chunk_with_context filtered Qdrant by
(chunk_start_offset, chunk_end_offset). Those fields are not part of
the always-indexed payload schema, and with strict_mode enabled they
yield 400 errors. Even with manually-added indexes the filter is
fragile if a doc is re-chunked. Switch to chunk_index (always
indexed) as the primary lookup key, falling back to offset-based
lookup when callers don't supply it.
Plumb chunk_index/total_chunks through both the management API
(api/visualization.py) and the OAuth viz route (auth/viz_routes.py).
Apply the same change to the highlighted-image lookup so all four
chunk-context Qdrant queries prefer the indexed field.
Skip the slow PDF re-parse fallback entirely for files: when both the
chunk_index and offset Qdrant lookups miss, re-downloading and
re-parsing the source PDF won't find the chunk either, and routinely
exceeds 30s on large documents - which is the proxy timeout in
Astrolabe. Notes/cards keep the document-fetch fallback (cheap).
Removes dead code (_get_file_path_from_qdrant) that was only used by
the now-unreachable file fallback path.
Companion change in the Astrolabe app passes chunk_index from search
results through to the new endpoint params.
---
_This PR was generated with the help of AI, and reviewed by a Human_
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The default property set in `search_files` omits `<oc:fileid>`, so
`find_by_type` returned descendant dicts with no `file_id` — which
`NextcloudClient.find_files_by_tag` then silently dropped via its
dedup-by-id guard. Net effect: tag-on-folder produced zero expanded
descendants in CI (single-user / nc31, nc32). Mirrors the explicit
property list already used in `WebDAVClient.find_by_tag`.
Also addresses three nits from the PR #765 bot review:
- trim multi-paragraph docstring on `_normalise_search_result`
- trim multi-line docstring on `find_files_by_tag`
- match `is not None` ID-extraction pattern in the descendant loop
- assert positional `mime_type` arg in the unit test
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
NextcloudClient.find_files_by_tag now mirrors the directory semantics
already used by the exclusion path (issue #710): when a tagged item is
a folder, walk its descendants via WebDAV SEARCH (Depth: infinity) and
include any files matching the MIME filter. Without this, tagging the
root of a corpus with `vector-index` indexed nothing because the tag
applies to the directory only, not to its children.
The vector scanner additionally consults EXCLUDED_TAGS now, so a folder
marked off-limits is skipped even if it (or an ancestor) carries the
include tag — defense-in-depth, matching the "exclusion wins" contract
already enforced by the MCP file tools.
Also addressed a recurring memory-style nit: pre-existing f-string log
lines in find_files_by_tag were converted to lazy %-style.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The two debug calls in get_tag_by_name were left as f-strings when the
method was migrated to _make_request in round 1. Convert to lazy
%-style formatting per repo convention (PR #764 review round 5).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Guard against malformed PROPFIND responses where tag["id"] is None
before calling get_files_by_tag (prevents
<oc:systemtag>None</oc:systemtag> dispatch).
- Add OCS-APIRequest: true header to get_tag_by_name and
get_files_by_tag to match every other PROPFIND/REPORT in the file —
fixes a latent reverse-proxy compatibility hazard.
- Add test_copy_resource_blocks_excluded_source to mirror the
existing move-source coverage; closes the asymmetric test gap.
- Add test_skips_tag_with_missing_id covering the new fail-open
branch in _resolve_one_tag.
- Reword _resolve_one_tag docstring: "distinct slot" was misleading
(tasks append rather than pre-allocate).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three important issues raised by the latest review on the
get_tag_by_name and get_files_by_tag methods:
1. Add explicit response.raise_for_status() after _make_request in both
methods. _make_request already raises HTTPStatusError on non-2xx so
the calls are redundant in practice, but keeping them visible at the
call site makes the contract self-documenting and prevents a future
refactor from silently feeding an error body into ET.fromstring.
2. Replace href_path.replace(webdav_prefix, "/") with a startswith +
slice. str.replace strips every occurrence of the prefix; while no
real Nextcloud path embeds the prefix mid-string, the fix removes
the theoretical exposure and matches the pattern used elsewhere in
the file.
3. Add Content-Type: text/xml to the systemtags PROPFIND headers.
Other PROPFIND-with-body calls in this file (list_directory line
240, list_attachments line 1041) include it; the systemtags PROPFIND
was the only outlier. Same header added to the systemtag REPORT for
symmetry.
No test changes — the existing get_files_by_tag mock test continues to
pass (the mock response yields valid XML so raise_for_status is a
no-op, and the user-relative path comparison is unaffected by the
prefix-strip swap on a non-adversarial path).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses two points from the latest PR #764 review:
1. The anyio.Lock in get_excluded_file_paths bought nothing under
anyio's cooperative multitasking model (single-threaded between
awaits, raw set mutations are already safe). _resolve_one_tag now
builds a local set of paths and appends it to a shared list — list
append between awaits is safe without a lock — and the caller
merges via set().union(*results) after the task group completes.
This removes the cognitive overhead the reviewer flagged without
changing the public API.
2. Adds tests/integration/test_tag_exclusion.py exercising the
resolution pipeline end-to-end against a real Nextcloud instance:
creates a system tag, tags a real file and a real directory,
verifies get_excluded_file_paths resolves both via real PROPFIND +
REPORT calls, and verifies is_path_excluded correctly classifies
exact matches, descendants of tagged directories, and unrelated
paths. Includes the disabled-feature short-circuit case.
Cleanup runs in reverse order (untag, delete files); per-run uuid
suffix avoids cross-run interference.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses the four points raised in the automated review on PR #764:
1. Scope guards on the four search tools (search_files, find_by_name,
find_by_type, list_favorites) so an excluded `scope` raises ToolError
instead of silently returning an empty result. Previously an LLM
could probe the asymmetry between list_directory (raises) and the
search tools (silent) to infer that an excluded directory exists.
The 4 search tools now mirror the early-guard pattern from
list_directory and avoid an unnecessary upstream query for known-
excluded scopes.
2. Concurrent per-tag resolution in get_excluded_file_paths via
anyio.create_task_group(). Previously the 2N network calls (1
PROPFIND + 1 REPORT per tag) ran serially. Per-tag fail-open
behaviour is preserved by extracting _resolve_one_tag, which
swallows its own exceptions so a single tag failure does not abort
the surrounding task group.
3. WebDAVClient.get_tag_by_name and get_files_by_tag now route through
_make_request, inheriting the @retry_on_429 decorator. Previously
they bypassed it; with tag exclusion invoked on every WebDAV tool
call, a transient 429 from the systemtags endpoint was hitting the
fail-open path instead of being transparently retried.
4. Test coverage: 6 new tests in test_webdav_tools_exclusion.py (4
scope-guard, 2 missing filter tests for find_by_type and
list_favorites) and 2 new tests in test_tag_exclusion.py (a
concurrency proof using an event-barrier that would deadlock under
sequential execution, and a fail-open-under-task-group test with
order-independent side_effect callables).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six findings raised in the PR review:
🔴 Blocking
- Fail-open on tag-resolution errors. get_excluded_file_paths now
wraps each tag's get_tag_by_name and get_files_by_tag call in
try/except; failures log a warning and the tag is skipped, rather
than propagating to the caller and disabling all WebDAV tools when
the systemtags endpoint is degraded. Documented in the docstring as
the intended fail-open behaviour (threat model is preventing
accidental exfiltration, not surviving server compromise).
🟡 Important
- nc_webdav_list_directory now raises ToolError when the listed path
itself is tagged, instead of silently returning an empty listing
after a wasted PROPFIND. Behaviour now mirrors the mutating tools.
- Destination error messages in move/copy/create_directory said "is
inside" but is_path_excluded matches exact paths too. Reworded to
"is or is inside".
🟢 Nits
- get_excluded_file_paths log message clarified: N counts
directly-tagged paths, not total descendants.
- Test isolation: tests/unit/conftest.py already has an autouse
_reload_dynaconf_after_test fixture that handles teardown. Removed
the redundant module-local fixture I had drafted; documented the
reliance in the module docstring instead.
- Added tests/unit/test_webdav_tools_exclusion.py: 12 server-layer
tests that register the WebDAV tools on a fresh FastMCP and invoke
each tool's underlying function with a mocked excluded set, asserting
ToolError is raised / results filtered as expected. Catches future
guard-integration regressions (e.g. wrong argument order).
Also added two unit tests for the new fail-open behaviour.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hide sensitive files/folders from the WebDAV MCP tool surface by
tagging them with a configured Nextcloud system tag. Defence-in-depth
control for users who connect LLMs to accounts holding contracts,
medical records, credentials, etc.
A new EXCLUDED_TAGS env var (comma-separated tag names, empty by
default) gates an exclusion layer that runs at the start of every
WebDAV tool call: tag names are resolved to tag IDs, those IDs are
expanded to the set of tagged paths, then listings/searches are
filtered and read/write/delete/move/copy operations on excluded paths
raise ToolError. Tagged folders exclude their descendants via prefix
match. Empty EXCLUDED_TAGS disables the feature entirely.
The threat model is preventing accidental data exfiltration via the
LLM tool surface — not hiding files from a determined operator. The
docs explicitly recommend creating exclusion tags with
user_assignable=false so the credentials the MCP server uses cannot
remove the tag.
Implementation:
- config.py: add `excluded_tags` to _DEFAULTS, Settings, and the
_field_map alongside other comma-separated env vars.
- client/webdav.py: get_files_by_tag now requests <d:resourcetype/>
and surfaces is_directory so tagged directories can recursively
exclude descendants.
- server/tag_exclusion.py (new): get_excluded_tag_names,
get_excluded_file_paths, is_path_excluded.
- server/webdav.py: exclusion guards in all 11 WebDAV tools;
read/write/create/delete/move/copy raise ToolError, list/search
tools silently filter excluded entries. Existing f-string log
calls converted to lazy %-style.
- tests: 17 new unit tests covering path-matching edge cases
(shared-prefix non-match, descendants of excluded dirs), tag-name
parsing, and get_excluded_file_paths with mocked WebDAV; 1 new
client test asserting <d:resourcetype/> -> is_directory parsing.
- docs/configuration.md: new "Tag-Based File Exclusion" section with
per-tool effect table, security guidance, and per-call cost note.
- README.md: feature mention under Key Features.
Closes#710.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
428 (Precondition Required, RFC 6585) is the correct semantic — the
request requires the client to complete a prerequisite step (Login Flow
v2 provisioning) before retrying. 412 (Precondition Failed) is for
header-based preconditions like ETags / If-Match.
No behavior change beyond the status code; same JSON payload.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The webhook API endpoints in api/webhooks.py forwarded the inbound MCP
OAuth bearer token directly to Nextcloud as the Authorization header.
Per ADR-022 / docs/login-flow-v2.md the data leg from MCP server to
Nextcloud must use HTTP Basic Auth with the user's stored Login Flow v2
app password — bearer-forwarding requires upstream user_oidc patches that
were never merged and is incompatible with admin endpoints gated by
@PasswordConfirmationRequired (e.g. webhook_listeners/api/v1/webhooks,
which 401s).
PR #760 papered over the symptom for /api/v1/apps by switching to the
permissive /cloud/capabilities endpoint, but the same architectural
mistake remained on list_webhooks / create_webhook / delete_webhook,
which still 500'd on the astrolabe admin UI's preset page.
Changes:
- New helper api/_auth.py:get_basic_auth_for_user(user_id) reads the
user's app password from encrypted storage and returns
(username, app_password). Mirrors context.py:_get_client_from_login_flow
but is callable from Starlette routes (no MCP Context required).
- All four endpoints in api/webhooks.py now use httpx.BasicAuth instead
of forwarding the OAuth bearer; ProvisioningRequiredError is mapped to
HTTP 412 so callers can render a "complete provisioning" CTA rather
than receiving an opaque 500.
- Outbound NC requests now identify the user by the username recorded at
Login Flow v2 provisioning time (which may differ from the IdP-issued
user_id) — flowed into WebhooksClient and used for logging.
Tests:
- tests/unit/test_management_apps_endpoint.py: assertions updated to
verify outbound NC request uses BasicAuth and carries no Authorization
header. Replaced "missing-Authorization → 500" test with a
ProvisioningRequiredError → 412 case.
- tests/unit/test_webhooks_api_auth.py (new): cross-endpoint coverage
for list_webhooks, create_webhook, delete_webhook and the new helper —
including 412 symmetry for all four endpoints.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The oidc app does a JS-driven re-authorize chain after login
(/apps/oidc/redirect → /apps/oidc/authorize → /apps/oidc/consent).
wait_for_load_state("networkidle") can fire during the brief gap before
the consent page renders, so a single _handle_oauth_consent_screen call
right after login often misses the consent div and the OAuth flow
deadlocks waiting for a callback that never arrives.
Move consent handling inside the callback-wait loop and poll for either
the consent page or the callback hit. Loop bound bumped to 60s to give
the JS-driven re-auth headroom.
Confirmed locally: integration test now passes against docker compose
--profile login-flow with the static OIDC client.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`tests/server/login_flow/test_management_api.py` had
`[pytest.mark.integration, pytest.mark.oauth]` while every other test in
`tests/server/login_flow/` uses `[pytest.mark.integration,
pytest.mark.login_flow]`. The single-user CI matrix filter is
`(integration and not keycloak and not login_flow and not
multi_user_basic)`, so the missing `login_flow` mark let this test
collect and run under single-user mode against `localhost:8004` (which
isn't up there, hitting the bug being reported), even though it's
specifically driving the login-flow MCP server.
Also `oauth` isn't a registered marker (see `[tool.pytest.ini_options]`
in pyproject.toml), so it was emitting an unregistered-marker warning.
Replacing the marker aligns this file with its siblings: single-user /
multi-user-basic / keycloak filters all deselect it now, and the
login-flow filter still picks it up.
Verified: `pytest --collect-only -m "<single-user filter>"` reports 2
deselected; `-m login_flow` collects both tests.
