Production was logging two cascading classes of Qdrant errors against the
welcomed-malamute deployment:
1. HTTP 400 — "Bad request: Index required but not found for \"doc_id\" of
one of the following types: [keyword]". The collection was created via
create_collection() with no payload indexes, so any FieldCondition
filter on doc_id failed at the Qdrant layer (placeholder writes/reads,
eviction, search context lookups).
2. Compounding the missing index, producers wrote a mix of int and str
doc_ids: webhook_parser stringified node_id, scanner stringified note
IDs, news IDs, and deck card IDs — but the file scanner passed the
numeric file_id through unchanged. A keyword index would not have
covered both kinds even if it had existed.
This change:
- Normalizes doc_id to str at every producer site (scanner.py:459,
DocumentTask.doc_id, indexed_*_ids reads from Qdrant).
- Tightens str|int annotations to str across placeholder.py,
eviction.py, search/verification.py, search/context.py,
SearchResult.id, and the auth/api visualization endpoints.
- Defensive str() coercion on doc_id reads in semantic.py /
bm25_hybrid.py / vector/visualization.py for the transition window
before the backfill runs.
- Adds an idempotent startup migration in get_qdrant_client():
- _ensure_keyword_payload_indexes creates KEYWORD indexes for
doc_id, user_id, and doc_type (tolerates "already exists" 400s).
- _backfill_doc_id_to_string scrolls the collection once and rewrites
int doc_ids to str. Skipped after a quick sample shows no legacy
int payloads.
- Public API preserved: SemanticSearchResult.id stays int via explicit
int(r.id) narrowing in server/semantic.py — surfaces a TypeError with
actionable context if a future doc_type ships non-numeric ids.
- Documents the startup migration in docs/configuration.md.
Tests: 11 new unit tests in tests/unit/vector/test_qdrant_client.py
covering happy path / already-exists / unrelated-400 for the index
helpers, and sample-skip / mixed-batch rewrite / payload=None edge cases
for the backfill. 889 unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Guard against malformed PROPFIND responses where tag["id"] is None
before calling get_files_by_tag (prevents
<oc:systemtag>None</oc:systemtag> dispatch).
- Add OCS-APIRequest: true header to get_tag_by_name and
get_files_by_tag to match every other PROPFIND/REPORT in the file —
fixes a latent reverse-proxy compatibility hazard.
- Add test_copy_resource_blocks_excluded_source to mirror the
existing move-source coverage; closes the asymmetric test gap.
- Add test_skips_tag_with_missing_id covering the new fail-open
branch in _resolve_one_tag.
- Reword _resolve_one_tag docstring: "distinct slot" was misleading
(tasks append rather than pre-allocate).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses two points from the latest PR #764 review:
1. The anyio.Lock in get_excluded_file_paths bought nothing under
anyio's cooperative multitasking model (single-threaded between
awaits, raw set mutations are already safe). _resolve_one_tag now
builds a local set of paths and appends it to a shared list — list
append between awaits is safe without a lock — and the caller
merges via set().union(*results) after the task group completes.
This removes the cognitive overhead the reviewer flagged without
changing the public API.
2. Adds tests/integration/test_tag_exclusion.py exercising the
resolution pipeline end-to-end against a real Nextcloud instance:
creates a system tag, tags a real file and a real directory,
verifies get_excluded_file_paths resolves both via real PROPFIND +
REPORT calls, and verifies is_path_excluded correctly classifies
exact matches, descendants of tagged directories, and unrelated
paths. Includes the disabled-feature short-circuit case.
Cleanup runs in reverse order (untag, delete files); per-run uuid
suffix avoids cross-run interference.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses the four points raised in the automated review on PR #764:
1. Scope guards on the four search tools (search_files, find_by_name,
find_by_type, list_favorites) so an excluded `scope` raises ToolError
instead of silently returning an empty result. Previously an LLM
could probe the asymmetry between list_directory (raises) and the
search tools (silent) to infer that an excluded directory exists.
The 4 search tools now mirror the early-guard pattern from
list_directory and avoid an unnecessary upstream query for known-
excluded scopes.
2. Concurrent per-tag resolution in get_excluded_file_paths via
anyio.create_task_group(). Previously the 2N network calls (1
PROPFIND + 1 REPORT per tag) ran serially. Per-tag fail-open
behaviour is preserved by extracting _resolve_one_tag, which
swallows its own exceptions so a single tag failure does not abort
the surrounding task group.
3. WebDAVClient.get_tag_by_name and get_files_by_tag now route through
_make_request, inheriting the @retry_on_429 decorator. Previously
they bypassed it; with tag exclusion invoked on every WebDAV tool
call, a transient 429 from the systemtags endpoint was hitting the
fail-open path instead of being transparently retried.
4. Test coverage: 6 new tests in test_webdav_tools_exclusion.py (4
scope-guard, 2 missing filter tests for find_by_type and
list_favorites) and 2 new tests in test_tag_exclusion.py (a
concurrency proof using an event-barrier that would deadlock under
sequential execution, and a fail-open-under-task-group test with
order-independent side_effect callables).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six findings raised in the PR review:
🔴 Blocking
- Fail-open on tag-resolution errors. get_excluded_file_paths now
wraps each tag's get_tag_by_name and get_files_by_tag call in
try/except; failures log a warning and the tag is skipped, rather
than propagating to the caller and disabling all WebDAV tools when
the systemtags endpoint is degraded. Documented in the docstring as
the intended fail-open behaviour (threat model is preventing
accidental exfiltration, not surviving server compromise).
🟡 Important
- nc_webdav_list_directory now raises ToolError when the listed path
itself is tagged, instead of silently returning an empty listing
after a wasted PROPFIND. Behaviour now mirrors the mutating tools.
- Destination error messages in move/copy/create_directory said "is
inside" but is_path_excluded matches exact paths too. Reworded to
"is or is inside".
🟢 Nits
- get_excluded_file_paths log message clarified: N counts
directly-tagged paths, not total descendants.
- Test isolation: tests/unit/conftest.py already has an autouse
_reload_dynaconf_after_test fixture that handles teardown. Removed
the redundant module-local fixture I had drafted; documented the
reliance in the module docstring instead.
- Added tests/unit/test_webdav_tools_exclusion.py: 12 server-layer
tests that register the WebDAV tools on a fresh FastMCP and invoke
each tool's underlying function with a mocked excluded set, asserting
ToolError is raised / results filtered as expected. Catches future
guard-integration regressions (e.g. wrong argument order).
Also added two unit tests for the new fail-open behaviour.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hide sensitive files/folders from the WebDAV MCP tool surface by
tagging them with a configured Nextcloud system tag. Defence-in-depth
control for users who connect LLMs to accounts holding contracts,
medical records, credentials, etc.
A new EXCLUDED_TAGS env var (comma-separated tag names, empty by
default) gates an exclusion layer that runs at the start of every
WebDAV tool call: tag names are resolved to tag IDs, those IDs are
expanded to the set of tagged paths, then listings/searches are
filtered and read/write/delete/move/copy operations on excluded paths
raise ToolError. Tagged folders exclude their descendants via prefix
match. Empty EXCLUDED_TAGS disables the feature entirely.
The threat model is preventing accidental data exfiltration via the
LLM tool surface — not hiding files from a determined operator. The
docs explicitly recommend creating exclusion tags with
user_assignable=false so the credentials the MCP server uses cannot
remove the tag.
Implementation:
- config.py: add `excluded_tags` to _DEFAULTS, Settings, and the
_field_map alongside other comma-separated env vars.
- client/webdav.py: get_files_by_tag now requests <d:resourcetype/>
and surfaces is_directory so tagged directories can recursively
exclude descendants.
- server/tag_exclusion.py (new): get_excluded_tag_names,
get_excluded_file_paths, is_path_excluded.
- server/webdav.py: exclusion guards in all 11 WebDAV tools;
read/write/create/delete/move/copy raise ToolError, list/search
tools silently filter excluded entries. Existing f-string log
calls converted to lazy %-style.
- tests: 17 new unit tests covering path-matching edge cases
(shared-prefix non-match, descendants of excluded dirs), tag-name
parsing, and get_excluded_file_paths with mocked WebDAV; 1 new
client test asserting <d:resourcetype/> -> is_directory parsing.
- docs/configuration.md: new "Tag-Based File Exclusion" section with
per-tool effect table, security guidance, and per-call cost note.
- README.md: feature mention under Key Features.
Closes#710.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Coerce refresh_expires_in to int before arithmetic in both callback
paths so IdPs that serialize the field as a JSON string (e.g. AWS
Cognito) don't trigger an unhandled TypeError 500.
- Drop the orphaned oauth_session row written by _check_logged_in. The
canonical Flow 2 row is created by generate_oauth_url_for_flow2 keyed
by `state`, which is what the unified callback looks up; the
flow2_<hex> session_id was never matched and just churned the table
for 10 minutes per call.
- Match delete_cookie attributes (httponly, secure, samesite) to the
set_cookie call on logout so browsers reliably evict the cookie even
on implementations that consider security flags during deletion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Drop "(in-place)" from filter-helper docstrings; callers should
consume the return value, mutation is an implementation detail.
- Document that deck_get_archived_stacks always returns cards (an
archived stack without its cards has no audit value); point to
description_max_length for size control.
- Document that deck_get_cards applies filtering client-side, so it
is network-equivalent to deck_get_stack(include_cards=True).
- Pin the empty-list contract: a stack with all-archived cards and
include_archived_cards=False yields cards == [] (loaded but empty),
not cards is None (explicitly suppressed).
- Add explicit one-character-over-limit truncation test alongside the
existing exact-boundary test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Seven findings from the latest review on #758 (3 medium, 4 low/nit):
Medium:
- storage.py: replace 5 ``assert self.cipher is not None`` sites with
explicit ``RuntimeError`` so missing TOKEN_ENCRYPTION_KEY can't silently
become an AttributeError under ``python -O``
- session_backend.py: document the silent-invalidation invariant —
refresh-token TTL expiry without explicit logout deliberately makes
the browser session unusable; future readers must not relax it
- server/oauth_tools.py: drop user_id from the Flow 2 session_id
identifier — use ``flow2_{secrets.token_hex(16)}`` so audit logs and
DB rows don't carry user_id in the session_id field
Low / nit:
- token_utils.py: drop _fetch_locks dict entry in finally so a probed
deployment can't grow the lock dict without bound; coalescing test
now pins the invariant with len(_fetch_locks) == 0
- browser_oauth_routes.py: strip trailing slash from settings.nextcloud_host
before constructing the well-known URL so a host configured as
``https://cloud.example.com/`` doesn't produce a double-slash
- browser_oauth_routes.py: add comment explaining the three-layer CSRF
policy on the mcp_session cookie set (SameSite=Lax + POST-only logout
+ Origin/Referer check)
- oauth_routes.py: convert all 23 f-string log calls to lazy %-style
per the CLAUDE.md / memory feedback_lazy_logging convention
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Move description_max_length validation to tool layer
(_validate_description_max_length), matching the existing
_validate_comment_message pattern; helper now trusts callers per
CLAUDE.md ("validate at system boundaries only").
- Fix mutation/return inconsistency: deck_get_stacks now uses a list
comprehension to capture _apply_stack_filters' return, matching
deck_get_stack / deck_get_archived_stacks.
- Rename include_archived -> include_archived_cards on deck_get_cards
and _apply_card_filters for consistency with deck_get_stacks.
- Route deck_get_archived_stacks through _apply_stack_filters so
future filters apply uniformly to active + archived paths.
- Trim _truncate_card_descriptions docstring to one line; add inline
comment in _apply_stack_filters explaining the breaking-change
default (mirrors Deck UI archived-card filtering).
- Replace fragile call_args[0][1] with call_args.args[1] in the
archived-stacks client test.
- Modernize Optional[X] -> X | None throughout deck.py (adjacent
cleanup called out in the review).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Validate description_max_length is positive (raises ValueError on 0
or negative); the prior code would have wiped descriptions to a
single ellipsis character on description_max_length=0.
- Extract filter logic into testable module-level helpers
(_apply_board_filters, _apply_stack_filters, _apply_card_filters)
and replace the dense `continue`-based loop in deck_get_stacks with
the reviewer's elif form.
- Document the truncation length quirk in the helper docstring (result
is description_max_length + 1 chars when truncation fires).
- Add 13 new unit tests covering the include/exclude flags on board,
stacks, and flat card lists, plus the new validation paths and an
explicit "description fits within limit, no ellipsis" case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add filtering options to deck read tools to keep responses compact on
boards with accumulated cards/comments, and expose archived stacks so
agents can audit completed work that has been archived off the active
board.
- deck_get_board: include_acl, include_users, include_labels
- deck_get_stacks/deck_get_stack: include_cards, include_archived_cards,
description_max_length
- deck_get_cards: include_archived, description_max_length
- New deck_get_archived_stacks tool wrapping the existing client method
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Flow 2 (oauth_authorize_nextcloud) now generates a nonce, stores it on
the oauth_session row, forwards it to the IdP, and verifies it via
expected_nonce in oauth_callback_nextcloud — closes the last replay-
protection gap (round-3 finding 1).
- _origin_matches_self fails closed when mcp_server_url is missing
instead of allowing the logout, and the diagnostic log is promoted
from warning to error so the misconfiguration is monitorable
(round-3 finding 2). New regression test pins the new behaviour.
- The five user_id-accepting helpers in oauth_tools.py (get_provisioning_status,
provision_nextcloud_access, revoke_nextcloud_access, check_provisioning_status,
check_logged_in) are renamed with leading underscores to make the
trust boundary structural rather than documentary
(round-3 finding 3).
- create_browser_session and delete_browser_session now emit audit_log
rows so session establishment / teardown match the pattern used by
the rest of the security-relevant storage operations
(round-3 nit 5). delete_browser_session selects user_id before delete
so the audit row is attributable.
- oauth_login_callback no longer reflects raw IdP-error text or
exception strings into the HTML failure page; users see a generic
"internal error occurred" message + a correlation ID, with the
detail logged server-side keyed by the same ID (round-3 nit 6).
The XSS regression test is updated to pin the stricter contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- oauth_login_callback's integrated-mode token-exchange branch now reuses
the shared discovery cache via get_oidc_discovery (round-2 finding 1).
- AS proxy flow now generates an OIDC nonce in oauth_authorize, stores it
on ASProxySession, forwards it to the IdP, and passes it as
expected_nonce to verify_id_token in _oauth_callback_as_proxy
(round-2 finding 2).
- Consolidate the two parallel discovery caches: oauth_routes' local
_discovery_cache and _get_cached_discovery are removed; all callers
now go through token_utils.get_oidc_discovery, which acquires the
follow_redirects=True knob it needs for Nextcloud installs without
pretty URLs (round-2 finding 3).
- Demote per-user INFO logs in oauth_tools.py (check_logged_in,
get_provisioning_status) to DEBUG; the elicitation auth URL is no
longer logged because it contains a sensitive state token
(round-2 finding 4).
Also pin nonce binding behaviour with a new unit test that asserts
_oauth_callback_as_proxy forwards session.nonce to verify_id_token, and
update test mocks to track the cache consolidation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Blocking:
- AS proxy callback now calls verify_id_token before caching the proxy
code so a tampered IdP response can't smuggle identity claims.
Important:
- Browser OAuth flow generates and verifies an OIDC nonce; new alembic
migration 006 adds the nonce column to oauth_sessions.
- _origin_matches_self logs a warning when CSRF check is bypassed.
- oauth_tools.py uses get_shared_storage instead of fresh handles.
Nits:
- New token_utils.get_oidc_discovery shares the 5-minute cache with
verify_id_token; oauth_login (integrated) and _revoke_refresh_token_at_idp
now use it instead of issuing fresh discovery fetches.
- Drop typing.Optional from oauth_tools.py in favour of X | None.
CI:
- test.yml generates an ephemeral Fernet TOKEN_ENCRYPTION_KEY per run
with openssl, removing the dependency on a missing repo secret.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-launch hardening for the hosted Astrolabe Cloud offering. Addresses
all five findings raised in #626 (Tim Kaufmann, code review of v0.65.0).
Re-verified against master before fixing.
Finding 3 (LLM-controllable user_id) — drop user_id from the public
signatures of provision_nextcloud_access, revoke_nextcloud_access,
check_provisioning_status, check_logged_in. Tool wrappers now always
derive identity from the verified AccessToken; user_id is no longer
accepted as MCP input. Adds parameterized CI-guard test that locks the
schema.
Finding 2 (predictable session cookie) — replace mcp_session=<user_id>
cookie with a cryptographically random session_id mapped server-side
(new browser_sessions table, alembic 005). Cookie value is opaque,
expires, revocable. SessionAuthBackend looks up user_id via the new
mapping and additionally requires a refresh token to fail closed.
Finding 4 (logout doesn't revoke refresh token) — oauth_logout now
calls the IdP revocation_endpoint (RFC 7009) when advertised, deletes
the stored refresh token regardless, and clears the browser_sessions
row. Cleanup is best-effort: logout always 302s.
Finding 1 (unverified ID token decodes) — verify_id_token helper does
JWKS signature + issuer + audience + exp + nonce checks per OIDC core
3.1.3.7. Used by both OAuth callback handlers (browser + MCP). Removes
the four "verify_signature: False" decodes that previously trusted IdP
claims unconditionally. Drops dead-code _validate_token_audience in
token_broker. Refactors token_utils + provisioning_decorator to read
user_id from the verified AccessToken instead of re-decoding the JWT.
Finding 5 (hardcoded Fernet keys in docker-compose.yml) — replace the
three inline TOKEN_ENCRYPTION_KEY values with required env var
interpolation; document in env.sample.
Test coverage: 4 new unit test modules (signature pinning, browser
sessions, ID-token verification, logout + revoke + session backend).
693 unit tests pass; ruff/format/ty clean.
Migration note: existing browser admin-UI sessions become invalid on
rollout (cookies are looked up against the new browser_sessions table,
which starts empty). Users re-login. MCP API access is unaffected.
Tracked on Astrolabe Cloud POC board card #37.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six review items raised; four required code changes (#3, #4, #5, #6) and
two were resolved without code changes (#1 audit-only, #2 informational).
* search/verification.py — clarify the granularity asymmetry between the
whole-batch fail-open (structural API failure) and the per-item fail-open
(single bad stored doc_id). Future readers no longer need to derive why
the two paths have different blast radii from the code alone.
* models/semantic.py — `dropped_document_count` description now explicitly
notes that subtracting it from `verified_chunk_count` is not a meaningful
operation, since the two fields count different units (documents vs
chunks). Surfaces the unit mismatch where MCP clients actually see it.
* server/semantic.py — clarify the per-doc_type over-fetch comment so the
N×2 pre-merge Qdrant cost (vs the cross-app branch's 1×2) is explicit
rather than implied by "same 2× over-fetch budget".
* tests/unit/search/test_verification.py — add four new 429 unit tests
(notes/news/files/deck) mirroring the existing 5xx-keeps pattern. Locks
in that `_is_definitive_404_or_403` returns False for 429 so a future
refactor cannot accidentally treat rate-limit responses as permanent
revocations.
Audit confirmation for review item #1: all four `WebDAVClient.get_file_info`
call sites already handle the new `HTTPStatusError`-on-404 contract
(verification.py:156, tests/integration/test_rag.py:139,
tests/unit/client/test_webdav.py:153/190). No silent breakage internal to
this repo.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three minor fixes from the round-11 review on PR #750:
- bm25_hybrid.py:209 — Comment said `doc_id` is `int (notes) or str (files)`,
which is backwards. Notes, news_items, and deck_cards are stored as `str`
(scanner.py:241, 666, 867); files are stored as `int` (scanner.py:425).
Updated to point readers at scanner.py as the source of truth.
- verification.py:338 — Lowered the News-API 403/404 log line from `info`
to `debug`. The News app being uninstalled or disabled is a predictable
operational state (matching the other verifiers' debug-on-not-found
paths), so this should not generate operator-dashboard noise. Transient
errors immediately below stay at `warning` because they're unexpected.
- semantic.py:809 — `nc_get_vector_sync_status` was reading
`document_receive_stream` via `getattr(..., None)`, but the attribute is
guaranteed-defined on both `AppContext` and `OAuthAppContext` (as a
field with `None` default). The defensive `getattr` masked typos that
the eviction_task_group access at semantic.py:197-199 deliberately
surfaces. Switched to direct access; the `if … is None:` value-check
below is preserved (the attribute can legitimately be None before sync
starts).
Items deliberately deferred (with rationale in the plan file):
- News verifier semaphore-hold during get_items (reviewer: "not required
here, just worth tracking"; ADR already lists follow-ups).
- Hardcoded 2× over-fetch / VERIFICATION_OVERFETCH (TODO already in code).
- Integration test for the real Qdrant eviction filter (reviewer marked
low-priority; type-preservation chain is unit-tested).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Tighten verify_search_results signature: client: Any → NextcloudClientProtocol
- Collapse 3 copy-pasted lock-justification comments to a single-line pointer
- Add logger.debug timing around the verify_search_results call site
- Add logger.debug timing around the unbounded news.get_items fetch
- Rename SemanticSearchResponse.dropped_count → dropped_document_count to make
the chunks-vs-documents unit asymmetry explicit at the API boundary
- Drop unreachable duplicate 409 branch in WebDAVClient.move_resource
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add concurrency-safety comments to per-verifier accessible sets in
_verify_notes/_verify_files/_verify_deck_cards. Same rationale as
accessible_by_type in verify_search_results: anyio is cooperative,
set.add() is not an await point.
- Document 401 exclusion in _is_definitive_404_or_403 (treated as
transient because it usually signals expired credentials, not
permanent denial).
- Note multi-user compounding in the news verifier semaphore comment:
N concurrent users hold N slots out of the shared budget.
- Log inaccessible doc ids with a type tag (e.g. "int:42" vs "str:42")
so ghost-record logs disambiguate id types.
- Type the BatchVerifier alias and the four verifier function signatures
with NextcloudClientProtocol instead of Any (algorithms.py exposes
the right interface; the protocol is runtime_checkable).
- Surface verified_chunk_count vs dropped_count semantics in the
nc_semantic_search tool docstring Returns block (chunks vs unique
documents).
- Add comments to the two max_concurrent=20 sites in server/semantic.py
noting they are intentionally distinct from
settings.verification_concurrency (different request phases).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Rename `verified_count` → `verified_chunk_count` to make the count
granularity explicit at the field name (chunks vs unique docs).
- News verifier now fails open *per-item* on non-numeric stored doc_ids
(matches notes/files/deck shape); a single bad id no longer rescues
definitively-missing siblings from eviction.
- Update note-verifier integration test to use string doc_ids end-to-end
to match production storage (scanner.py:241 stringifies note ids).
- Add regression test for the closed-task-group race guard in
`verify_search_results` so the RuntimeError swallow is locked in.
- Convert remaining f-string logger calls in `server/semantic.py` to
lazy %-style formatting (per repo convention).
- Document `evict_on_missing` as a developer/test flag (no env var) and
flag the `get_file_info` 404→raise contract change in its docstring.
- Add a TODO(ADR-019) breadcrumb for the hardcoded 2× over-fetch so
future tuning has a clear hook.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes out the remaining nits flagged in the round-6 review.
Critical:
- _verify_files contract comment now enumerates all None-return cases
(404 + malformed PROPFIND XML) and documents the false-eviction
trade-off; self-healing via re-indexing recovers
- int(r.id) cast at the SemanticSearchResult boundary now raises a
TypeError with explicit doc_type/value context instead of bubbling
up as an opaque "Search failed: ..." McpError
Design observations:
- nc_semantic_search_answer docstring documents the per-note
round-trip cost from the post-verification race guard
- News verification latency hint added to configuration.md
- SemanticSearchResponse exposes verified_count + dropped_count so
short result pages on high-ghost-density indexes are
distinguishable from genuine scarcity. verify_search_results now
returns (kept, dropped_count); production caller and tests updated
Minor:
- Comment clarifies the .get() fallback in verify_search_results is
defensive only (run_verifier always populates the entry)
- Eviction task-group guard narrowed from except Exception to
except RuntimeError (the only documented failure mode of
TaskGroup.start_soon on a closed group)
- Indexer logs a warning when a deck_card task is missing
board_id/stack_id, surfacing data-quality issues at index time
rather than at verification time
- New unit test covers the news verifier's non-numeric-id fail-open
path (one bad doc_id keeps the entire batch)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- _verify_deck_cards: hoist int(board_id|stack_id|doc_id) out of the generic
except Exception into an explicit try/except (TypeError, ValueError) before
the network call, mirroring _verify_news_items. Malformed payloads now log
a specific warning instead of "unexpected error".
- _verify_news_items: add TODO(perf) above the get_items(batch_size=-1) call
to mark the known fetch-all cost as a future profiling target.
- SemanticSearchResult.id: revert from int|str back to int. The internal
SearchResult.id stays int|str for forward-compat; the MCP response model
narrows at the boundary. server/semantic.py casts r.id to int when
constructing the response so future string-id types fail loudly here
instead of silently widening the public API.
- nc_semantic_search: replace the terse "extra for access filtering" comment
with an ADR-019 NOTE block explaining the 2x over-fetch trade-off and the
ghost-density under-delivery case (self-heals via lazy eviction).
- tests/integration/test_verify_on_read.py: extend the module docstring to
call out that only the note verifier is exercised against real Nextcloud,
while file/deck_card/news_item are unit-only — documenting the suite split
for future contributors.
- ADR-019: rewrite "Module shape", "Verifier registry", example verifier,
and "Deduplication" sections to match the shipped BatchVerifier interface
(was per-id Verifier in the original draft). Add a "Why batch?" paragraph
explaining the design choice. Update implementation checklist — every
item is now [x] with corrected verifier names (plural) and the eviction
module path (vector/eviction.py).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements fire-and-forget eviction (ADR-019 §"Lazy eviction"): the
search response no longer waits on Qdrant deletes, instead spawning
evict() on a long-lived lifespan-owned task group. Falls back to inline
eviction in modes without vector sync and in unit tests.
Also: harden _verify_news_items against non-numeric ids (fail open
instead of crashing the verifier); document the get_file_info None-on-404
contract; add INDEXED_DOC_TYPES single source of truth in vector/scanner.py
referenced by the CI-guard test; write a Verify-on-Read Latency Budget
section in docs/configuration.md covering the unbounded news.get_items
fetch. Closes the two remaining ADR-019 implementation checklist items.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Cap all_results to limit*2 after sort in the per-doc_types branch of
nc_semantic_search to bound over-verification (was unbounded N-types).
- Switch BatchVerifier from (client, doc_ids, user_id) to (client, results,
semaphore). Verifiers now read file paths and deck board/stack ids from
SearchResult.metadata instead of doing fresh Qdrant scrolls — eliminates
one duplicate round-trip per file/deck-card verification.
- Bound per-id verification concurrency with a shared anyio.Semaphore
(default 20, matching server/semantic.py context-expansion convention).
Prevents httpx pool exhaustion / rate limiting on large search pages.
- Propagate stack_id from Qdrant payload to SearchResult.metadata in both
bm25_hybrid.py and semantic.py (board_id was already propagated).
- Drop now-unused _resolve_file_path / _resolve_deck_metadata helpers.
- Drop redundant int(d) in requested predicate from _verify_news_items.
- Rewrite eviction comment to be honest about inline (not background)
execution and the resulting latency coupling.
- ADR-019 status: Proposed -> Accepted.
- Add news property to NextcloudClientProtocol.
- Widen SearchResult.id and SemanticSearchResult.id to int | str to match
BatchVerifier signature and document support for future string-id types.
- Flip openWorldHint to True on nc_semantic_search_answer (it calls into
Nextcloud via nc_semantic_search).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The vector index lags Nextcloud (5-min webhook cron + scanner interval),
producing ghost records for deleted/unshared documents until the next
reconciliation. Verify each unique document against Nextcloud at query
time, drop inaccessible results, and lazily evict the corresponding
Qdrant points.
Per-doc_type batch verifiers: notes/files/deck cards run concurrently
per id; news items use a single fetch + intersect to avoid the per-item
fetch-all amplification. Transient errors fail open (keep result, log
warning) — only definitive 4xx drops. Multiple chunks of the same doc
collapse to one verification call.
Wired into nc_semantic_search before the limit trim and before context
expansion. nc_semantic_search_answer's per-note re-fetch retained as a
sub-second race guard since verification now happens upstream.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the seven outstanding items from the @claude review on PR #741:
1. Add empty `tests/client/talk/__init__.py` for pytest discovery parity
with `tests/client/{collectives,news}/`.
2. Standardise boolean query params to integers — `includeStatus` was the
string `"true"` in `list_conversations`/`list_participants` while every
other flag (`noStatusUpdate`, `lookIntoFuture`, `setReadMarker`,
`includeLastKnown`) used `1`/`0`.
3. Replace the `app:install || app:enable` chain in the spreed install hook
with `app:install --keep-disabled --force || true; app:enable spreed`,
so unrelated install failures surface as a clear "app not found" from
`app:enable` rather than being silently masked.
4. Add `_validate_token()` (alphanumeric whitelist) and call it from all
six TalkClient methods that interpolate the token into a URL path —
defence-in-depth against pathological tokens reaching httpx.
5. Rename `TalkConversation.type` to `room_type` with `Field(alias="type")`
and `populate_by_name=True`, so the field no longer shadows Python's
builtin while preserving spreed's wire format on input. MCP responses
now serialize `room_type` (field name) instead of `type`.
6. `mark_as_read` now passes `json=body or None` so the bodyless
"mark everything as read" call doesn't send a spurious `{}` body and
`Content-Type: application/json` header.
7. `_validate_message_text` rejects whitespace-only messages, not just
empty strings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment-only follow-up to surface non-obvious behavior at the call
sites flagged in review:
- server/talk.py: note the `uuid.uuid4().hex` 32-char no-dashes format
(spreed accepts either form).
- models/talk.py: warn that spreed returns `lastReadMessage: 0` rather
than `null` for unread rooms, so consumers should compare to ``None``
rather than rely on truthiness.
- 10-install-spreed-app.sh: document that the `app:install || app:enable`
fallback also masks unrelated install failures, and limit its use to
dev fixtures.
No runtime behavior changes; tests unchanged (still 13 unit + 7 integ).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four targeted fixes from the AI code review:
1. TalkConversation.description: drop the misleading `str | None`
union (spreed always sends `""`, never null) — type is now `str`
with default `""`.
2. get_messages: guard the X-Chat-Last-Given int parse with
try/except so a misbehaving proxy can't crash the read flow;
logs a warning and falls back to None.
3. get_messages: clamp `limit` to [1, 200] in the client (spreed
caps server-side at 200 and silently truncates) so the returned
`count` always matches what was actually requested. Both client
and server-tool docstrings updated to state the valid range.
4. Add an integration test covering the 32000-char message ceiling
in talk_send_message — the empty-message case was already tested,
the over-length case was not.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds 6 MCP tools so an LLM can read a user's Talk conversations and
post messages on their behalf, addressing the "read my chats and reply"
use case from issue #720:
- talk_list_conversations
- talk_get_conversation
- talk_get_messages
- talk_list_participants
- talk_send_message (auto-attaches a referenceId for retry dedup)
- talk_mark_as_read
Edit/delete messages, reactions, threads, and call/session ops are
intentionally out of scope for this first PR.
The TalkClient also exposes create_conversation/delete_conversation
for the integration test fixture; these are not registered as MCP
tools. A post-installation hook enables spreed in the docker dev env
so the integration suite has a real Talk backend to talk to.
Closes#720
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Wrap raw DeckComment returns in CardCommentResponse(BaseResponse) for
create/update so the success/timestamp envelope matches other deck tools
(#737 review issue 2).
- Rename ListCardCommentsResponse.total → count and clarify in the
description that it's the page size, not a server-side total — the Deck
list endpoint does not expose one (#737 review issue 3).
- Validate the documented 1000-character limit on create/update with an
inline length check + ValueError, matching the pattern in
api/management.py (#737 review issue 4).
- Use modern int | None union syntax for the new parent_id parameter
(#737 review issue 1); rest of the file is left in the existing
Optional[...] style.
Also add an MCP-level test that the >1000 char message is rejected, and
update the existing comment tests to unwrap the new comment field.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Expose four new MCP tools backed by existing DeckClient comment methods:
- deck_get_card_comments — list with limit/offset pagination
- deck_create_card_comment — top-level or threaded (via parent_id)
- deck_update_card_comment — author-only on the server
- deck_delete_card_comment — author-only, destructive, idempotent
Adds ListCardCommentsResponse and CardCommentOperationResponse models, and
extends the client unit tests to cover replies, deletion, pagination, and
the request shape for updates.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
AWS Cognito provides refresh tokens automatically with the authorization
code flow but does not list offline_access as a supported scope. Check
the IdP's scopes_supported discovery field before including it in
requests, and always accept refresh tokens from responses regardless.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove dead monkeypatch in test_stdio_calls_get_stdio_mcp
- Add _reload_config() teardown to single_user_env fixture
- Tighten AVAILABLE_APPS type to Callable[[FastMCP], None]
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Revert default transport to streamable-http (not a breaking change)
- Extract AVAILABLE_APPS constant to server/__init__.py (DRY)
- Wrap get_stdio_mcp ValueError in click.ClickException for clean errors
- Fix test_stdio.py: call _reload_config() so dynaconf sees env changes
- Use lazy %-style logging in stdio.py
- Add private API comments in test assertions
- Derive --enable-app CLI choices from AVAILABLE_APPS
- README: show explicit --transport stdio in uvx examples
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Many identity providers (AWS Cognito, Okta, Azure AD) reject or mishandle
colons in OAuth scope names. This migrates all custom scopes from
`resource:action` to `resource.action` format (e.g., `notes:read` →
`notes.read`), which is universally accepted and aligns with industry
conventions (Microsoft, Google).
Includes Alembic migration 004 for stored scope strings and ADR-024
documenting the rationale and RFC references.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
pythonvCard4 parses vCard BDAY fields into datetime.date objects, but
the Contact model expects Optional[str]. This caused a validation error
that crashed the entire contact list. Convert at the client layer
(consistent with the calendar client pattern) with a defensive check
at the server mapping layer.
Closes#672
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix emoji clearing bug: use _UNSET sentinel in update_collective so
emoji=None sends {"emoji": null} instead of raising ValueError
- Move collectives_get_trashed_collectives to Read Tools section
- Remove redundant is_trash field from ListTrashedPagesResponse
- Add page lifecycle note to collectives_trash_page docstring
- Add unit test for clearing collective emoji via update_collective
- Add integration test for clearing collective emoji via MCP tool
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix inconsistent error code in set_collective_emoji (400 → -32603)
- Allow clearing emoji via set_collective_emoji(emoji=None)
- Remove destructiveHint from trash operations (soft deletes are recoverable)
- Change delete_collective to idempotentHint=False (requires trash precondition)
- Add restore_collective and get_trashed_collectives tools
- Add unit tests for ValueError guard, clear-emoji path, and new tools
- Add integration test for full trash/restore/delete lifecycle
- Verify move_page returns new title in response message
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Rename collectives_update_collective to collectives_set_collective_emoji
(more precise since only emoji is settable)
- Use standard JSON-RPC error code -32603 (INTERNAL_ERROR) instead of -1
- Handle UnicodeDecodeError when reading page content via WebDAV
- Replace brittle 'Welcome' content assertion with length check
Fixes CI: test_update_operations_not_idempotent no longer matches the
renamed tool, which is correctly idempotent (no ETag involved).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix assign_tag sending Content-Type header with no body
- Mark collectives_update_collective as idempotent (no ETag involved)
- Raise OCSError when 'data' key missing instead of silent fallback
- Tighten color validator to 3 or 6 hex chars only
- Add comment explaining null emoji semantics in set_page_emoji
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Validate OCS envelope in trash_collective, delete_collective, trash_page
- Guard _unwrap_ocs against non-OCS responses with informative OCSError
- Remove _get_ocs_headers() indirection, use class constants directly
- Split headers: _OCS_HEADERS (GET) vs _OCS_HEADERS_JSON (with body)
- Fix docstring claiming emoji param is required when it is optional
- Rename misleading test, add test for non-OCS envelope handling
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add collectives_trash_collective and collectives_delete_collective MCP
tools with proper destructiveHint annotations. Refactor integration test
fixture to use MCP tools for cleanup instead of direct httpx/OCS calls.
Optimize _get_ocs_headers() to class-level constant.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bugs:
- assign_tag/remove_tag now call _unwrap_ocs to surface OCS-level errors
- trash_page changed to idempotentHint=False (trashing twice errors)
- WebDAV path parts stripped of slashes to prevent double-slash paths
Robustness:
- _unwrap_ocs uses ocs.get("data", {}) instead of ocs["data"]
- Unit test added for missing data key in OCS envelope
Minor:
- MCP error codes use -1 (project convention) instead of HTTP status codes
- update_collective docstring notes that emoji is required
- CollectiveTag.color validated as hex format via field_validator
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bug fixes:
- Catch OCSError/HTTPStatusError in all server tools, convert to McpError
- Guard update_collective against empty body (raise ValueError)
- Use restore_page response data in status message
ADR-017 annotation fix:
- Distinguish "remove" (reversible association) from "delete" (permanent):
remove_tag and deck_remove_label_from_card no longer set destructiveHint
- Update annotation test to exclude "remove" from destructive keywords
Data model improvements:
- Add trashTimestamp field to PageInfo
- Create ListTrashedPagesResponse with is_trash context flag
- Add collective_id to ListTagsResponse
Test robustness:
- Read NC credentials from environment variables (not hardcoded)
- Filter landing page by parentId == 0 instead of assuming pages[0]
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add destructiveHint=True to collectives_remove_tag (matches "remove"
keyword pattern in annotation tests)
- Change collectives_update_collective to idempotentHint=False (update
operations are non-idempotent per project convention)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Validate OCS envelope status before unwrapping data (raise OCSError on
statuscode >= 400)
- Fix test data: filePath should be "" for root-level pages, not filename
- Catch specific exceptions (HTTPStatusError, OSError) instead of bare
Exception in WebDAV content fetch, include error in log message
- Return updated resource data from update_collective, move_page, and
set_page_emoji instead of discarding API responses
- Fix create_page docstring to mention collectivePath/filePath/fileName
- Remove unused additional_headers parameter from _get_ocs_headers
- Add unit test for OCS error status validation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
