Round-4 claude-review findings (no blockers):
- 🟡 Untested server-layer metering hook (raised across rounds): extracted the
nc_semantic_search embeddings_queries recording into a module-level
record_search_usage() helper (mirroring record_indexing_usage) and added
tests/unit/server/test_semantic_metering.py — value = query token count,
flag-off no-op, None token → 0, doc_types metadata bounding, best-effort
failure swallowed.
- 🟡 Dedup-hit skipped metering invisibly: the existing dedup info log now
states "no embedding/usage recorded" so a "fewer embeddings_queries rows than
expected" audit lands on the dedup path directly.
Deferred 🟢 nits (stated on the PR): search 0-token rows are recorded
deliberately (the query embedding ran; zero is a sum no-op) — documented in the
helper; embed_tokens closure locality and the OpenAI embed() dual path are
unchanged (correct as-is / separate refactor).
Deck #67.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-1 claude-review findings:
- 🔴 Multi-doc_type search billed N embedding calls as 1. nc_semantic_search
loops search() once per doc_type on one BM25HybridSearchAlgorithm instance,
and each call re-embedded the query, so only the last query_token_count was
recorded. Cache the dense embedding per query on the (per-request) instance
so the query is embedded — and metered — exactly once regardless of how many
doc_types are searched. This also removes the redundant per-type embed work
and avoids billing a user N× for one logical query.
- 🟡 Ollama embed() now delegates to embed_with_usage() so single and batch
embeds use the same /api/embed endpoint (was the legacy /api/embeddings),
keeping _detect_dimension and other embed() callers consistent.
- 🟢 round() instead of truncating int() when coercing provider-reported token
counts (forward-compatible if a provider ever returns a float).
Tests: per-instance query-embedding cache (embedded once across 3 doc_types;
re-embeds on a different query).
Deferred (stated on the PR): mistral/openai single-embed dual path (changes
tested error/request semantics on the cloud-critical path — separate refactor),
bedrock boto3 sync-in-async (pre-existing; no new invoke_model calls per doc).
Deck #67.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
embeddings_queries now records the embedding request's token count (the unit
upstream providers bill on) instead of an operation count, and fires on the
indexing path too. Previously only semantic search recorded it (value=1), so a
re-indexing run produced no embeddings_queries events at all — only pages_chunks.
- Provider layer: additive embed_with_usage / embed_batch_with_usage surface the
per-request token count (Mistral/OpenAI usage.total_tokens, Bedrock Titan
inputTextTokenCount, Ollama prompt_eval_count); a char-based estimate is the
fallback (Simple, and any provider/response without a token field). Gateway and
EmbeddingService forward through. The count travels as a return value / a
per-request SearchAlgorithm attribute — never on the singleton — so concurrent
indexing + search can't mis-attribute bills.
- Indexing (vector/processor.py): records embeddings_queries (value=batch tokens)
alongside the existing pages_chunks event.
- Search (server/semantic.py): value is now the query embedding's token count,
relayed from BM25HybridSearchAlgorithm via query_token_count.
The astrolabe_embeddings_queries Stripe meter (sum aggregation) now sums tokens
with no CP/Terraform change. The meter "queries"->tokens naming/unit
clarification (homelab-terraform #254) + CP rollup/portal copy is a follow-up.
Deck #67.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Assert the open card stays visible under status="open" in the
deck_get_stack integration test (completes the partition check).
- Move the _append_archived_cards docstring closing quotes to their own line.
deck_get_stack's status="archived" + include_cards=False path is left as-is:
a single get_stack call is the cheapest way to obtain the stack metadata
there — routing it through the archived fast-path would fetch every archived
stack on the board just to strip the cards, which is heavier, not lighter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- deck_get_stack: fetch active + archived concurrently for status="all", and
for status="archived" source the stack from /stacks/archived in a single
call (skip the active fetch whose open cards are filtered out anyway),
matching deck_get_cards' pattern.
- Type the `client` param of _archived_cards_by_stack as NextcloudClient.
- Extend the stacks/overview integration test to assert status="archived"
(only the archived card) in addition to status="all".
- Document the third_party/astrolabe submodule mount policy in CLAUDE.md:
unmounted by default (CI installs the published app-store version); mount
only for tightly-coupled feature work needing CI integration, then revert.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The active Deck listing endpoints (StackService::findAll /
CardMapper::findAllForStacks and StackService::find / CardMapper::findAll)
filter out archived cards at the SQL level — only the /stacks/archived
endpoint returns them. The client-side status="all"/"archived" filters in
deck_get_cards, deck_get_stacks, deck_get_stack and deck_get_board_overview
therefore operated on a list the server had already stripped of archived
cards, so they could never surface one. deck_get_card (by ID) bypasses the
filter, which is why it appeared to work. Fixes#842.
When status is "all" or "archived", also fetch /stacks/archived
(client.deck.get_archived_stacks) and merge those cards back in per stack —
concurrently with the active fetch where applicable. status="open"/"done"
are unchanged and cost no extra call.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- semantic.py: normalize both None and [] doc_types to null in the
metadata so a future `metadata->'doc_types' IS NULL` query counts the
all-types case consistently.
- test: use a fixed past date in test_occurred_at_roundtrip instead of a
future literal (deterministic, no "why this date" confusion).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Non-blocking follow-ups from the merge-ready review:
- semantic.py: bound the doc_types copied into embeddings_queries metadata
to _USAGE_METADATA_MAX_DOC_TYPES (16). doc_types is caller-supplied with
no max_length on the tool signature; capping the stored copy keeps one
JSONB row from ballooning (not a billing/injection risk — CP ignores
metadata, binds are parameterized).
- migration: note that `metric` is intentionally unconstrained Text and
that adding a third metric requires keeping the CP-side catalog in sync,
else the rollup silently ignores the new rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- hooks: document why user_id in metadata is safe — it stays tenant-local
(the CP rollup aggregates GROUP BY (day, metric) into usage_daily, which
has no metadata column, so it never reaches Stripe) and is retained to
keep Deck #67's future per-user attribution derivable from the app DB.
- migration: instantiate the SQLite-side column types (sa.Text() etc.) for
visual parity with the instantiated Postgres types.
- tests: assert the WARNING contract in the unserializable-metadata test
too; add an autouse fixture that resets UsageEventStore._shared_instance
so a stray shared() call can't leak across tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- remove accidentally-committed .claude/scheduled_tasks.lock (Claude Code
runtime artifact swept in by `git add -A`) and gitignore it; the rest
of .claude/ stays tracked.
- store: cache UsageEventStore.shared() as a process-wide instance so the
hot search path doesn't allocate a fresh wrapper per metered query (the
wrapper is stateless beyond its storage handle).
- hooks: pass enabled=True directly (the outer guard already confirmed
the flag) instead of re-reading settings.usage_metering_enabled.
- migration: document the no-TTL retention design (control-plane rollup
owns the lifecycle; the data plane only appends).
- tests: assert the best-effort error path logs at WARNING (observability
contract).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- store: add optional `enabled` param to record_usage_event so hot-path
callers (nc_semantic_search) pass the already-resolved flag instead of
forcing a second uncached Settings build (ADR-024); falls back to
get_settings() when None so the store stays self-gating for standalone
use.
- hooks: thread enabled= through both call sites; bump the outer
shared()/construction failure log from debug → warning so "metering
enabled but no billing data" is visible at the default INFO level.
- migration: instantiate postgresql.JSONB() to match the sibling
TIMESTAMP(timezone=True) column.
- tests: fix the misleading "asyncpg returns JSONB as a JSON string"
comment; add occurred_at dialect round-trip test and an enabled-param
short-circuit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deck #67 data-plane slice: tenant Pods record billable operations
(embedding queries, pages/chunks embedded) into an app-DB usage_events
table that the control plane later pulls read-only into the billing
ledger and syncs to Stripe Meter Events.
- migration 007: usage_events table (Postgres TIMESTAMPTZ/JSONB/UUID
with portable SQLite fallbacks), indexed (occurred_at, metric) for the
CP rollup's per-day range scan + GROUP BY metric.
- UsageEventStore: best-effort, flag-gated writer reusing the shared
RefreshTokenStorage engine; ON CONFLICT (event_id) DO NOTHING for
idempotent retries; dialect-branched occurred_at bind. All work
(incl. metadata JSON encode) is swallowed so a metering failure never
surfaces to the user op.
- USAGE_METERING_ENABLED flag (default off) wired through Settings +
env map; off-path touches no storage, so OSS self-hosters get an empty
table and zero write overhead.
- two recording hooks: embeddings_queries (per nc_semantic_search, which
nc_semantic_search_answer reuses) and pages_chunks (after dense
embedding succeeds, covering both in-process and procrastinate paths).
- storage.acquire()/.dialect public seams so the sibling store doesn't
reach into the underscored internal.
- tests parametrized over SQLite + Postgres: flag-off no-op, roundtrip,
ON CONFLICT dedup, JSON/NULL metadata, and the best-effort swallow of
both DB errors and unserializable metadata.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tagging an existing file/folder emits only OCP\SystemTag\MapperEvent — never a
Node*Event — so tagged PDFs were previously only picked up by the hourly
scanner. Subscribe to the tag event and reconcile membership so adding/removing
the `vector-index` tag (re)indexes in near-real time.
- webhook_presets: add OCP\SystemTag\MapperEvent to the files_sync preset
(NC 32+, where MapperEvent gained getWebhookSerializable(); harmless on older
servers — it just never fires).
- webhook_parser: parse MapperEvent (objectType=files) into a path-less file
"reconcile" task. The payload carries only a fileid + tagIds (no name/path),
so assign and unassign both collapse to a reconcile.
- processor._reconcile_tag_event: resolve the fileid against the user's current
vector-index PDFs (find_files_by_tag). Present -> index with the resolved
path/etag; absent -> flip to delete. Naturally handles "an unrelated tag
changed" and a tagged folder's own fileid (no-op; the scanner still expands
folders to descendants).
- Unit tests for the parser branch and the reconcile.
The matching admin-UI preset change ships separately in the astrolabe app repo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend the documents-vs-chunks split to the remaining status surfaces so all
three report consistently (Deck #195):
- nc_get_vector_sync_status MCP tool + VectorSyncStatusResponse: add
indexed_documents (distinct) and indexed_chunks; keep indexed_count as a
deprecated alias of indexed_chunks. Reuses count_indexed.
- userinfo HTML page (/app/vector-sync/status): show Indexed Documents AND
Indexed Chunks rows; switch its count to count_indexed (which also excludes
placeholder points — the old raw count included them).
- /api/v1/vector-sync/status: restore indexed_count as a deprecated alias of
indexed_chunks so existing consumers (integration tests, pre-#115 UI) keep
working; the change is now purely additive for indexed_count.
Tests: VectorSyncStatusResponse documents/chunks/alias + zeroed defaults.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round 3 review follow-ups:
- Enforce the folder cap (MAX_PATH_PREFIXES=20) inside normalize_path_prefixes
so the REST/viz endpoints are bounded too, not just the MCP tool's Field
and the PHP client. Single server-side enforcement point; the MCP tool's
Field(max_length=...) now references the same constant.
- Widen the SearchAlgorithm ABC and both concrete implementations'
path_prefixes param to Iterable[str] | None, matching the widening of
build_base_filter_conditions from the prior round.
- Add a normalize_path_prefixes cap test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round 2 review follow-ups:
- Add Field(max_length=20) to the nc_semantic_search path_prefixes param so
an LLM client can't build an unbounded OR-filter (mirrors the cap the
Astrolabe PHP controller applies on the UI path).
- Note in normalize_path_prefixes that the two-pass collect-then-strip is
deliberate (the `if path_prefix:` guard is truthy for whitespace-only
input; the strip pass is what drops it).
- Tests: exercise build_base_filter_conditions with 3 folders (guards the
list comprehension) and parametrize the no-path case over None, empty
list, and blank-only inputs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend the ADR-027 Phase 2 path filter from a single path_prefix to a
list of folders. The new normalize_path_prefixes() helper is the single
source of truth for trimming, dropping blanks, and de-duplicating, and
folds the legacy single path_prefix into the list for backward
compatibility.
build_base_filter_conditions() adds one MatchText to the must clause for
a single folder (unchanged shape) and OR-s multiple folders via a nested
Filter(should=[...]) so a file under any selected folder matches while
still AND-ing against the ACL/doc_type/date conditions.
path_prefixes is threaded through every search surface: the
nc_semantic_search MCP tool, the visualization API (JSON body), and the
viz route (CSV query param). The Astrolabe frontend folder picker that
produces these lists ships in a companion astrolabe PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-architect document ingest from the shared NATS-glued document-processor to a
per-tenant, in-process model owned by nextcloud-mcp-server (Deck #183). The MCP
server now owns both sides of ingest:
- Producer (api role): the scanner defers one job per changed document into the
app's Postgres via procrastinate (queueing_lock dedup; no execution lock, so a
crashed worker can't deadlock a doc — Qdrant upserts are idempotent).
- Consumer (worker role): `nextcloud-mcp-server worker` drains the queue and runs
the existing process_document pipeline; a periodic task reclaims jobs orphaned
in `doing` by a crash.
INGEST_QUEUE selects the transport (auto: postgres when DATABASE_URL is Postgres,
else the in-process anyio queue for SQLite/dev). procrastinate manages its own
tables (applied on a fresh DB at startup and by `db upgrade`). The vector-sync
status surface reads job counts from Postgres in postgres mode. procrastinate +
psycopg3 ship in the [postgres] extra; the app's own engine still uses asyncpg
(driver unification is a follow-up handled in the rendered Helm chart).
NATS JetStream, the Postgres-queue stub, the bus status subscriber, and nats-py
are removed.
BREAKING CHANGE: the external-NATS-ingest env vars are removed
(INGEST_MODE, STATUS_BACKEND, INGEST_BUS_URL, INGEST_BUS_NUM_REPLICAS,
FACT_EVENT_EMITTER). Use INGEST_QUEUE (memory|postgres) and the `worker`
command instead. TENANT_ID is retained (no longer NATS-subject-charset-validated).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a path_prefix filter to semantic search, honoured on both the MCP tool and
the dense-only visualization/API paths through the shared filter contract.
- build_base_filter_conditions: append FieldCondition(file_path,
MatchText(path_prefix)) when set. file_path is only on doc_type == "file"
points, so a non-empty path_prefix implicitly restricts to files.
- Promote path_prefix to an explicit keyword param on the SearchAlgorithm ABC
and both algorithms; thread it through nc_semantic_search (blank ⇒ no filter),
the /api/v1 search endpoints, and the viz route.
- Add a file_path TEXT payload index to _PAYLOAD_INDEX_FIELDS (no content
re-index; idempotent startup migration). MatchText tokenizes on server Qdrant
and matches by substring on local/embedded qdrant-client — both serve folder
scoping.
- Update ADR-027 (Phase 2 implemented; readiness table; semantics note). Tests.
Refs ADR-027 Phase 2. Deck #177.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a modified_after/modified_before date-range filter to semantic search,
honoured on both the MCP tool path (BM25HybridSearchAlgorithm) and the
dense-only visualization/API path (SemanticSearchAlgorithm) through one shared
contract.
- Promote modified_after/modified_before to explicit keyword params on the
SearchAlgorithm ABC and both concrete algorithms; factor the shared
placeholder+ownership+doc_type+date filter into
access_filter.build_base_filter_conditions so new filters land in one place.
- nc_semantic_search: accept RFC 3339 / ISO 8601 (or Unix seconds) bounds via
utils.validation.parse_modified_timestamp; Annotated/Field constraints on the
numeric args; explicit McpError guard for after > before. Thread the parsed
bounds through the cross-app and per-doc_type dispatch.
- /api/v1 search endpoints + viz route parse the same formats and 400 on bad or
inverted ranges.
- Add a modified_at INTEGER payload index to _PAYLOAD_INDEX_FIELDS; the
idempotent _ensure_payload_indexes() startup path migrates existing
collections with no content re-index.
- Update ADR-027 to resolve the review feedback (validation placement, shared
algorithm contract, deferral of nc_semantic_search_answer, payload index,
RFC-3339-at-the-boundary rationale). Add unit tests.
Refs ADR-027. Deck #177.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Modernize the new models (DeckCardSummary, DeckCommentSummary,
StackOverview, BoardOverviewResponse + the loosened unions) to PEP 604
syntax (list[...] / X | None), per CLAUDE.md.
- Make status="done" exclude archived cards so open/done/archived partition
the board with no overlap (a done+archived card is reported only as
"archived"); document the semantics in docstrings and docs/deck.md, add a
partition unit test.
- deck_get_archived_stacks: pass through label/assigned_to filters (status
stays archived-only by definition); note the limitation in the docstring.
- Rename _validate_description_max_length → _validate_positive_length (now a
generic positive-length guard).
- Soften deck_get_board_overview docstring: it views board state and omits
the ACL/user/label-management fields deck_get_board exposes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deck read tools returned too many tokens to be usable as boards grow — even
deck_get_stacks(description_max_length=1) exceeded the MCP token limit because
every card was fully serialized in list views.
- Add compact projection models (DeckCardSummary, DeckCommentSummary,
StackOverview, BoardOverviewResponse) and a uniform detail="summary"|"full"
knob (summary default) on deck_get_cards / get_stacks / get_stack /
get_archived_stacks.
- Add pre-serialization filtering: status (open/done/archived/all), label,
assigned_to.
- Add deck_get_board_overview: board title + label legend + stacks with
compact card rows + counts in a single call.
- Compact comments: detail / message_max_length / newest-first order on
deck_get_card_comments.
- Docs + unit/integration tests.
BREAKING CHANGE: deck list tools now default to detail="summary" and
status="open". The include_archived_cards parameter is replaced by status
(use status="all" to include archived cards); pass detail="full" to restore
the previous per-card shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🟡 Important: nc_semantic_search's include_context branch did not forward
accessible_owners to get_chunk_with_context, so context expansion for shared
files stayed self-only, found nothing in Qdrant, and silently fell back to the
plain excerpt. Forward accessible_owners (the per-file file_accessible_by_id
gate still enforces access).
🟡 Performance: auth/viz_routes.py's multi-doc_type branch sorted but did not
cap the candidate pool before verify-on-read, so N doc_types × limit*2 went
into verification (N× the Nextcloud round-trips). Cap to limit*2 after the
sort, matching server/semantic.py and the cross-app branch.
Also clear the SonarCloud gate (new_duplicated_lines_density 5.1% > 3%) the
ACL wiring introduced: extract the duplicated /api/v1 client-resolution +
owner-expansion + verify-on-read block from unified_search/vector_search into a
shared _search_with_acl helper, define a constant for the repeated
"Nextcloud host not configured" literal (S1192), and reword the access_filter
move_to_end comment so it isn't misread as commented-out code (S125) while
adding the other-owner count to its debug log (review nits).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1. Don't log unverified result titles: both search algorithms logged top-5
titles at DEBUG before verify-on-read; with owner-level share expansion the
unverified set can contain other users' docs. Algorithms now log a count
only; the verifying callers (server/semantic, viz_routes, api/visualization)
log verified titles after verify-on-read.
2. Cross-user FILE chunk context: get_chunk_with_context + the Qdrant chunk
helpers now take accessible_owners and use build_ownership_filter. For files
the expanded scope is honoured only after a per-file file_accessible_by_id
check (accessible_owners is owner-level, so the gate prevents a one-file
share recipient from reading any of the owner's cached chunks). note/deck/
news stay self-only (per-user APIs) — a documented gap. Both chunk endpoints
pass accessible_owners.
3. Algorithm usage: SemanticSearchAlgorithm is not dead (it backs the dense-only
option on the viz/API surfaces); added a clarifying comment in server/
semantic.py. Additionally wired accessible_owners + verify-on-read into the
/api/v1 search routes (unified_search, vector_search) so the astrolabe
surface is ACL-aware too — degrading gracefully to self-only/unverified for
non-provisioned callers instead of 401.
4. Overlapping conditions: build_ownership_filter no longer lists self in the
owner_id MatchAny branch (self is already covered by the user_id branch);
the owner_id branch carries only the OTHER owners.
Tests: build_ownership_filter dedup + chunk-bbox filter-shape updates; new
ACL-aware get_indexed_doc_types, cached-chunk lookup, and end-to-end cross-user
file chunk-context (recipient gets the chunk, non-recipient denied) tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OAuth provisioning tools (check_provisioning_status, revoke_nextcloud_
access) only consulted the refresh-token store + Astrolabe status, ignoring the
app_passwords store that Login Flow v2 (nc_auth_provision_access) and the
management API write to — the same store require_provisioning / get_client use
to grant tool access. Result: status reported "not provisioned" while tools
worked, and revoke said "nothing to revoke" while the credential persisted.
- _get_provisioning_status: also check storage.get_app_password_with_scopes,
reporting is_provisioned with credential_type=app_password,
flow_type=login_flow_v2.
- _revoke_nextcloud_access: when the credential is an app password, delete it
from storage + invalidate the scope cache (no IdP token to revoke);
refresh-token revocation via the Token Broker is unchanged.
- tests/unit/test_oauth_tools_app_password_provisioning.py: cover status +
revoke for the app-password path.
- bump astrolabe submodule (deprovision MCP on disable); fix a stale assertion
in the migrated bg-sync test (one-click flow has no separate app-password
generation step).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two fixes surfaced while testing Login Flow v2 provisioning behind a split
internal/external host (Docker: server↔Nextcloud over http://app, browser
over http://localhost:8080):
1. login_url pointed at the internal host. Nextcloud builds the login URL
from the request host, so the browser-facing URL came back as
http://app/login/v2/flow/... — unreachable from the user's browser. The
poll endpoint was already rewritten to the internal host (correct, the
server polls it); now LoginFlowV2Client also rewrites the login_url origin
to settings.nextcloud_public_issuer_url when set (passed at all 5
construction sites). When unset, behaviour is unchanged.
2. The app-password format guard rejected raw session tokens. core/
getapppassword returns a long alphanumeric token, not the dashed 25-char
Security-settings format, so the dashed-only regex 400'd the one-click
opt-in handoff. Relax APP_PASSWORD_PATTERN to `^[a-zA-Z0-9-]{20,256}$`;
the authoritative validation is still the BasicAuth check against Nextcloud.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The vector index has always been strictly per-user: every Qdrant payload
carries a `user_id` and the search filter is `user_id == querying_user`.
A file Alice indexed cannot be discovered by Bob even if she has shared
it with him — Bob would have to re-index it under his own user_id to
make it searchable, which means duplicate index entries for every share
recipient.
Switch to ownership-with-ACL-expansion:
- New `nextcloud_mcp_server.search.access_filter` module:
- `list_accessible_owners(sharing_client, user_id)` calls the OCS
Sharing API (`shared_with_me=true`) and returns
`{user_id} ∪ {uid_owner of each share}`. Fails open to `[user_id]`
so a misbehaving Sharing API doesn't black-hole search.
- `build_ownership_filter(user_id, accessible_owners)` returns a
Qdrant `Filter` whose `should` branch matches either the new
`owner_id IN accessible_owners` field or the legacy `user_id` field.
The legacy branch keeps points indexed before this change reachable
without a migration backfill.
- Indexer payload (`vector/processor.py`) now writes `owner_id` alongside
`user_id`. `DocumentTask` gains an optional `owner_id` field; today the
scanner always runs as the owner so the processor falls back to
`user_id`, but the field is plumbed so a future shared-with-me crawler
can set the true owner without reshaping the payload contract.
- `SemanticSearchAlgorithm.search` and `BM25HybridSearchAlgorithm.search`
accept `accessible_owners` via kwargs and use the new ownership filter.
Default behaviour with no kwarg is unchanged (self-only).
- Both user-facing callers — the MCP tool path (`server/semantic.py`) and
the visualization Starlette route (`auth/viz_routes.py`) — compute
`accessible_owners` from the authenticated Nextcloud client before
invoking the search algorithm. Eviction, scanner deletion, placeholder,
and chunk-context paths intentionally keep the legacy `user_id`
semantics (those are "operations on a specific user's records", not
cross-user reads).
- 10 new unit tests in `tests/unit/search/test_access_filter.py` cover
self-only default, owner expansion, dedup, fallback fields, OCS
failure, and the legacy `should`-branch shape.
Pairs with cbcoutinho/astrolabe#89 — together they let an Astrolabe user
find content owners have shared with them without going through any
re-authorization flow or re-indexing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #719 fixed the contact-create path so all documented fields persist to the
vCard, but the read path (list/search via MCP) still returned ``organization:
null`` / ``note: null`` / ``title: null`` because pythonvCard4 has no typed
parser for ORG/TITLE — they land in ``Contact.custom`` — and the server-side
mapper never read ``note``/``urls``/``categories``/``photo`` even when present.
Reads now surface what the write side persisted:
- ``client/contacts.py``: new ``_first_custom`` helper pulls raw values from
``Contact.custom`` for ORG/TITLE/unencoded PHOTO. ``list_contacts``
extends its per-contact dict with org/title/note/url/categories/photo.
- ``server/contacts.py``: ``_raw_contact_to_model`` maps the new keys onto
``Contact.organization`` / ``.title`` / ``.note`` / ``.urls`` / ``.categories``
/ ``.photo``. URL accepts both list and plain-string shapes; categories
accepts comma-separated strings for forward-compat.
Coverage:
- Unit: ``TestFirstCustom`` (five cases incl. bare-string library shape) and
three new ``_raw_contact_to_model`` cases covering the full field set,
plain-string URL, and comma-string categories.
- Integration: ``test_mcp_contacts_workflow`` now decodes the
``nc_contacts_search_contacts`` response and asserts
``organization`` / ``note`` round-trip — direct regression coverage for
elvisdragonmao's report on issue #716.
Verified end-to-end against the local single-user docker stack: creating a
contact with ``{organization, title, note, url, categories}`` and reading it
back via ``nc_contacts_search_contacts`` returns every field populated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Nextcloud Deck PR #7910 added IWebhookCompatibleEvent to CardCreated/
Updated/DeletedEvent and BoardUpdatedEvent, so Deck can finally emit
real-time webhooks via core's webhook_listeners app. Wire this into
the existing preset → parser → DocumentTask pipeline that already
backs Notes / Calendar / Tables / Forms / Files sync.
- Add deck_sync preset (app=deck, 4 events) and drop the stale
"Deck does not support webhooks" comment.
- Teach webhook_parser to convert Deck card events into
DocumentTask(doc_type=deck_card, operation=index|delete) with
stack_id metadata. BoardUpdatedEvent logs delivery at INFO and
returns None — the polling scanner reconciles affected cards.
- Cover three new unit tests for the deck create/delete/board-update
paths plus symmetric fail-open tests for missing card.id /
node.id in _parse_deck_event and _parse_file_event.
The astrolabe admin UI auto-discovers the new preset via
filter_presets_by_installed_apps(); no astrolabe-side wiring is
required for it to appear in the Webhook Management card grid.
Note: requires Deck ≥1.18.x (where PR #7910 lands); the preset is
hidden when the Deck app isn't installed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The CalDAV REPORT in `_search_events_by_date` unconditionally requested
server-side `<C:expand>`. Per RFC 4791 §9.6.5 the server then normalizes
every expanded DTSTART/DTEND to UTC `Z`, which destroyed two pieces of
information on the read path:
- RFC 5545 floating local times came back as fake-UTC (a `+00:00` suffix
that did not match the stored value), so a 2:30 PM floating event was
indistinguishable from a 14:30 UTC event in the MCP response.
- TZID-bound events lost their IANA TZID context — a "10am America/New_York"
event came back as `14:00:00+00:00`, making it impossible for callers to
reconstruct DST-aware recurrence semantics.
Replace `<C:expand>` with client-side recurrence expansion via the
`recurring-ical-events` library (promoted from transitive to direct dep),
so the wire response retains its original DTSTART format. Surface the
TZID parameter as new `start_tz`/`end_tz` fields on `CalendarEventSummary`.
Add an optional `timezone` (IANA name) parameter to `nc_calendar_create_event`
and `nc_calendar_update_event` so callers can pin a TZID for naive input;
the helper attaches `ZoneInfo(...)` and emits a paired `VTIMEZONE`
component. Naive input without `timezone` continues to store as RFC 5545
floating local time (with a warning logged). Offset-aware input continues
to store as UTC `Z`.
Drive-by: switch the update path's DTSTART/DTEND assignment from raw
`datetime` to `vDDDTypes(dt)` wrappers — the previous code produced invalid
iCal like `DTSTART:2026-05-14 10:00:00+00:00` for any TZ-aware update.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #781 review round 1:
- 🔴 Fix notesPath key: the Notes API returns the folder under camelCase
``notesPath`` (see models/notes.py:43), but `deck_attach_note` was
looking up snake_case ``notes_path`` and silently falling back to
``"Notes"``. Users with a non-default notes folder would have produced
shares pointing at non-existent files (404 on click in Deck UI).
- 🔴 Add wire-through unit test that would have caught the above:
extract `_resolve_note_attach_path(client, note_id)` as a testable
helper that encapsulates the camelCase-key lookup. Three new tests:
custom notesPath honored, missing key falls back to default, null
category handled.
- 🟡 Modernize new fields on `DeckAttachmentExtendedData` to PEP 604
(`X | None`) per CLAUDE.md.
- 🟡 Drop unnecessary string forward reference on
`ListAttachmentsResponse.results` — DeckAttachment is defined earlier
in the same module.
- 🟢 Move `pytestmark = pytest.mark.unit` to module level in
test_sharing_client.py to match the convention in test_deck_server.py.
Per user request: `deck_attach_file` is now scoped `deck.write` +
``files.read`` (was just `deck.write`) so the generic file-share
permission story is consistent — only `deck_attach_note` keeps
`notes.read` since it specifically reads from the Notes app. Docstring
updated to emphasise the tool is generic over the user's Files
(PDFs/images/etc., not just markdown).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds four tools that expose Deck card attachments via the MCP surface:
deck_attach_file, deck_attach_note, deck_list_attachments, and
deck_delete_attachment. The attach* variants share an existing Files
entry (or Notes-app note) with the card via OCS shareType=12 — same
mechanism the Deck UI's "Share from Files" picker uses, no file copy.
This replaces the prior workaround of appending bulky activity content
as Deck card comments: per-PR/per-event narrative now lives in NC Notes
and surfaces on the tracking card as a clickable attachment that opens
the original note in place.
Implementation reuses existing client methods (SharingClient.create_share,
DeckClient.get/delete_attachment, NotesClient.get_settings/get_note);
no new client code. _SHARE_TYPE_DECK is centralised with a CI-guard test
to prevent silent drift, and SharingClient.create_share's wire format is
pinned to what the Deck Vue source sends.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Production was logging two cascading classes of Qdrant errors against the
welcomed-malamute deployment:
1. HTTP 400 — "Bad request: Index required but not found for \"doc_id\" of
one of the following types: [keyword]". The collection was created via
create_collection() with no payload indexes, so any FieldCondition
filter on doc_id failed at the Qdrant layer (placeholder writes/reads,
eviction, search context lookups).
2. Compounding the missing index, producers wrote a mix of int and str
doc_ids: webhook_parser stringified node_id, scanner stringified note
IDs, news IDs, and deck card IDs — but the file scanner passed the
numeric file_id through unchanged. A keyword index would not have
covered both kinds even if it had existed.
This change:
- Normalizes doc_id to str at every producer site (scanner.py:459,
DocumentTask.doc_id, indexed_*_ids reads from Qdrant).
- Tightens str|int annotations to str across placeholder.py,
eviction.py, search/verification.py, search/context.py,
SearchResult.id, and the auth/api visualization endpoints.
- Defensive str() coercion on doc_id reads in semantic.py /
bm25_hybrid.py / vector/visualization.py for the transition window
before the backfill runs.
- Adds an idempotent startup migration in get_qdrant_client():
- _ensure_keyword_payload_indexes creates KEYWORD indexes for
doc_id, user_id, and doc_type (tolerates "already exists" 400s).
- _backfill_doc_id_to_string scrolls the collection once and rewrites
int doc_ids to str. Skipped after a quick sample shows no legacy
int payloads.
- Public API preserved: SemanticSearchResult.id stays int via explicit
int(r.id) narrowing in server/semantic.py — surfaces a TypeError with
actionable context if a future doc_type ships non-numeric ids.
- Documents the startup migration in docs/configuration.md.
Tests: 11 new unit tests in tests/unit/vector/test_qdrant_client.py
covering happy path / already-exists / unrelated-400 for the index
helpers, and sample-skip / mixed-batch rewrite / payload=None edge cases
for the backfill. 889 unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a search tool that finds contacts by free-text substring match
across the four fields users actually look at:
- full name (FN)
- nickname
- any email address
- any phone number (compared as digits-only so "+1 234-567-890"
matches a search for "2345678")
Without this tool, an MCP client that wants to "find John's email"
has to pull every contact via list_contacts and filter client-side,
which costs a CardDAV REPORT per addressbook even when the user only
has a single hit. Doing the filter server-side (still cheap — it
streams the full vcards but discards non-matches before serialising
the response) keeps the tool surface symmetric with the rest of the
contacts API: list, get, create, update, delete, *search*.
When ``addressbook`` is omitted the search spans every addressbook
the authenticated user can read.
License: AGPL-3.0, matching the project.
- Guard against malformed PROPFIND responses where tag["id"] is None
before calling get_files_by_tag (prevents
<oc:systemtag>None</oc:systemtag> dispatch).
- Add OCS-APIRequest: true header to get_tag_by_name and
get_files_by_tag to match every other PROPFIND/REPORT in the file —
fixes a latent reverse-proxy compatibility hazard.
- Add test_copy_resource_blocks_excluded_source to mirror the
existing move-source coverage; closes the asymmetric test gap.
- Add test_skips_tag_with_missing_id covering the new fail-open
branch in _resolve_one_tag.
- Reword _resolve_one_tag docstring: "distinct slot" was misleading
(tasks append rather than pre-allocate).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses two points from the latest PR #764 review:
1. The anyio.Lock in get_excluded_file_paths bought nothing under
anyio's cooperative multitasking model (single-threaded between
awaits, raw set mutations are already safe). _resolve_one_tag now
builds a local set of paths and appends it to a shared list — list
append between awaits is safe without a lock — and the caller
merges via set().union(*results) after the task group completes.
This removes the cognitive overhead the reviewer flagged without
changing the public API.
2. Adds tests/integration/test_tag_exclusion.py exercising the
resolution pipeline end-to-end against a real Nextcloud instance:
creates a system tag, tags a real file and a real directory,
verifies get_excluded_file_paths resolves both via real PROPFIND +
REPORT calls, and verifies is_path_excluded correctly classifies
exact matches, descendants of tagged directories, and unrelated
paths. Includes the disabled-feature short-circuit case.
Cleanup runs in reverse order (untag, delete files); per-run uuid
suffix avoids cross-run interference.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses the four points raised in the automated review on PR #764:
1. Scope guards on the four search tools (search_files, find_by_name,
find_by_type, list_favorites) so an excluded `scope` raises ToolError
instead of silently returning an empty result. Previously an LLM
could probe the asymmetry between list_directory (raises) and the
search tools (silent) to infer that an excluded directory exists.
The 4 search tools now mirror the early-guard pattern from
list_directory and avoid an unnecessary upstream query for known-
excluded scopes.
2. Concurrent per-tag resolution in get_excluded_file_paths via
anyio.create_task_group(). Previously the 2N network calls (1
PROPFIND + 1 REPORT per tag) ran serially. Per-tag fail-open
behaviour is preserved by extracting _resolve_one_tag, which
swallows its own exceptions so a single tag failure does not abort
the surrounding task group.
3. WebDAVClient.get_tag_by_name and get_files_by_tag now route through
_make_request, inheriting the @retry_on_429 decorator. Previously
they bypassed it; with tag exclusion invoked on every WebDAV tool
call, a transient 429 from the systemtags endpoint was hitting the
fail-open path instead of being transparently retried.
4. Test coverage: 6 new tests in test_webdav_tools_exclusion.py (4
scope-guard, 2 missing filter tests for find_by_type and
list_favorites) and 2 new tests in test_tag_exclusion.py (a
concurrency proof using an event-barrier that would deadlock under
sequential execution, and a fail-open-under-task-group test with
order-independent side_effect callables).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six findings raised in the PR review:
🔴 Blocking
- Fail-open on tag-resolution errors. get_excluded_file_paths now
wraps each tag's get_tag_by_name and get_files_by_tag call in
try/except; failures log a warning and the tag is skipped, rather
than propagating to the caller and disabling all WebDAV tools when
the systemtags endpoint is degraded. Documented in the docstring as
the intended fail-open behaviour (threat model is preventing
accidental exfiltration, not surviving server compromise).
🟡 Important
- nc_webdav_list_directory now raises ToolError when the listed path
itself is tagged, instead of silently returning an empty listing
after a wasted PROPFIND. Behaviour now mirrors the mutating tools.
- Destination error messages in move/copy/create_directory said "is
inside" but is_path_excluded matches exact paths too. Reworded to
"is or is inside".
🟢 Nits
- get_excluded_file_paths log message clarified: N counts
directly-tagged paths, not total descendants.
- Test isolation: tests/unit/conftest.py already has an autouse
_reload_dynaconf_after_test fixture that handles teardown. Removed
the redundant module-local fixture I had drafted; documented the
reliance in the module docstring instead.
- Added tests/unit/test_webdav_tools_exclusion.py: 12 server-layer
tests that register the WebDAV tools on a fresh FastMCP and invoke
each tool's underlying function with a mocked excluded set, asserting
ToolError is raised / results filtered as expected. Catches future
guard-integration regressions (e.g. wrong argument order).
Also added two unit tests for the new fail-open behaviour.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hide sensitive files/folders from the WebDAV MCP tool surface by
tagging them with a configured Nextcloud system tag. Defence-in-depth
control for users who connect LLMs to accounts holding contracts,
medical records, credentials, etc.
A new EXCLUDED_TAGS env var (comma-separated tag names, empty by
default) gates an exclusion layer that runs at the start of every
WebDAV tool call: tag names are resolved to tag IDs, those IDs are
expanded to the set of tagged paths, then listings/searches are
filtered and read/write/delete/move/copy operations on excluded paths
raise ToolError. Tagged folders exclude their descendants via prefix
match. Empty EXCLUDED_TAGS disables the feature entirely.
The threat model is preventing accidental data exfiltration via the
LLM tool surface — not hiding files from a determined operator. The
docs explicitly recommend creating exclusion tags with
user_assignable=false so the credentials the MCP server uses cannot
remove the tag.
Implementation:
- config.py: add `excluded_tags` to _DEFAULTS, Settings, and the
_field_map alongside other comma-separated env vars.
- client/webdav.py: get_files_by_tag now requests <d:resourcetype/>
and surfaces is_directory so tagged directories can recursively
exclude descendants.
- server/tag_exclusion.py (new): get_excluded_tag_names,
get_excluded_file_paths, is_path_excluded.
- server/webdav.py: exclusion guards in all 11 WebDAV tools;
read/write/create/delete/move/copy raise ToolError, list/search
tools silently filter excluded entries. Existing f-string log
calls converted to lazy %-style.
- tests: 17 new unit tests covering path-matching edge cases
(shared-prefix non-match, descendants of excluded dirs), tag-name
parsing, and get_excluded_file_paths with mocked WebDAV; 1 new
client test asserting <d:resourcetype/> -> is_directory parsing.
- docs/configuration.md: new "Tag-Based File Exclusion" section with
per-tool effect table, security guidance, and per-call cost note.
- README.md: feature mention under Key Features.
Closes#710.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Coerce refresh_expires_in to int before arithmetic in both callback
paths so IdPs that serialize the field as a JSON string (e.g. AWS
Cognito) don't trigger an unhandled TypeError 500.
- Drop the orphaned oauth_session row written by _check_logged_in. The
canonical Flow 2 row is created by generate_oauth_url_for_flow2 keyed
by `state`, which is what the unified callback looks up; the
flow2_<hex> session_id was never matched and just churned the table
for 10 minutes per call.
- Match delete_cookie attributes (httponly, secure, samesite) to the
set_cookie call on logout so browsers reliably evict the cookie even
on implementations that consider security flags during deletion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Drop "(in-place)" from filter-helper docstrings; callers should
consume the return value, mutation is an implementation detail.
- Document that deck_get_archived_stacks always returns cards (an
archived stack without its cards has no audit value); point to
description_max_length for size control.
- Document that deck_get_cards applies filtering client-side, so it
is network-equivalent to deck_get_stack(include_cards=True).
- Pin the empty-list contract: a stack with all-archived cards and
include_archived_cards=False yields cards == [] (loaded but empty),
not cards is None (explicitly suppressed).
- Add explicit one-character-over-limit truncation test alongside the
existing exact-boundary test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Seven findings from the latest review on #758 (3 medium, 4 low/nit):
Medium:
- storage.py: replace 5 ``assert self.cipher is not None`` sites with
explicit ``RuntimeError`` so missing TOKEN_ENCRYPTION_KEY can't silently
become an AttributeError under ``python -O``
- session_backend.py: document the silent-invalidation invariant —
refresh-token TTL expiry without explicit logout deliberately makes
the browser session unusable; future readers must not relax it
- server/oauth_tools.py: drop user_id from the Flow 2 session_id
identifier — use ``flow2_{secrets.token_hex(16)}`` so audit logs and
DB rows don't carry user_id in the session_id field
Low / nit:
- token_utils.py: drop _fetch_locks dict entry in finally so a probed
deployment can't grow the lock dict without bound; coalescing test
now pins the invariant with len(_fetch_locks) == 0
- browser_oauth_routes.py: strip trailing slash from settings.nextcloud_host
before constructing the well-known URL so a host configured as
``https://cloud.example.com/`` doesn't produce a double-slash
- browser_oauth_routes.py: add comment explaining the three-layer CSRF
policy on the mcp_session cookie set (SameSite=Lax + POST-only logout
+ Origin/Referer check)
- oauth_routes.py: convert all 23 f-string log calls to lazy %-style
per the CLAUDE.md / memory feedback_lazy_logging convention
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
