Merge pull request #834 from cbcoutinho/fix/verify-on-read-tag-gate

fix(search): gate verify-on-read file results on vector-index tag membership

docs/ADR-019-verify-on-read-for-semantic-search.md

@@ -4,6 +4,22 @@
 **Date**: 2026-05-01
 **Depends On**: ADR-007 (Background Vector Sync), ADR-010 (Webhook-Based Vector Sync)
 
+> **Update (2026-06-02) — tag-aware file verification.** The `file` verifier
+> described below as a per-id WebDAV check (`PROPFIND`, later
+> `file_accessible_by_id`) now gates on current **`vector-index` tag
+> membership** instead. It issues a single
+> `find_files_by_tag(<VECTOR_SYNC_PDF_TAG>, mime_type_filter="application/pdf")`
+> REPORT per search (plus a one-shot `EXCLUDED_TAGS` lookup) and keeps only
+> files in that set — i.e. exactly what the scanner indexes. This is the
+> "fetch once and intersect" shape (like `news_item`), not per-id fan-out, and
+> it closes a gap the original design missed: a file *removed from the tag* (as
+> opposed to deleted/unshared) stayed accessible and so survived the old check,
+> lingering in results until the scanner's grace-period sweep. **Decision:** the
+> gate is strict for all file results, own and shared — a shared file survives
+> only if the owner's (userVisible) tag surfaces in the *searcher's* tag REPORT
+> (validated by `tests/integration/test_acl_shared_search.py`). See
+> `docs/configuration.md` → "Verify-on-Read Latency Budget" for the cost.
+
 ## Context
 
 The vector index in Qdrant is a *recall layer*, not the source of truth. Authoritative state for every indexed document — whether a note exists, whether a file is still shared with the user, whether a deck card is on a board the user can read — lives in Nextcloud, not in our index. Whenever those two views drift, semantic search returns **ghost records**: results that point to documents the user can no longer access (or that no longer exist at all).

docs/configuration.md

@@ -645,10 +645,12 @@ This adds Nextcloud round-trips to the search path that operators should be
 aware of:
 
 - **Per-search cost**: one Nextcloud round-trip per *unique* `(doc_id, doc_type)`
-  in the result set. Chunking means a 10-result page typically references 3-5
-  unique documents, so verification adds 3-5 round-trips. With the default
-  20-way concurrency this is one parallel batch — usually under 100 ms on a
-  healthy connection.
+  in the result set — except `file` and `news_item`, which each batch into a
+  single call per search regardless of how many results they contribute (see
+  the Files and News caveats below). Chunking means a 10-result page typically
+  references 3-5 unique documents, so verification adds 3-5 round-trips. With
+  the default 20-way concurrency this is one parallel batch — usually under
+  100 ms on a healthy connection.
 - **Concurrency**: all verifications fan out under a shared semaphore.
   Tunable via the `VERIFICATION_CONCURRENCY` env var (settings field
   `verification_concurrency`, default 20) — lower it if your Nextcloud
@@ -664,14 +666,45 @@ aware of:
   search that surfaces news results. Disabling News in the indexer or running
   with a smaller backlog mitigates this; per-item paginated verification is
   tracked as a future improvement.
-- **Eviction**: when verification finds a definitive miss (404 / 403), the
-  corresponding Qdrant points are deleted in the background on a lifespan-owned
-  task group — fire-and-forget, does **not** block the search response.
-  Eviction failures are logged but never propagated; the next query will
-  re-verify and re-attempt (self-healing).
+- **Files caveat**: `file` results are gated on current **`vector-index` tag
+  membership**, not bare access — the verifier issues a single
+  `find_files_by_tag(<tag>, mime_type_filter="application/pdf")` REPORT per
+  search that contains any file result (plus a one-shot `EXCLUDED_TAGS`
+  lookup), then keeps only files in that set. This matches exactly what the
+  scanner indexes, so a file removed from the tag (or deleted, or moved under
+  an excluded folder) drops out of results immediately rather than waiting for
+  the scanner sweep. The REPORT expands tagged folders via a `Depth: infinity`
+  SEARCH, so deployments that tag whole directory trees pay that walk once per
+  search; configure `VECTOR_SYNC_PDF_TAG` to change the tag name. The `file`
+  verifier's latency therefore scales with **both** the `Depth: infinity` folder
+  expansion **and** the `EXCLUDED_TAGS` lookup: that lookup fans out ~2 WebDAV
+  calls (1 PROPFIND + 1 REPORT) *per excluded tag*, concurrently, while holding
+  a single verification slot — so a deployment with a long `EXCLUDED_TAGS` list
+  and/or deeply tagged trees issues many parallel Nextcloud requests per search.
+  Operators in that situation may want to **lower `VERIFICATION_CONCURRENCY`** so
+  the file verifier's internal fan-out does not overwhelm the backend.
+- **Shared files**: a file an owner tagged and shared with the searcher only
+  survives verification if the owner's **`userVisible`** tag surfaces in the
+  *searcher's* tag REPORT. The MCP server's own tag-creation path
+  (`WebDAVClient.get_or_create_tag`) defaults to `user_visible=True`, so tags it
+  creates are fine. **Migration caveat**: if the `vector-index` tag was created
+  some other way — manually via `occ tag:add … --user-visible=false`, or in a
+  deployment predating this release — it may be `user_visible=False` (the
+  Nextcloud default for system-managed tags). In that case an owner's tag will
+  **not** surface in a recipient's systemtag REPORT, so every shared-file result
+  is *silently dropped* for recipients after upgrading — no error, just a
+  narrower result set. Verify the tag's visibility (Administration → *Collaborative
+  tags*, or `occ tag:list`) and, if it is not user-visible, recreate it as
+  user-visible so shared search keeps working.
+- **Eviction**: when verification finds a definitive miss (a 404 / 403, or — for
+  files — absence from the tag set), the corresponding Qdrant points are deleted
+  in the background on a lifespan-owned task group — fire-and-forget, does
+  **not** block the search response. Eviction failures are logged but never
+  propagated; the next query will re-verify and re-attempt (self-healing).
 - **Failure modes**: transient errors (5xx, network) keep results visible
   (fail open) so a flaky link does not silently shrink result pages; only
-  *definitive* 404 / 403 drops them.
+  *definitive* misses (404 / 403, or a file no longer in the tag set) drop them.
+  If the file tag REPORT itself errors, all file results are kept (fail open).
 
 If eviction ever needs to be disabled (debugging, benchmarking), the
 `evict_on_missing=False` keyword argument on `verify_search_results()` skips