fix(search): gate verify-on-read file results on vector-index tag membership

Verify-on-read only checked file *accessibility* (file_accessible_by_id),
never tag membership, so a file removed from the `vector-index` tag (but
still readable) kept surfacing in semantic search, and stale points only
got evicted when they happened to rank in a search's top-K.

Rework `_verify_files` to gate on current `vector-index` tag membership via
a single batch `find_files_by_tag(tag, mime_type_filter="application/pdf")`
REPORT per search (plus a one-shot EXCLUDED_TAGS lookup for exclusion-wins
parity) — exactly what the scanner indexes. A file is kept iff it is in that
set, so untagged / deleted / excluded files drop out immediately and the
existing eviction wiring reclaims their Qdrant points. The gate is strict
for all file results, own and shared. Mirrors the batch-fetch-and-intersect
shape of `_verify_news_items` (one semaphore slot, fail-open on fetch error,
malformed-id keep).

- Promote the tag name to a `vector_sync_pdf_tag` Settings field (dynaconf
  env mapping VECTOR_SYNC_PDF_TAG) used by both scanner and verifier;
  drop the scanner's direct os.getenv.
- Expose `find_files_by_tag` on NextcloudClientProtocol.
- Rewrite the file-verifier unit tests (tagged/untagged/deleted/excluded/
  fail-open/non-numeric); update the ACL + verify-on-read integration tests
  to seed tagged PDFs.
- Amend ADR-019 and the configuration.md verify-on-read latency budget.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-02 21:14:44 +02:00
co-authored by Claude Opus 4.8
parent 7e4b83dc94
commit d4dbf01b0a
9 changed files with 358 additions and 220 deletions
@@ -4,6 +4,22 @@
**Date**: 2026-05-01
**Depends On**: ADR-007 (Background Vector Sync), ADR-010 (Webhook-Based Vector Sync)
> **Update (2026-06-02) — tag-aware file verification.** The `file` verifier
> described below as a per-id WebDAV check (`PROPFIND`, later
> `file_accessible_by_id`) now gates on current **`vector-index` tag
> membership** instead. It issues a single
> `find_files_by_tag(<VECTOR_SYNC_PDF_TAG>, mime_type_filter="application/pdf")`
> REPORT per search (plus a one-shot `EXCLUDED_TAGS` lookup) and keeps only
> files in that set — i.e. exactly what the scanner indexes. This is the
> "fetch once and intersect" shape (like `news_item`), not per-id fan-out, and
> it closes a gap the original design missed: a file *removed from the tag* (as
> opposed to deleted/unshared) stayed accessible and so survived the old check,
> lingering in results until the scanner's grace-period sweep. **Decision:** the
> gate is strict for all file results, own and shared — a shared file survives
> only if the owner's (userVisible) tag surfaces in the *searcher's* tag REPORT
> (validated by `tests/integration/test_acl_shared_search.py`). See
> `docs/configuration.md` → "Verify-on-Read Latency Budget" for the cost.
## Context
The vector index in Qdrant is a *recall layer*, not the source of truth. Authoritative state for every indexed document — whether a note exists, whether a file is still shared with the user, whether a deck card is on a board the user can read — lives in Nextcloud, not in our index. Whenever those two views drift, semantic search returns **ghost records**: results that point to documents the user can no longer access (or that no longer exist at all).