fix(search): gate verify-on-read file results on vector-index tag membership
Verify-on-read only checked file *accessibility* (file_accessible_by_id), never tag membership, so a file removed from the `vector-index` tag (but still readable) kept surfacing in semantic search, and stale points only got evicted when they happened to rank in a search's top-K. Rework `_verify_files` to gate on current `vector-index` tag membership via a single batch `find_files_by_tag(tag, mime_type_filter="application/pdf")` REPORT per search (plus a one-shot EXCLUDED_TAGS lookup for exclusion-wins parity) — exactly what the scanner indexes. A file is kept iff it is in that set, so untagged / deleted / excluded files drop out immediately and the existing eviction wiring reclaims their Qdrant points. The gate is strict for all file results, own and shared. Mirrors the batch-fetch-and-intersect shape of `_verify_news_items` (one semaphore slot, fail-open on fetch error, malformed-id keep). - Promote the tag name to a `vector_sync_pdf_tag` Settings field (dynaconf env mapping VECTOR_SYNC_PDF_TAG) used by both scanner and verifier; drop the scanner's direct os.getenv. - Expose `find_files_by_tag` on NextcloudClientProtocol. - Rewrite the file-verifier unit tests (tagged/untagged/deleted/excluded/ fail-open/non-numeric); update the ACL + verify-on-read integration tests to seed tagged PDFs. - Amend ADR-019 and the configuration.md verify-on-read latency budget. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
7e4b83dc94
commit
d4dbf01b0a
@@ -4,6 +4,22 @@
|
||||
**Date**: 2026-05-01
|
||||
**Depends On**: ADR-007 (Background Vector Sync), ADR-010 (Webhook-Based Vector Sync)
|
||||
|
||||
> **Update (2026-06-02) — tag-aware file verification.** The `file` verifier
|
||||
> described below as a per-id WebDAV check (`PROPFIND`, later
|
||||
> `file_accessible_by_id`) now gates on current **`vector-index` tag
|
||||
> membership** instead. It issues a single
|
||||
> `find_files_by_tag(<VECTOR_SYNC_PDF_TAG>, mime_type_filter="application/pdf")`
|
||||
> REPORT per search (plus a one-shot `EXCLUDED_TAGS` lookup) and keeps only
|
||||
> files in that set — i.e. exactly what the scanner indexes. This is the
|
||||
> "fetch once and intersect" shape (like `news_item`), not per-id fan-out, and
|
||||
> it closes a gap the original design missed: a file *removed from the tag* (as
|
||||
> opposed to deleted/unshared) stayed accessible and so survived the old check,
|
||||
> lingering in results until the scanner's grace-period sweep. **Decision:** the
|
||||
> gate is strict for all file results, own and shared — a shared file survives
|
||||
> only if the owner's (userVisible) tag surfaces in the *searcher's* tag REPORT
|
||||
> (validated by `tests/integration/test_acl_shared_search.py`). See
|
||||
> `docs/configuration.md` → "Verify-on-Read Latency Budget" for the cost.
|
||||
|
||||
## Context
|
||||
|
||||
The vector index in Qdrant is a *recall layer*, not the source of truth. Authoritative state for every indexed document — whether a note exists, whether a file is still shared with the user, whether a deck card is on a board the user can read — lives in Nextcloud, not in our index. Whenever those two views drift, semantic search returns **ghost records**: results that point to documents the user can no longer access (or that no longer exist at all).
|
||||
|
||||
Reference in New Issue
Block a user