fix(search): gate verify-on-read file results on vector-index tag membership

Verify-on-read only checked file *accessibility* (file_accessible_by_id),
never tag membership, so a file removed from the `vector-index` tag (but
still readable) kept surfacing in semantic search, and stale points only
got evicted when they happened to rank in a search's top-K.

Rework `_verify_files` to gate on current `vector-index` tag membership via
a single batch `find_files_by_tag(tag, mime_type_filter="application/pdf")`
REPORT per search (plus a one-shot EXCLUDED_TAGS lookup for exclusion-wins
parity) — exactly what the scanner indexes. A file is kept iff it is in that
set, so untagged / deleted / excluded files drop out immediately and the
existing eviction wiring reclaims their Qdrant points. The gate is strict
for all file results, own and shared. Mirrors the batch-fetch-and-intersect
shape of `_verify_news_items` (one semaphore slot, fail-open on fetch error,
malformed-id keep).

- Promote the tag name to a `vector_sync_pdf_tag` Settings field (dynaconf
  env mapping VECTOR_SYNC_PDF_TAG) used by both scanner and verifier;
  drop the scanner's direct os.getenv.
- Expose `find_files_by_tag` on NextcloudClientProtocol.
- Rewrite the file-verifier unit tests (tagged/untagged/deleted/excluded/
  fail-open/non-numeric); update the ACL + verify-on-read integration tests
  to seed tagged PDFs.
- Amend ADR-019 and the configuration.md verify-on-read latency budget.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-02 21:14:44 +02:00
co-authored by Claude Opus 4.8
parent 7e4b83dc94
commit d4dbf01b0a
9 changed files with 358 additions and 220 deletions
+9
View File
@@ -93,6 +93,10 @@ _DEFAULTS: dict[str, Any] = {
# leave work stuck behind the 5x-scan-interval staleness gate.
# Escape hatch only — leave on by default.
"vector_sync_orphan_sweep_enabled": True,
# System tag that marks files for vector indexing. The scanner indexes
# files carrying this tag; verify-on-read gates results on current
# membership of this tag (ADR-019).
"vector_sync_pdf_tag": "vector-index",
# Verify-on-read concurrency cap (ADR-019)
"verification_concurrency": 20,
# Qdrant
@@ -625,6 +629,10 @@ class Settings:
vector_sync_queue_max_size: int = 10000
vector_sync_user_poll_interval: int = 60 # seconds - OAuth mode user discovery
vector_sync_orphan_sweep_enabled: bool = True # card #101
# System tag marking files for vector indexing. The scanner indexes files
# carrying this tag and verify-on-read gates results on current membership
# (ADR-019), so an untagged file drops out of search immediately.
vector_sync_pdf_tag: str = "vector-index"
# Verify-on-read concurrency (ADR-019). Cap on parallel Nextcloud
# round-trips during search-result verification fan-out. Lower this if the
@@ -1217,6 +1225,7 @@ def get_settings() -> Settings:
"vector_sync_queue_max_size": "VECTOR_SYNC_QUEUE_MAX_SIZE",
"vector_sync_user_poll_interval": "VECTOR_SYNC_USER_POLL_INTERVAL",
"vector_sync_orphan_sweep_enabled": "VECTOR_SYNC_ORPHAN_SWEEP_ENABLED",
"vector_sync_pdf_tag": "VECTOR_SYNC_PDF_TAG",
# Verify-on-read (ADR-019)
"verification_concurrency": "VERIFICATION_CONCURRENCY",
# Qdrant settings