feat: dedup shared-file parsing/embedding across users in vector sync

A file shared across many users — directly, or via a group folder shared
to a group — was parsed and embedded once per user. Chunk point IDs are
user-agnostic (uuid5(tenant_id, doc_id=fileid, chunk_index)), but the
per-user freshness gate filtered Qdrant by user_id, so two readers
ping-ponged: each overwrote the other's points and each kept seeing "not
indexed for me", reprocessing every scan. Production telemetry (note
386945, finding #5) measured identical docs re-processed every few hours
at 7-13s each, with PDF parse ~62% of per-doc cost.

Layer 1 — tenant-wide dedup:
- Thread the scanner's tag-REPORT etag into the file DocumentTask and the
  chunk payload; index `etag` as a KEYWORD field.
- vector/sharing_state.find_indexed_content scrolls tenant-wide (no
  user_id filter) for a non-placeholder point matching
  (doc_id, doc_type, etag), gated on embedding_identity in Python so a
  model switch correctly forces a re-embed.
- Scanner skips enqueue and the processor skips fetch/parse/embed when a
  match exists (cross-worker race-guard before WebDAV read). Dedup is
  fail-safe: a Qdrant error degrades to "process normally".

Layer 2 — observed-access ACL (no admin / GroupFolders API needed):
- Each point carries `acl_principals` = the set of user:<uid> whose
  scanner has observed (hence can read) the file. The per-user tag REPORT
  is the access oracle; group membership/GroupFolders enumeration is
  admin-only and unavailable in multi-user modes.
- build_ownership_filter ORs MatchAny(acl_principals, ["user:<me>"]) so a
  deduplicated shared/group-folder point surfaces to every reader;
  verify-on-read (_verify_files) remains the precise ACL gate.
- Deletion/eviction become "release one user": drop the principal and
  delete the points only when the set empties, so one user untagging a
  shared file doesn't evict it for the others. Legacy points without the
  field keep the original per-user delete.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-04 12:55:17 +02:00
co-authored by Claude Opus 4.8
parent 0919513f21
commit 1c93e7286d
9 changed files with 671 additions and 68 deletions
+15 -4
View File
@@ -162,10 +162,14 @@ def build_ownership_filter(
"""Build the Qdrant ``Filter`` constraining a search to readable points.
Matches points whose ``owner_id`` is in ``accessible_owners`` (excluding
self) OR whose ``user_id`` equals ``user_id``. The ``user_id`` branch covers
*all* of the caller's own content — both new points (where
``owner_id == user_id``) and legacy points indexed before ``owner_id``
existed — so self is intentionally NOT repeated in the ``owner_id`` branch.
self) OR whose ``user_id`` equals ``user_id`` OR whose ``acl_principals``
contains ``user:<user_id>``. The ``user_id`` branch covers *all* of the
caller's own content — both new points (where ``owner_id == user_id``) and
legacy points indexed before ``owner_id`` existed — so self is intentionally
NOT repeated in the ``owner_id`` branch. The ``acl_principals`` branch covers
files that were indexed once and deduplicated across users (user-agnostic
point IDs): such a point's ``user_id``/``owner_id`` are the first indexer's,
so only the observed-access principal set surfaces it to other readers.
Args:
user_id: Querying user (matched by the ``user_id`` branch, which is the
@@ -188,6 +192,13 @@ def build_ownership_filter(
other_owners = [owner for owner in owners if owner != user_id]
conditions: list[Condition] = [
FieldCondition(key="user_id", match=MatchValue(value=user_id)),
# Observed-access branch: a file deduplicated across users carries one
# user-agnostic point set whose ``user_id``/``owner_id`` are the first
# indexer's. ``acl_principals`` lists every user whose scanner has seen
# (hence can read) the file, so this branch surfaces a shared/group-folder
# file to every reader even when they were not the indexer. Verify-on-read
# (_verify_files) is the precise ACL gate on the returned candidates.
FieldCondition(key="acl_principals", match=MatchAny(any=[f"user:{user_id}"])),
]
if other_owners:
conditions.insert(