feat: dedup shared-file parsing/embedding across users in vector sync

A file shared across many users — directly, or via a group folder shared
to a group — was parsed and embedded once per user. Chunk point IDs are
user-agnostic (uuid5(tenant_id, doc_id=fileid, chunk_index)), but the
per-user freshness gate filtered Qdrant by user_id, so two readers
ping-ponged: each overwrote the other's points and each kept seeing "not
indexed for me", reprocessing every scan. Production telemetry (note
386945, finding #5) measured identical docs re-processed every few hours
at 7-13s each, with PDF parse ~62% of per-doc cost.

Layer 1 — tenant-wide dedup:
- Thread the scanner's tag-REPORT etag into the file DocumentTask and the
  chunk payload; index `etag` as a KEYWORD field.
- vector/sharing_state.find_indexed_content scrolls tenant-wide (no
  user_id filter) for a non-placeholder point matching
  (doc_id, doc_type, etag), gated on embedding_identity in Python so a
  model switch correctly forces a re-embed.
- Scanner skips enqueue and the processor skips fetch/parse/embed when a
  match exists (cross-worker race-guard before WebDAV read). Dedup is
  fail-safe: a Qdrant error degrades to "process normally".

Layer 2 — observed-access ACL (no admin / GroupFolders API needed):
- Each point carries `acl_principals` = the set of user:<uid> whose
  scanner has observed (hence can read) the file. The per-user tag REPORT
  is the access oracle; group membership/GroupFolders enumeration is
  admin-only and unavailable in multi-user modes.
- build_ownership_filter ORs MatchAny(acl_principals, ["user:<me>"]) so a
  deduplicated shared/group-folder point surfaces to every reader;
  verify-on-read (_verify_files) remains the precise ACL gate.
- Deletion/eviction become "release one user": drop the principal and
  delete the points only when the set empties, so one user untagging a
  shared file doesn't evict it for the others. Legacy points without the
  field keep the original per-user delete.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-04 12:55:17 +02:00
co-authored by Claude Opus 4.8
parent 0919513f21
commit 1c93e7286d
9 changed files with 671 additions and 68 deletions
+10 -23
View File
@@ -9,10 +9,7 @@ simply re-verify and re-attempt.
import logging
from qdrant_client.models import FieldCondition, Filter, MatchValue
from nextcloud_mcp_server.config import get_settings
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
from nextcloud_mcp_server.vector.sharing_state import release_document_for_user
logger = logging.getLogger(__name__)
@@ -22,39 +19,29 @@ async def delete_document_points(
doc_type: str,
user_id: str,
) -> None:
"""Remove all Qdrant points for a single document.
"""Revoke one user's access to a document's points (verify-on-read eviction).
Deletes both real chunk points and any leftover placeholder points for the
given (user_id, doc_id, doc_type) tuple. Safe to call when the document is
A document can be indexed once and shared across users (user-agnostic point
IDs), so eviction must *release* this user rather than blindly delete: it
drops ``user:<user_id>`` from the point's ``acl_principals`` and removes the
points only when no reader remains. Legacy points without a principal set
fall back to the original per-user delete. Safe to call when the document is
not present — Qdrant returns successfully with zero points affected.
Args:
doc_id: Document ID (str — keyword-indexed in Qdrant payload)
doc_type: Document type (note, file, deck_card, news_item)
user_id: Owner of the points being evicted
user_id: User whose access is being revoked
Raises:
Exception: If the underlying Qdrant client raises. Callers in the
search hot path should catch and log; eviction failures must not
block search responses.
"""
qdrant_client = await get_qdrant_client()
settings = get_settings()
await qdrant_client.delete(
collection_name=settings.get_collection_name(),
points_selector=Filter(
must=[
FieldCondition(key="user_id", match=MatchValue(value=user_id)),
FieldCondition(key="doc_id", match=MatchValue(value=doc_id)),
FieldCondition(key="doc_type", match=MatchValue(value=doc_type)),
]
),
)
await release_document_for_user(doc_id, doc_type, user_id)
logger.info(
"Evicted Qdrant points for %s_%s (user=%s); "
"document was inaccessible at verification time",
"Released %s_%s for user=%s; document was inaccessible at verification time",
doc_type,
doc_id,
user_id,