feat: dedup shared-file parsing/embedding across users in vector sync
A file shared across many users — directly, or via a group folder shared to a group — was parsed and embedded once per user. Chunk point IDs are user-agnostic (uuid5(tenant_id, doc_id=fileid, chunk_index)), but the per-user freshness gate filtered Qdrant by user_id, so two readers ping-ponged: each overwrote the other's points and each kept seeing "not indexed for me", reprocessing every scan. Production telemetry (note 386945, finding #5) measured identical docs re-processed every few hours at 7-13s each, with PDF parse ~62% of per-doc cost. Layer 1 — tenant-wide dedup: - Thread the scanner's tag-REPORT etag into the file DocumentTask and the chunk payload; index `etag` as a KEYWORD field. - vector/sharing_state.find_indexed_content scrolls tenant-wide (no user_id filter) for a non-placeholder point matching (doc_id, doc_type, etag), gated on embedding_identity in Python so a model switch correctly forces a re-embed. - Scanner skips enqueue and the processor skips fetch/parse/embed when a match exists (cross-worker race-guard before WebDAV read). Dedup is fail-safe: a Qdrant error degrades to "process normally". Layer 2 — observed-access ACL (no admin / GroupFolders API needed): - Each point carries `acl_principals` = the set of user:<uid> whose scanner has observed (hence can read) the file. The per-user tag REPORT is the access oracle; group membership/GroupFolders enumeration is admin-only and unavailable in multi-user modes. - build_ownership_filter ORs MatchAny(acl_principals, ["user:<me>"]) so a deduplicated shared/group-folder point surfaces to every reader; verify-on-read (_verify_files) remains the precise ACL gate. - Deletion/eviction become "release one user": drop the principal and delete the points only when the set empties, so one user untagging a shared file doesn't evict it for the others. Legacy points without the field keep the original per-user delete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
0919513f21
commit
1c93e7286d
@@ -31,6 +31,7 @@ from nextcloud_mcp_server.vector.placeholder import (
|
||||
)
|
||||
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
|
||||
from nextcloud_mcp_server.vector.queue.ports import TaskProducer
|
||||
from nextcloud_mcp_server.vector.sharing_state import claim_existing_index
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -448,6 +449,24 @@ async def scan_user_documents(
|
||||
except (ValueError, KeyError):
|
||||
pass
|
||||
|
||||
# Tenant-wide content dedup (Layer 1 / observed-access ACL): if
|
||||
# this exact file content (fileid + etag) is already indexed under
|
||||
# the current embedding model by ANY user in the tenant, skip
|
||||
# re-parsing/re-embedding and just record that this user can read
|
||||
# it. Eliminates the per-user reprocessing ping-pong that arises
|
||||
# because chunk point IDs are user-agnostic (note 386945 #5).
|
||||
etag = str(file_info.get("etag") or "")
|
||||
if etag and await claim_existing_index(file_id, "file", etag, user_id):
|
||||
_potentially_deleted.pop((user_id, file_id), None)
|
||||
logger.debug(
|
||||
"Dedup: file %s (ID: %s) already indexed in tenant; "
|
||||
"granted access to %s without reprocessing",
|
||||
file_path,
|
||||
file_id,
|
||||
user_id,
|
||||
)
|
||||
continue
|
||||
|
||||
if initial_sync:
|
||||
# Send everything on first sync - write placeholder first
|
||||
await write_placeholder_point(
|
||||
@@ -455,6 +474,7 @@ async def scan_user_documents(
|
||||
doc_type="file",
|
||||
user_id=user_id,
|
||||
modified_at=modified_at,
|
||||
etag=etag,
|
||||
file_path=file_path,
|
||||
)
|
||||
await send_stream.send(
|
||||
@@ -465,6 +485,7 @@ async def scan_user_documents(
|
||||
operation="index",
|
||||
modified_at=modified_at,
|
||||
file_path=file_path,
|
||||
etag=etag,
|
||||
)
|
||||
)
|
||||
file_queued += 1
|
||||
@@ -525,6 +546,7 @@ async def scan_user_documents(
|
||||
doc_type="file",
|
||||
user_id=user_id,
|
||||
modified_at=modified_at,
|
||||
etag=etag,
|
||||
file_path=file_path,
|
||||
)
|
||||
await send_stream.send(
|
||||
@@ -535,6 +557,7 @@ async def scan_user_documents(
|
||||
operation="index",
|
||||
modified_at=modified_at,
|
||||
file_path=file_path,
|
||||
etag=etag,
|
||||
)
|
||||
)
|
||||
file_queued += 1
|
||||
|
||||
Reference in New Issue
Block a user