Chris CoutinhoandClaude Opus 4.8
1c93e7286d
A file shared across many users — directly, or via a group folder shared to a group — was parsed and embedded once per user. Chunk point IDs are user-agnostic (uuid5(tenant_id, doc_id=fileid, chunk_index)), but the per-user freshness gate filtered Qdrant by user_id, so two readers ping-ponged: each overwrote the other's points and each kept seeing "not indexed for me", reprocessing every scan. Production telemetry (note 386945, finding #5) measured identical docs re-processed every few hours at 7-13s each, with PDF parse ~62% of per-doc cost. Layer 1 — tenant-wide dedup: - Thread the scanner's tag-REPORT etag into the file DocumentTask and the chunk payload; index `etag` as a KEYWORD field. - vector/sharing_state.find_indexed_content scrolls tenant-wide (no user_id filter) for a non-placeholder point matching (doc_id, doc_type, etag), gated on embedding_identity in Python so a model switch correctly forces a re-embed. - Scanner skips enqueue and the processor skips fetch/parse/embed when a match exists (cross-worker race-guard before WebDAV read). Dedup is fail-safe: a Qdrant error degrades to "process normally". Layer 2 — observed-access ACL (no admin / GroupFolders API needed): - Each point carries `acl_principals` = the set of user:<uid> whose scanner has observed (hence can read) the file. The per-user tag REPORT is the access oracle; group membership/GroupFolders enumeration is admin-only and unavailable in multi-user modes. - build_ownership_filter ORs MatchAny(acl_principals, ["user:<me>"]) so a deduplicated shared/group-folder point surfaces to every reader; verify-on-read (_verify_files) remains the precise ACL gate. - Deletion/eviction become "release one user": drop the principal and delete the points only when the set empties, so one user untagging a shared file doesn't evict it for the others. Legacy points without the field keep the original per-user delete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
49 lines
1.8 KiB
Python
49 lines
1.8 KiB
Python
"""Lazy eviction of stale documents from the vector index.
|
|
|
|
Used by the verify-on-read path (ADR-019) to remove points for documents that
|
|
have been deleted or unshared in Nextcloud but not yet reconciled by the
|
|
webhook/scanner sync loop. Eviction is fire-and-forget from the search hot
|
|
path; failures are logged but never propagated, since the next query will
|
|
simply re-verify and re-attempt.
|
|
"""
|
|
|
|
import logging
|
|
|
|
from nextcloud_mcp_server.vector.sharing_state import release_document_for_user
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
async def delete_document_points(
|
|
doc_id: str,
|
|
doc_type: str,
|
|
user_id: str,
|
|
) -> None:
|
|
"""Revoke one user's access to a document's points (verify-on-read eviction).
|
|
|
|
A document can be indexed once and shared across users (user-agnostic point
|
|
IDs), so eviction must *release* this user rather than blindly delete: it
|
|
drops ``user:<user_id>`` from the point's ``acl_principals`` and removes the
|
|
points only when no reader remains. Legacy points without a principal set
|
|
fall back to the original per-user delete. Safe to call when the document is
|
|
not present — Qdrant returns successfully with zero points affected.
|
|
|
|
Args:
|
|
doc_id: Document ID (str — keyword-indexed in Qdrant payload)
|
|
doc_type: Document type (note, file, deck_card, news_item)
|
|
user_id: User whose access is being revoked
|
|
|
|
Raises:
|
|
Exception: If the underlying Qdrant client raises. Callers in the
|
|
search hot path should catch and log; eviction failures must not
|
|
block search responses.
|
|
"""
|
|
await release_document_for_user(doc_id, doc_type, user_id)
|
|
|
|
logger.info(
|
|
"Released %s_%s for user=%s; document was inaccessible at verification time",
|
|
doc_type,
|
|
doc_id,
|
|
user_id,
|
|
)
|