Chris CoutinhoandClaude Opus 4.7
719b3b5034
Production was logging two cascading classes of Qdrant errors against the
welcomed-malamute deployment:
1. HTTP 400 — "Bad request: Index required but not found for \"doc_id\" of
one of the following types: [keyword]". The collection was created via
create_collection() with no payload indexes, so any FieldCondition
filter on doc_id failed at the Qdrant layer (placeholder writes/reads,
eviction, search context lookups).
2. Compounding the missing index, producers wrote a mix of int and str
doc_ids: webhook_parser stringified node_id, scanner stringified note
IDs, news IDs, and deck card IDs — but the file scanner passed the
numeric file_id through unchanged. A keyword index would not have
covered both kinds even if it had existed.
This change:
- Normalizes doc_id to str at every producer site (scanner.py:459,
DocumentTask.doc_id, indexed_*_ids reads from Qdrant).
- Tightens str|int annotations to str across placeholder.py,
eviction.py, search/verification.py, search/context.py,
SearchResult.id, and the auth/api visualization endpoints.
- Defensive str() coercion on doc_id reads in semantic.py /
bm25_hybrid.py / vector/visualization.py for the transition window
before the backfill runs.
- Adds an idempotent startup migration in get_qdrant_client():
- _ensure_keyword_payload_indexes creates KEYWORD indexes for
doc_id, user_id, and doc_type (tolerates "already exists" 400s).
- _backfill_doc_id_to_string scrolls the collection once and rewrites
int doc_ids to str. Skipped after a quick sample shows no legacy
int payloads.
- Public API preserved: SemanticSearchResult.id stays int via explicit
int(r.id) narrowing in server/semantic.py — surfaces a TypeError with
actionable context if a future doc_type ships non-numeric ids.
- Documents the startup migration in docs/configuration.md.
Tests: 11 new unit tests in tests/unit/vector/test_qdrant_client.py
covering happy path / already-exists / unrelated-400 for the index
helpers, and sample-skip / mixed-batch rewrite / payload=None edge cases
for the backfill. 889 unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
62 lines
2.1 KiB
Python
62 lines
2.1 KiB
Python
"""Lazy eviction of stale documents from the vector index.
|
|
|
|
Used by the verify-on-read path (ADR-019) to remove points for documents that
|
|
have been deleted or unshared in Nextcloud but not yet reconciled by the
|
|
webhook/scanner sync loop. Eviction is fire-and-forget from the search hot
|
|
path; failures are logged but never propagated, since the next query will
|
|
simply re-verify and re-attempt.
|
|
"""
|
|
|
|
import logging
|
|
|
|
from qdrant_client.models import FieldCondition, Filter, MatchValue
|
|
|
|
from nextcloud_mcp_server.config import get_settings
|
|
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
async def delete_document_points(
|
|
doc_id: str,
|
|
doc_type: str,
|
|
user_id: str,
|
|
) -> None:
|
|
"""Remove all Qdrant points for a single document.
|
|
|
|
Deletes both real chunk points and any leftover placeholder points for the
|
|
given (user_id, doc_id, doc_type) tuple. Safe to call when the document is
|
|
not present — Qdrant returns successfully with zero points affected.
|
|
|
|
Args:
|
|
doc_id: Document ID (str — keyword-indexed in Qdrant payload)
|
|
doc_type: Document type (note, file, deck_card, news_item)
|
|
user_id: Owner of the points being evicted
|
|
|
|
Raises:
|
|
Exception: If the underlying Qdrant client raises. Callers in the
|
|
search hot path should catch and log; eviction failures must not
|
|
block search responses.
|
|
"""
|
|
qdrant_client = await get_qdrant_client()
|
|
settings = get_settings()
|
|
|
|
await qdrant_client.delete(
|
|
collection_name=settings.get_collection_name(),
|
|
points_selector=Filter(
|
|
must=[
|
|
FieldCondition(key="user_id", match=MatchValue(value=user_id)),
|
|
FieldCondition(key="doc_id", match=MatchValue(value=doc_id)),
|
|
FieldCondition(key="doc_type", match=MatchValue(value=doc_type)),
|
|
]
|
|
),
|
|
)
|
|
|
|
logger.info(
|
|
"Evicted Qdrant points for %s_%s (user=%s); "
|
|
"document was inaccessible at verification time",
|
|
doc_type,
|
|
doc_id,
|
|
user_id,
|
|
)
|