fix(vector): normalize doc_id to str + add Qdrant keyword payload indexes

Production was logging two cascading classes of Qdrant errors against the
welcomed-malamute deployment:

1. HTTP 400 — "Bad request: Index required but not found for \"doc_id\" of
   one of the following types: [keyword]". The collection was created via
   create_collection() with no payload indexes, so any FieldCondition
   filter on doc_id failed at the Qdrant layer (placeholder writes/reads,
   eviction, search context lookups).

2. Compounding the missing index, producers wrote a mix of int and str
   doc_ids: webhook_parser stringified node_id, scanner stringified note
   IDs, news IDs, and deck card IDs — but the file scanner passed the
   numeric file_id through unchanged. A keyword index would not have
   covered both kinds even if it had existed.

This change:

- Normalizes doc_id to str at every producer site (scanner.py:459,
  DocumentTask.doc_id, indexed_*_ids reads from Qdrant).
- Tightens str|int annotations to str across placeholder.py,
  eviction.py, search/verification.py, search/context.py,
  SearchResult.id, and the auth/api visualization endpoints.
- Defensive str() coercion on doc_id reads in semantic.py /
  bm25_hybrid.py / vector/visualization.py for the transition window
  before the backfill runs.
- Adds an idempotent startup migration in get_qdrant_client():
  - _ensure_keyword_payload_indexes creates KEYWORD indexes for
    doc_id, user_id, and doc_type (tolerates "already exists" 400s).
  - _backfill_doc_id_to_string scrolls the collection once and rewrites
    int doc_ids to str. Skipped after a quick sample shows no legacy
    int payloads.
- Public API preserved: SemanticSearchResult.id stays int via explicit
  int(r.id) narrowing in server/semantic.py — surfaces a TypeError with
  actionable context if a future doc_type ships non-numeric ids.
- Documents the startup migration in docs/configuration.md.

Tests: 11 new unit tests in tests/unit/vector/test_qdrant_client.py
covering happy path / already-exists / unrelated-400 for the index
helpers, and sample-skip / mixed-batch rewrite / payload=None edge cases
for the backfill. 889 unit tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-05-08 19:30:51 +02:00
co-authored by Claude Opus 4.7
parent 0690378915
commit 719b3b5034
17 changed files with 493 additions and 66 deletions
+8 -6
View File
@@ -48,7 +48,7 @@ class DocumentTask:
"""Document task for processing queue."""
user_id: str
doc_id: int | str # int for files/notes, str for legacy
doc_id: str # Always str — see vector/qdrant_client.py keyword index
doc_type: str # "note", "file", "calendar"
operation: str # "index" or "delete"
modified_at: int
@@ -228,7 +228,7 @@ async def scan_user_documents(
)
indexed_doc_ids = {
point.payload["doc_id"]
str(point.payload["doc_id"])
for point in (scroll_result[0] or [])
if point.payload is not None
}
@@ -401,7 +401,7 @@ async def scan_user_documents(
)
indexed_file_ids = {
point.payload["doc_id"]
str(point.payload["doc_id"])
for point in (file_scroll_result[0] or [])
if point.payload is not None
}
@@ -456,7 +456,9 @@ async def scan_user_documents(
for file_info in tagged_files:
# Files are already filtered by MIME type in find_files_by_tag()
file_count += 1
file_id = file_info["id"] # Use numeric file ID, not path
# Normalize file ID to str — Qdrant doc_id payload is keyword-indexed
# and producers across doc_types must agree on a single type.
file_id = str(file_info["id"])
file_path = file_info["path"] # Keep path for logging
nextcloud_file_ids.add(file_id)
@@ -679,7 +681,7 @@ async def scan_news_items(
limit=10000,
)
indexed_item_ids = {
point.payload["doc_id"]
str(point.payload["doc_id"])
for point in (scroll_result[0] or [])
if point.payload is not None
}
@@ -858,7 +860,7 @@ async def scan_deck_cards(
limit=10000,
)
indexed_card_ids = {
point.payload["doc_id"]
str(point.payload["doc_id"])
for point in (scroll_result[0] or [])
if point.payload is not None
}