fix(vector): address PR review round 16 — type-aware index check, comments
Detect pre-existing payload indexes with the wrong schema type in `_ensure_payload_indexes`. The previous "field already in existing_schema → skip" branch silently survived a collection migrated from the int-doc_id era where `doc_id` is indexed as INTEGER, letting `MatchValue(value="123")` searches keep failing with HTTP 400 on Qdrant Cloud strict mode — exactly the production failure this PR was meant to fix. New behaviour: compare `existing_schema[field].data_type` against the declared type; on mismatch log a WARNING and append to `failed_fields` so the consolidated end-of-function summary picks it up. No auto-repair (operator intervention only — see docs/configuration.md recovery procedure). New test exercises the doc_id-INTEGER scenario end-to-end and asserts both the per-field WARNING and the summary line. Clarify the `_verify_news_items` malformed-doc_id rationale: the news API has no per-item endpoint, so a malformed doc_id genuinely cannot be verified against the source of truth. We err toward false-positive (keep) over false-negative (drop) — same conservative posture as `_verify_notes` and `_verify_deck_cards`. The producer-side validation is the real security boundary; the verifier is defence-in-depth. Both the inline comment and the WARNING message now spell this out. Add a TODO in `get_last_indexed_timestamp` flagging the O(N) cost on every incremental sync tick. The previous single-page `limit=10_000` silently bounded the scroll; paginating fixed correctness but made the unbounded cost visible. The follow-up tracker (canonical TODO at `api/visualization.py`) covers migrating the max-`indexed_at` to a sentinel point or collection metadata for O(1) lookup. Consolidate the duplicate non-numeric-doc_type TODOs at `api/visualization.py:508` and `auth/viz_routes.py:570` into a single canonical comment in `visualization.py`; `viz_routes.py` is reduced to a back-reference. Removes the rot risk of "fixed in one place, forgotten in the other." The canonical comment also references the O(1) timestamp follow-up in `scanner.py`. Document the `batch_size = 256` (qdrant_client.py) vs `_DELETION_TRACKING_PAGE_SIZE = 1024` (scanner.py) split with cross-referencing comments at each site: the smaller batch is for the read-write backfill upsert path (Qdrant accepts ~256-point chunks comfortably); the larger page is for read-only deletion-tracking scrolls where no per-page write round-trip applies. Replace `assert qdrant_client is not None` in `scan_user_documents` with `cast(AsyncQdrantClient, qdrant_client)` plus an explanatory comment. `assert` is silently elided under `-O`; `cast` is the conventional zero-cost narrower for branches the type checker can't infer from the surrounding `if not initial_sync` ternary. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
68506f96c5
commit
8f4f5c0079
@@ -158,6 +158,49 @@ async def test_ensure_payload_indexes_skips_fields_already_indexed(mocker, caplo
|
||||
assert not any("doc_id" in m for m in info_messages), info_messages
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
async def test_ensure_payload_indexes_warns_on_wrong_schema_type(mocker, caplog):
|
||||
"""Pre-existing index with wrong schema type surfaces as a WARNING.
|
||||
|
||||
The bug this PR fixes: a collection migrated from the int-doc_id era
|
||||
can have ``doc_id`` indexed as INTEGER, which silently survives the
|
||||
"field already in schema → skip" branch and lets ``MatchValue(value="123")``
|
||||
keep failing with HTTP 400 on Qdrant Cloud strict mode. Confirm the
|
||||
type-aware check fires a WARNING, marks the field as failed (so the
|
||||
consolidated end-of-function summary picks it up), and does NOT
|
||||
attempt to recreate the index — operator intervention is the only
|
||||
safe path.
|
||||
"""
|
||||
client = mocker.AsyncMock()
|
||||
# PayloadIndexInfo-like stand-in: only ``data_type`` is read.
|
||||
wrong = SimpleNamespace(data_type=PayloadSchemaType.INTEGER)
|
||||
client.get_collection.return_value = SimpleNamespace(
|
||||
payload_schema={"doc_id": wrong}
|
||||
)
|
||||
|
||||
with caplog.at_level("WARNING", logger="nextcloud_mcp_server.vector.qdrant_client"):
|
||||
await _ensure_payload_indexes(client, "test-collection")
|
||||
|
||||
# No create attempt for the mismatched field.
|
||||
created_fields = {
|
||||
c.kwargs["field_name"] for c in client.create_payload_index.await_args_list
|
||||
}
|
||||
assert "doc_id" not in created_fields
|
||||
|
||||
warning_messages = [
|
||||
r.getMessage() for r in caplog.records if r.levelname == "WARNING"
|
||||
]
|
||||
# Per-field warning describes both observed and expected types.
|
||||
assert any(
|
||||
"doc_id" in m and "INTEGER" in m and "KEYWORD" in m for m in warning_messages
|
||||
), warning_messages
|
||||
# Consolidated summary at end of function includes the field too.
|
||||
assert any(
|
||||
"Payload index creation incomplete" in m and "doc_id" in m
|
||||
for m in warning_messages
|
||||
), warning_messages
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
async def test_ensure_payload_indexes_logs_400_as_warning(mocker, caplog):
|
||||
"""Any 400 from create_payload_index is logged at WARNING and skipped.
|
||||
|
||||
Reference in New Issue
Block a user