test(integration): address round-7 review — keep RAG assertion live, fix races

- test_no_results_for_unrelated_query: replace pytest.skip with `unrelated =
  ... or 0.0` and fall through. The physics query almost always returns nothing
  on this corpus, so the skip meant the comparison (and the manual-is-indexed
  check) never ran. Treating no-results as score 0.0 keeps the test live and
  vacuously satisfies `0.0 <= relevant`.
- test_sampling: the three limit/threshold/max-tokens tests now gate on a
  representative created note being searchable (search_term + note_id) instead
  of a bare idle signal that can fire before the new notes are enqueued.
- _get_with_retry: only sleep between attempts, not before giving up.
- _search_helpers: log the id/doc_type schema-drift mismatch at WARNING (CI runs
  --log-cli-level=WARN) so it surfaces instead of hiding behind a timeout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-17 23:19:41 +02:00
co-authored by Claude Opus 4.8
parent 4c7c627e51
commit ca313e7271
4 changed files with 29 additions and 16 deletions
+9 -6
View File
@@ -432,13 +432,16 @@ async def test_no_results_for_unrelated_query(nc_mcp_client, indexed_manual_pdf)
query happened to retrieve any chunk at all). Comparing against a relevant
query on the same corpus is self-calibrating and stable.
"""
unrelated = await _top_score(
nc_mcp_client, "quantum entanglement hadron collider particle physics"
# No results for the nonsense query is the ideal outcome — treat as score
# 0.0 and fall through, so the comparison (and the manual-is-indexed check
# below) still runs instead of the test silently skipping every time the
# physics query finds nothing.
unrelated = (
await _top_score(
nc_mcp_client, "quantum entanglement hadron collider particle physics"
)
or 0.0
)
if unrelated is None:
# No results for the nonsense query is the ideal outcome; skip (rather
# than a silent pass) so the test report shows the path was taken.
pytest.skip("No results for nonsense physics query — the ideal outcome")
relevant = await _top_score(
nc_mcp_client, "how do I enable two-factor authentication"