feat: implement RAG evaluation framework with CLI tooling

- Add ADR-013 documenting RAG evaluation architecture
- Implement two-part evaluation: Context Recall (retrieval) + Answer Correctness (generation)
- Create Click CLI for ground truth generation and corpus upload
- Add pytest fixtures and tests for retrieval/generation quality
- Use BeIR/nfcorpus dataset with 5 selected test queries
- Support Ollama and Anthropic LLM providers
- Generate synthetic ground truth answers offline
- Add comprehensive documentation in tests/rag_evaluation/README.md

The framework separates one-time setup (generate/upload) from test execution,
making tests much faster (~6-12 min vs ~15-25 min per run).

Tests are manual only (not in CI) and require external LLM access.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

tests/rag_evaluation/README.md

@@ -0,0 +1,277 @@
+# RAG Evaluation Tests
+
+This directory contains tests for evaluating the Retrieval-Augmented Generation (RAG) system in the Nextcloud MCP server, specifically the `nc_semantic_search_answer` tool.
+
+## Architecture
+
+The RAG system has two components that are tested independently:
+
+1. **Retrieval** - Vector sync/embedding pipeline (indexed Nextcloud documents → vector database)
+2. **Generation** - MCP client LLM synthesis (retrieved context → natural language answer)
+
+See [ADR-013](../../docs/ADR-013-rag-evaluation.md) for full architectural details.
+
+## Test Structure
+
+```
+tests/rag_evaluation/
+├── README.md                       # This file
+├── conftest.py                     # Pytest fixtures
+├── llm_providers.py                # LLM provider abstraction (Ollama/Anthropic)
+├── fixtures/
+│   └── ground_truth.json           # Pre-generated reference answers
+├── test_retrieval_quality.py       # Retrieval evaluation (Context Recall)
+└── test_generation_quality.py      # Generation evaluation (Answer Correctness)
+```
+
+## Metrics
+
+### Retrieval Evaluation
+- **Metric**: Context Recall
+- **Method**: Heuristic - Check if ground-truth document IDs appear in top-k results
+- **Target**: ≥80% recall
+
+### Generation Evaluation
+- **Metric**: Answer Correctness
+- **Method**: LLM-as-judge - Compare RAG answer vs ground truth (binary true/false)
+- **Evaluation**: External LLM evaluates semantic equivalence
+
+## Dataset
+
+**BeIR/nfcorpus** - Medical/biomedical corpus with ~3,600 documents
+
+**Test Queries** (5 selected):
+1. PLAIN-2630: "Alkylphenol Endocrine Disruptors and Allergies" (21 relevant docs)
+2. PLAIN-2660: "How Long to Detox From Fish Before Pregnancy?" (20 relevant docs)
+3. PLAIN-2510: "Coffee and Artery Function" (16 relevant docs)
+4. PLAIN-2430: "Preventing Brain Loss with B Vitamins?" (15 relevant docs)
+5. PLAIN-2690: "Chronic Headaches and Pork Tapeworms" (14 relevant docs)
+
+## Setup
+
+### 1. Install Dependencies
+
+```bash
+uv sync --group dev
+```
+
+This installs:
+- `anthropic>=0.42.0` - For Anthropic LLM evaluation
+- `click>=8.1.8` - For CLI interface
+- `datasets>=3.3.0` - For BeIR nfcorpus dataset loading
+
+### 2. Configure LLM Provider
+
+Set environment variables for your LLM provider:
+
+**Option A: Ollama (default, local/remote)**
+```bash
+export RAG_EVAL_PROVIDER=ollama
+export OLLAMA_HOST=https://ollama.example.com  # or RAG_EVAL_OLLAMA_BASE_URL
+export RAG_EVAL_OLLAMA_MODEL=llama3.2:1b
+```
+
+**Option B: Anthropic (cloud)**
+```bash
+export RAG_EVAL_PROVIDER=anthropic
+export RAG_EVAL_ANTHROPIC_API_KEY=sk-ant-...
+export RAG_EVAL_ANTHROPIC_MODEL=claude-3-5-sonnet-20241022
+```
+
+### 3. One-Time Setup: Generate Ground Truth
+
+Generate synthetic reference answers for the 5 test queries:
+
+```bash
+uv run python tools/rag_eval_cli.py generate
+```
+
+**What this does:**
+- Downloads nfcorpus dataset to `tests/rag_evaluation/fixtures/nfcorpus/` (cached locally)
+- For each of the 5 selected queries, extracts highly relevant documents
+- Uses configured LLM to synthesize a reference answer
+- Saves to `tests/rag_evaluation/fixtures/ground_truth.json`
+
+**Optional flags:**
+- `--provider ollama|anthropic` - Override LLM provider
+- `--model MODEL_NAME` - Override model name
+- `--force-download` - Re-download nfcorpus dataset
+
+### 4. One-Time Setup: Upload Corpus to Nextcloud
+
+Upload all 3,633 nfcorpus documents as Nextcloud notes:
+
+```bash
+uv run python tools/rag_eval_cli.py upload \
+    --nextcloud-url http://localhost:8000 \
+    --username admin \
+    --password admin
+```
+
+**What this does:**
+- Downloads nfcorpus dataset (if not already cached)
+- Uploads all documents as notes in Nextcloud
+- Saves document ID → note ID mapping to `tests/rag_evaluation/fixtures/note_mapping.json`
+
+**Optional flags:**
+- `--category CATEGORY` - Custom category for notes (default: `nfcorpus_rag_eval`)
+- `--force-download` - Re-download nfcorpus dataset
+
+**Important:** This step requires:
+- A running Nextcloud instance with vector sync enabled
+- Notes app installed
+- Valid credentials
+
+**Duration:** ~10-15 minutes to upload 3,633 documents
+
+## Running Tests
+
+### Run All RAG Evaluation Tests
+
+```bash
+uv run pytest tests/rag_evaluation/ -v
+```
+
+### Run Specific Test Suites
+
+**Retrieval Quality Only:**
+```bash
+uv run pytest tests/rag_evaluation/test_retrieval_quality.py -v
+```
+
+**Generation Quality Only:**
+```bash
+uv run pytest tests/rag_evaluation/test_generation_quality.py -v
+```
+
+### Run Individual Tests
+
+```bash
+uv run pytest tests/rag_evaluation/test_retrieval_quality.py::test_retrieval_context_recall -v
+uv run pytest tests/rag_evaluation/test_generation_quality.py::test_answer_correctness -v
+```
+
+## Test Execution Flow
+
+**Prerequisites** (one-time setup):
+1. Generated ground truth (`tools/rag_eval_cli.py generate`)
+2. Uploaded corpus to Nextcloud (`tools/rag_eval_cli.py upload`)
+
+### Retrieval Quality Tests
+
+1. **Setup** (`nfcorpus_test_data` fixture):
+   - Loads pre-generated ground truth from `fixtures/ground_truth.json`
+   - Loads note mapping from `fixtures/note_mapping.json`
+   - Returns test cases with expected note IDs
+
+2. **Test** (`test_retrieval_context_recall`):
+   - For each query: Perform semantic search (top-10)
+   - Extract retrieved note IDs
+   - Calculate Context Recall = (expected ∩ retrieved) / expected
+   - Assert recall ≥ 80%
+
+3. **Cleanup**:
+   - None required (notes persist in Nextcloud for reuse)
+
+### Generation Quality Tests
+
+1. **Setup**:
+   - Same as retrieval tests (reuses `nfcorpus_test_data` fixture)
+   - Creates evaluation LLM provider
+
+2. **Test** (`test_answer_correctness`):
+   - For each query: Call `nc_semantic_search_answer` MCP tool
+   - Extract generated answer
+   - Use LLM-as-judge to compare vs ground truth
+   - Assert semantic equivalence (TRUE/FALSE)
+
+3. **Cleanup**:
+   - LLM provider closed
+
+## Expected Test Duration
+
+**One-time setup:**
+- **Generate ground truth**: ~5-10 minutes (5 queries with LLM generation)
+- **Upload corpus**: ~10-15 minutes (3,633 documents)
+- **Total setup**: ~15-25 minutes
+
+**Test execution** (after setup):
+- **Retrieval tests**: ~1-2 minutes (5 queries, no upload/cleanup)
+- **Generation tests**: ~5-10 minutes (RAG generation + LLM evaluation)
+- **Total per run**: ~6-12 minutes
+
+**Note**: These are NOT smoke tests and are NOT run in CI.
+
+## Limitations & Future Work
+
+**Current Limitations:**
+- Only 5 test queries (limited statistical confidence)
+- Medical domain bias (may not represent production use cases)
+- Synthetic ground truth (LLM-generated, not human-validated)
+- Manual test execution (requires external LLM access)
+
+**Future Enhancements:**
+- Expand to 50-100 queries for statistical significance
+- Add custom test dataset with production-representative documents
+- Implement additional metrics (faithfulness, context relevance, answer relevance)
+- Create automated benchmarking dashboard
+- Test multi-hop reasoning (synthesis questions)
+- Evaluate out-of-scope handling ("I don't know" responses)
+
+## Troubleshooting
+
+### Tests Fail with "Ground truth file not found"
+
+Run the generate command first:
+```bash
+uv run python tools/rag_eval_cli.py generate
+```
+
+### Tests Fail with "Note mapping file not found"
+
+Run the upload command first:
+```bash
+uv run python tools/rag_eval_cli.py upload --nextcloud-url http://localhost:8000 --username admin --password admin
+```
+
+### Tests Fail with "MCP sampling client not yet implemented"
+
+The `mcp_sampling_client` fixture is a placeholder. You need to implement MCP client creation with sampling support. See the TODO in `conftest.py`.
+
+### Upload Command Fails
+
+Common issues:
+1. **Nextcloud not running**: Ensure Nextcloud is accessible at the URL
+2. **Invalid credentials**: Verify username/password
+3. **Notes app not installed**: Install Notes app in Nextcloud
+4. **Network timeout**: Increase timeout in CLI (currently 60s)
+
+### LLM Timeout
+
+If ground truth generation times out:
+1. Increase timeout in `llm_providers.py` (currently 10 min)
+2. Use a faster model: `--model llama3.2:1b`
+3. Check Ollama/Anthropic service availability
+
+### Dataset Download Fails
+
+The nfcorpus dataset is downloaded automatically. If download fails:
+1. Check internet connection
+2. Manually download from: https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/nfcorpus.zip
+3. Extract to `tests/rag_evaluation/fixtures/nfcorpus/`
+4. Or use HuggingFace datasets cache: `~/.cache/huggingface/datasets/BeIR___nfcorpus/`
+
+### Vector Sync Not Indexing Documents
+
+After uploading, vector sync must index the documents:
+1. Check vector sync is enabled in Nextcloud
+2. Trigger manual sync if needed
+3. Wait for background job to process all documents
+4. Verify in Qdrant that vectors exist for uploaded notes
+
+## References
+
+- [ADR-013: RAG Evaluation Testing Framework](../../docs/ADR-013-rag-evaluation.md)
+- [ADR-008: MCP Sampling for Semantic Search](../../docs/ADR-008-mcp-sampling-for-semantic-search.md)
+- [BeIR Benchmark](https://github.com/beir-cellar/beir)
+- [NFCorpus Dataset](https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/)

tests/rag_evaluation/__init__.py

@@ -0,0 +1 @@
+"""RAG evaluation tests for the Nextcloud MCP semantic search system."""

tests/rag_evaluation/conftest.py

@@ -0,0 +1,145 @@
+"""Pytest fixtures for RAG evaluation tests.
+
+IMPORTANT: Before running these tests, you must:
+1. Generate ground truth: uv run python tools/rag_eval_cli.py generate
+2. Upload corpus: uv run python tools/rag_eval_cli.py upload --nextcloud-url http://localhost:8000 --username admin --password admin
+
+This ensures that the ground truth and note mappings are available.
+"""
+
+import json
+from pathlib import Path
+from typing import Any
+
+import pytest
+
+from tests.rag_evaluation.llm_providers import create_llm_provider
+
+# Paths
+FIXTURES_DIR = Path(__file__).parent / "fixtures"
+GROUND_TRUTH_FILE = FIXTURES_DIR / "ground_truth.json"
+NOTE_MAPPING_FILE = FIXTURES_DIR / "note_mapping.json"
+
+
+@pytest.fixture(scope="session")
+def ground_truth_data() -> list[dict[str, Any]]:
+    """Load pre-generated ground truth data.
+
+    Returns:
+        List of test cases with query, ground truth answer, and expected doc IDs
+
+    Raises:
+        FileNotFoundError: If ground_truth.json doesn't exist
+    """
+    if not GROUND_TRUTH_FILE.exists():
+        raise FileNotFoundError(
+            f"Ground truth file not found: {GROUND_TRUTH_FILE}\n"
+            "Run: uv run python tools/rag_eval_cli.py generate"
+        )
+
+    with open(GROUND_TRUTH_FILE) as f:
+        return json.load(f)
+
+
+@pytest.fixture(scope="session")
+def note_mapping() -> dict[str, int]:
+    """Load document ID → note ID mapping.
+
+    Returns:
+        Dict mapping nfcorpus document ID to Nextcloud note ID
+
+    Raises:
+        FileNotFoundError: If note_mapping.json doesn't exist
+    """
+    if not NOTE_MAPPING_FILE.exists():
+        raise FileNotFoundError(
+            f"Note mapping file not found: {NOTE_MAPPING_FILE}\n"
+            "Run: uv run python tools/rag_eval_cli.py upload --nextcloud-url ... --username ... --password ..."
+        )
+
+    with open(NOTE_MAPPING_FILE) as f:
+        return json.load(f)
+
+
+@pytest.fixture(scope="session")
+def nfcorpus_test_data(
+    ground_truth_data: list[dict[str, Any]],
+    note_mapping: dict[str, int],
+):
+    """Prepare nfcorpus test data for evaluation.
+
+    This fixture combines ground truth answers with note mappings to create
+    test cases ready for retrieval and generation quality tests.
+
+    Args:
+        ground_truth_data: Pre-generated ground truth answers
+        note_mapping: Document ID → note ID mapping
+
+    Returns:
+        List of test cases with query, ground truth, expected doc IDs, and note IDs
+    """
+    test_cases = []
+
+    for gt in ground_truth_data:
+        # Map expected document IDs to note IDs
+        expected_note_ids = [
+            note_mapping.get(doc_id)
+            for doc_id in gt["expected_document_ids"]
+            if doc_id in note_mapping
+        ]
+
+        # Filter out None values (docs that weren't uploaded)
+        expected_note_ids = [nid for nid in expected_note_ids if nid is not None]
+
+        test_cases.append(
+            {
+                "query_id": gt["query_id"],
+                "query_text": gt["query_text"],
+                "ground_truth_answer": gt["ground_truth_answer"],
+                "expected_document_ids": gt["expected_document_ids"],
+                "expected_note_ids": expected_note_ids,
+                "highly_relevant_count": gt["highly_relevant_count"],
+            }
+        )
+
+    return test_cases
+
+
+@pytest.fixture(scope="session")
+async def evaluation_llm():
+    """Create LLM provider for evaluation (separate from MCP client).
+
+    Environment variables:
+      RAG_EVAL_PROVIDER: Provider type (ollama or anthropic)
+      RAG_EVAL_OLLAMA_BASE_URL: Ollama base URL (or OLLAMA_HOST)
+      RAG_EVAL_OLLAMA_MODEL: Ollama model name
+      RAG_EVAL_ANTHROPIC_API_KEY: Anthropic API key
+      RAG_EVAL_ANTHROPIC_MODEL: Anthropic model name
+
+    Returns:
+        LLM provider instance (OllamaProvider or AnthropicProvider)
+    """
+    llm = create_llm_provider()
+    yield llm
+    await llm.close()
+
+
+@pytest.fixture(scope="session")
+async def mcp_sampling_client():
+    """Create MCP client that supports sampling for RAG generation.
+
+    This fixture creates an MCP client configured to support sampling,
+    which is required for testing the nc_semantic_search_answer tool.
+
+    TODO: Implement MCP client with sampling support
+    For now, this is a placeholder.
+
+    Returns:
+        MCP client instance with sampling enabled
+    """
+    # TODO: Implement MCP client creation with sampling support
+    # This will require:
+    # 1. Creating an MCP client configured for sampling
+    # 2. Authenticating with Nextcloud
+    # 3. Ensuring sampling is enabled
+    pytest.skip("MCP sampling client not yet implemented")

tests/rag_evaluation/llm_providers.py

@@ -0,0 +1,145 @@
+"""LLM provider abstraction for RAG evaluation.
+
+Supports Ollama (local) and Anthropic (cloud) providers for both ground truth
+generation and evaluation.
+"""
+
+import os
+from typing import Protocol
+
+import httpx
+from anthropic import AsyncAnthropic
+
+
+class LLMProvider(Protocol):
+    """Protocol for LLM providers."""
+
+    async def generate(self, prompt: str, max_tokens: int = 500) -> str:
+        """Generate text from a prompt.
+
+        Args:
+            prompt: The prompt to generate from
+            max_tokens: Maximum tokens to generate
+
+        Returns:
+            Generated text
+        """
+        ...
+
+
+class OllamaProvider:
+    """Ollama provider for local LLM inference."""
+
+    def __init__(self, base_url: str, model: str):
+        """Initialize Ollama provider.
+
+        Args:
+            base_url: Ollama API base URL (e.g., http://localhost:11434)
+            model: Model name (e.g., llama3.1:8b)
+        """
+        self.base_url = base_url.rstrip("/")
+        self.model = model
+        self.client = httpx.AsyncClient(timeout=600.0)  # 10 min timeout for generation
+
+    async def generate(self, prompt: str, max_tokens: int = 500) -> str:
+        """Generate text using Ollama API."""
+        response = await self.client.post(
+            f"{self.base_url}/api/generate",
+            json={
+                "model": self.model,
+                "prompt": prompt,
+                "stream": False,
+                "options": {
+                    "num_predict": max_tokens,
+                    "temperature": 0.7,
+                },
+            },
+        )
+        response.raise_for_status()
+        data = response.json()
+        return data["response"]
+
+    async def close(self):
+        """Close the HTTP client."""
+        await self.client.aclose()
+
+
+class AnthropicProvider:
+    """Anthropic provider for cloud LLM inference."""
+
+    def __init__(self, api_key: str, model: str):
+        """Initialize Anthropic provider.
+
+        Args:
+            api_key: Anthropic API key
+            model: Model name (e.g., claude-3-5-sonnet-20241022)
+        """
+        self.client = AsyncAnthropic(api_key=api_key)
+        self.model = model
+
+    async def generate(self, prompt: str, max_tokens: int = 500) -> str:
+        """Generate text using Anthropic API."""
+        message = await self.client.messages.create(
+            model=self.model,
+            max_tokens=max_tokens,
+            temperature=0.7,
+            messages=[{"role": "user", "content": prompt}],
+        )
+        return message.content[0].text
+
+    async def close(self):
+        """Close the client (no-op for Anthropic)."""
+        pass
+
+
+def create_llm_provider(
+    provider: str | None = None,
+    ollama_base_url: str | None = None,
+    ollama_model: str | None = None,
+    anthropic_api_key: str | None = None,
+    anthropic_model: str | None = None,
+) -> LLMProvider:
+    """Create an LLM provider from environment variables or arguments.
+
+    Args:
+        provider: Provider type ('ollama' or 'anthropic'). Defaults to RAG_EVAL_PROVIDER env var or 'ollama'
+        ollama_base_url: Ollama base URL. Defaults to RAG_EVAL_OLLAMA_BASE_URL or 'http://localhost:11434'
+        ollama_model: Ollama model. Defaults to RAG_EVAL_OLLAMA_MODEL or 'llama3.1:8b'
+        anthropic_api_key: Anthropic API key. Defaults to RAG_EVAL_ANTHROPIC_API_KEY env var
+        anthropic_model: Anthropic model. Defaults to RAG_EVAL_ANTHROPIC_MODEL or 'claude-3-5-sonnet-20241022'
+
+    Returns:
+        LLMProvider instance
+
+    Raises:
+        ValueError: If provider is invalid or required credentials are missing
+    """
+    # Get provider from args or env
+    provider = provider or os.environ.get("RAG_EVAL_PROVIDER", "ollama")
+
+    if provider == "ollama":
+        # Try RAG_EVAL_OLLAMA_BASE_URL, then OLLAMA_HOST, then default
+        base_url = (
+            ollama_base_url
+            or os.environ.get("RAG_EVAL_OLLAMA_BASE_URL")
+            or os.environ.get("OLLAMA_HOST")
+            or "http://localhost:11434"
+        )
+        model = ollama_model or os.environ.get("RAG_EVAL_OLLAMA_MODEL", "llama3.2:1b")
+        return OllamaProvider(base_url=base_url, model=model)
+
+    elif provider == "anthropic":
+        api_key = anthropic_api_key or os.environ.get("RAG_EVAL_ANTHROPIC_API_KEY")
+        if not api_key:
+            raise ValueError(
+                "Anthropic API key required. Set RAG_EVAL_ANTHROPIC_API_KEY environment variable."
+            )
+        model = anthropic_model or os.environ.get(
+            "RAG_EVAL_ANTHROPIC_MODEL", "claude-3-5-sonnet-20241022"
+        )
+        return AnthropicProvider(api_key=api_key, model=model)
+
+    else:
+        raise ValueError(
+            f"Invalid provider: {provider}. Must be 'ollama' or 'anthropic'."
+        )

tests/rag_evaluation/test_generation_quality.py

@@ -0,0 +1,139 @@
+"""Tests for RAG generation quality (Answer Correctness metric).
+
+These tests evaluate whether the MCP client LLM generates factually correct
+answers from retrieved context using the nc_semantic_search_answer tool.
+
+Metric: Answer Correctness
+- Measures: Is the generated answer factually correct?
+- Method: LLM-as-judge - Compare RAG answer vs ground truth (binary true/false)
+- Evaluation: External LLM evaluates semantic equivalence
+"""
+
+import pytest
+
+
+@pytest.mark.integration
+async def test_answer_correctness(
+    mcp_sampling_client,
+    evaluation_llm,
+    nfcorpus_test_data,
+):
+    """Test that RAG system generates factually correct answers.
+
+    For each test query:
+    1. Execute full RAG pipeline via nc_semantic_search_answer MCP tool
+    2. Extract generated answer from RAG response
+    3. Use LLM-as-judge to compare against ground truth (binary true/false)
+    4. Assert answer is semantically equivalent to ground truth
+
+    This tests the quality of the generation component (MCP client LLM).
+    """
+    results_summary = []
+
+    for test_case in nfcorpus_test_data:
+        query = test_case["query_text"]
+        ground_truth = test_case["ground_truth_answer"]
+
+        print(f"\n{'=' * 80}")
+        print(f"Query: {query}")
+
+        # Execute full RAG pipeline
+        print("Executing RAG pipeline...")
+        rag_result = await mcp_sampling_client.call_tool(
+            "nc_semantic_search_answer",
+            arguments={"query": query, "limit": 5},
+        )
+
+        rag_answer = rag_result["generated_answer"]
+
+        print(f"RAG Answer preview: {rag_answer[:200]}...")
+        print(f"Ground Truth preview: {ground_truth[:200]}...")
+
+        # LLM-as-judge evaluation
+        evaluation_prompt = f"""Compare these two answers and respond with only TRUE or FALSE.
+
+Question: {query}
+
+Generated Answer: {rag_answer}
+
+Ground Truth Answer: {ground_truth}
+
+Are these answers semantically equivalent (do they convey the same factual information)?
+Respond with only: TRUE or FALSE"""
+
+        print("Evaluating answer correctness...")
+        evaluation_result = await evaluation_llm.generate(
+            evaluation_prompt,
+            max_tokens=10,
+        )
+
+        is_correct = evaluation_result.strip().upper() == "TRUE"
+
+        result = {
+            "query_id": test_case["query_id"],
+            "query": query,
+            "rag_answer_length": len(rag_answer),
+            "ground_truth_length": len(ground_truth),
+            "is_correct": is_correct,
+            "evaluation_result": evaluation_result.strip(),
+        }
+        results_summary.append(result)
+
+        print(f"  Evaluation: {evaluation_result.strip()}")
+        print(f"  Status: {'✓ CORRECT' if is_correct else '✗ INCORRECT'}")
+
+        # Assert answer correctness
+        assert is_correct, (
+            f"Answer mismatch for query: {query}\n\n"
+            f"Generated Answer:\n{rag_answer}\n\n"
+            f"Ground Truth:\n{ground_truth}\n\n"
+            f"Evaluation: {evaluation_result.strip()}"
+        )
+
+    # Print summary
+    print(f"\n{'=' * 80}")
+    print("Answer Correctness Summary:")
+    print(f"  Total queries: {len(results_summary)}")
+    print(f"  Correct: {sum(r['is_correct'] for r in results_summary)}")
+    print(f"  Incorrect: {sum(not r['is_correct'] for r in results_summary)}")
+    accuracy = sum(r["is_correct"] for r in results_summary) / len(results_summary)
+    print(f"  Accuracy: {accuracy:.2%}")
+    print(f"{'=' * 80}")
+
+
+@pytest.mark.integration
+async def test_answer_contains_sources(mcp_sampling_client, nfcorpus_test_data):
+    """Test that RAG answers include source citations.
+
+    This is a basic quality check - we verify that the nc_semantic_search_answer
+    tool returns both a generated answer and source documents.
+    """
+    for test_case in nfcorpus_test_data:
+        query = test_case["query_text"]
+
+        # Execute RAG pipeline
+        rag_result = await mcp_sampling_client.call_tool(
+            "nc_semantic_search_answer",
+            arguments={"query": query, "limit": 5},
+        )
+
+        # Check response structure
+        assert "generated_answer" in rag_result, "Response missing 'generated_answer'"
+        assert "sources" in rag_result, "Response missing 'sources'"
+
+        # Check sources are provided
+        sources = rag_result["sources"]
+        assert len(sources) > 0, f"No sources returned for query: {query}"
+
+        # Check each source has required fields
+        for i, source in enumerate(sources):
+            assert "document_id" in source or "id" in source, (
+                f"Source {i} missing document ID"
+            )
+            assert "excerpt" in source or "content" in source or "text" in source, (
+                f"Source {i} missing content"
+            )
+
+        print(f"Query: {query}")
+        print(f"  Sources provided: {len(sources)}")
+        print("  Status: ✓ PASS")

tests/rag_evaluation/test_retrieval_quality.py

@@ -0,0 +1,143 @@
+"""Tests for RAG retrieval quality (Context Recall metric).
+
+These tests evaluate whether the vector sync/embedding pipeline successfully
+retrieves documents containing the answer to a query.
+
+Metric: Context Recall
+- Measures: Did we retrieve documents containing the answer?
+- Method: Heuristic - Check if ground-truth document IDs appear in top-k results
+- Target: ≥80% recall (at least 80% of expected docs in top-10 results)
+"""
+
+import pytest
+
+
+@pytest.mark.integration
+async def test_retrieval_context_recall(nc_client, nfcorpus_test_data):
+    """Test that semantic search retrieves documents containing the answer.
+
+    For each test query:
+    1. Perform semantic search (retrieval only, no generation)
+    2. Extract retrieved document IDs from top-k results
+    3. Calculate Context Recall: intersection of retrieved and expected docs
+    4. Assert recall meets threshold (≥80%)
+
+    This tests the quality of the vector sync/embedding pipeline.
+    """
+    # Top-k documents to retrieve
+    k = 10
+
+    # Minimum acceptable recall
+    min_recall = 0.8
+
+    results_summary = []
+
+    for test_case in nfcorpus_test_data:
+        query = test_case["query_text"]
+        expected_note_ids = set(test_case["expected_note_ids"])
+
+        # Perform semantic search (retrieval only)
+        search_results = await nc_client.notes.semantic_search(
+            query=query,
+            limit=k,
+        )
+
+        # Extract retrieved note IDs
+        retrieved_note_ids = {result["id"] for result in search_results}
+
+        # Calculate Context Recall
+        intersection = expected_note_ids & retrieved_note_ids
+        recall = len(intersection) / len(expected_note_ids) if expected_note_ids else 0
+
+        # Store results
+        result = {
+            "query_id": test_case["query_id"],
+            "query": query,
+            "expected_count": len(expected_note_ids),
+            "retrieved_count": len(retrieved_note_ids),
+            "intersection_count": len(intersection),
+            "recall": recall,
+            "passed": recall >= min_recall,
+        }
+        results_summary.append(result)
+
+        # Print detailed result for this query
+        print(f"\n{'=' * 80}")
+        print(f"Query: {query}")
+        print(f"  Expected docs: {len(expected_note_ids)}")
+        print(f"  Retrieved (top-{k}): {len(retrieved_note_ids)}")
+        print(f"  Intersection: {len(intersection)}")
+        print(f"  Context Recall: {recall:.2%}")
+        print(f"  Status: {'✓ PASS' if result['passed'] else '✗ FAIL'}")
+
+        # Assert recall meets threshold
+        assert recall >= min_recall, (
+            f"Context Recall {recall:.2%} below threshold {min_recall:.2%} "
+            f"for query: {query}\n"
+            f"Expected {len(expected_note_ids)} docs, found {len(intersection)} in top-{k}"
+        )
+
+    # Print summary
+    print(f"\n{'=' * 80}")
+    print("Context Recall Summary:")
+    print(f"  Total queries: {len(results_summary)}")
+    print(f"  Passed: {sum(r['passed'] for r in results_summary)}")
+    print(f"  Failed: {sum(not r['passed'] for r in results_summary)}")
+    print(
+        f"  Average recall: {sum(r['recall'] for r in results_summary) / len(results_summary):.2%}"
+    )
+    print(f"{'=' * 80}")
+
+
+@pytest.mark.integration
+async def test_retrieval_top1_precision(nc_client, nfcorpus_test_data):
+    """Test that the top-1 retrieved document is highly relevant.
+
+    This is a stricter test than context recall - we verify that
+    the single most relevant document (rank 1) is in the expected set.
+
+    This tests whether the ranking is good, not just retrieval.
+    """
+    results_summary = []
+
+    for test_case in nfcorpus_test_data:
+        query = test_case["query_text"]
+        expected_note_ids = set(test_case["expected_note_ids"])
+
+        # Perform semantic search
+        search_results = await nc_client.notes.semantic_search(
+            query=query,
+            limit=1,  # Only top-1
+        )
+
+        # Check if top result is in expected set
+        if search_results:
+            top_result_id = search_results[0]["id"]
+            is_relevant = top_result_id in expected_note_ids
+        else:
+            is_relevant = False
+
+        result = {
+            "query_id": test_case["query_id"],
+            "query": query,
+            "top_result_id": search_results[0]["id"] if search_results else None,
+            "is_relevant": is_relevant,
+        }
+        results_summary.append(result)
+
+        print(f"\nQuery: {query}")
+        print(f"  Top-1 relevant: {'✓ YES' if is_relevant else '✗ NO'}")
+
+        # This is informational - we don't assert here
+        # Some queries may have multiple valid top results
+
+    # Print summary
+    precision_at_1 = sum(r["is_relevant"] for r in results_summary) / len(
+        results_summary
+    )
+    print(f"\n{'=' * 80}")
+    print(f"Precision@1: {precision_at_1:.2%}")
+    print(
+        f"  ({sum(r['is_relevant'] for r in results_summary)}/{len(results_summary)} queries)"
+    )
+    print(f"{'=' * 80}")