feat: Implement BM25 hybrid search with native Qdrant RRF fusion
Replace custom keyword/fuzzy search algorithms with industry-standard BM25 sparse vectors, combined with dense semantic vectors using Qdrant's native Reciprocal Rank Fusion (RRF). This consolidates search architecture and improves relevance for both semantic and keyword queries. Key changes: - Add fastembed dependency for BM25 sparse vector generation - Update Qdrant collection schema to support named vectors (dense + sparse) - Create BM25SparseEmbeddingProvider using FastEmbed's Qdrant/bm25 model - Implement BM25HybridSearchAlgorithm with native Qdrant RRF prefetch - Update document processor to generate both dense and sparse embeddings - Simplify nc_semantic_search() tool to use BM25 hybrid only - Remove legacy keyword.py, fuzzy.py, and custom hybrid.py (736 lines) - Update ADR-014 with implementation notes and test results Benefits: - Consolidated architecture (single Qdrant database) - Native database-level RRF fusion (more efficient) - Industry-standard BM25 (replaces brittle custom keyword search) - Better relevance across semantic and keyword queries - Simplified codebase (-285 net lines) Tests: All 125 tests passing (118 unit, 7 integration) Implements ADR-014: Replace Custom Keyword Search with BM25 Hybrid Search 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -1,13 +1,11 @@
|
||||
"""Search algorithms module for unified multi-algorithm search.
|
||||
"""Search algorithms module for BM25 hybrid search.
|
||||
|
||||
This module provides a unified interface for different search algorithms:
|
||||
- Semantic search (vector similarity)
|
||||
- Keyword search (token-based matching)
|
||||
- Fuzzy search (character overlap)
|
||||
- Hybrid search (RRF fusion of multiple algorithms)
|
||||
This module provides BM25 hybrid search combining:
|
||||
- Dense semantic vectors (vector similarity via embeddings)
|
||||
- Sparse BM25 vectors (keyword-based retrieval)
|
||||
|
||||
All algorithms share the same interface and can be used interchangeably by both
|
||||
MCP tools and the visualization pane.
|
||||
Results are fused using Qdrant's native Reciprocal Rank Fusion (RRF) for
|
||||
optimal relevance across both semantic and keyword queries.
|
||||
"""
|
||||
|
||||
from nextcloud_mcp_server.search.algorithms import (
|
||||
@@ -16,9 +14,7 @@ from nextcloud_mcp_server.search.algorithms import (
|
||||
SearchResult,
|
||||
get_indexed_doc_types,
|
||||
)
|
||||
from nextcloud_mcp_server.search.fuzzy import FuzzySearchAlgorithm
|
||||
from nextcloud_mcp_server.search.hybrid import HybridSearchAlgorithm
|
||||
from nextcloud_mcp_server.search.keyword import KeywordSearchAlgorithm
|
||||
from nextcloud_mcp_server.search.bm25_hybrid import BM25HybridSearchAlgorithm
|
||||
from nextcloud_mcp_server.search.semantic import SemanticSearchAlgorithm
|
||||
|
||||
__all__ = [
|
||||
@@ -27,7 +23,5 @@ __all__ = [
|
||||
"SearchResult",
|
||||
"get_indexed_doc_types",
|
||||
"SemanticSearchAlgorithm",
|
||||
"KeywordSearchAlgorithm",
|
||||
"FuzzySearchAlgorithm",
|
||||
"HybridSearchAlgorithm",
|
||||
"BM25HybridSearchAlgorithm",
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user