Merge pull request #840 from cbcoutinho/worktree-purrfect-zooming-breeze
feat(search): multi-folder path filter for semantic search
This commit is contained in:
@@ -68,7 +68,7 @@ Filters can only be applied to fields that exist in the Qdrant payload (built in
|
||||
|---|---|---|---|
|
||||
| Modified-date range | `modified_at` | `int` (Unix ts) | ✅ **Ready** — value present on every point; needs a payload index (added in Phase 1, no content re-index) |
|
||||
| Document type | `doc_type` | keyword-indexed `str` | ✅ Implemented |
|
||||
| Directory / path | `file_path` (files only) | `str` | ✅ **Ready (Phase 2)** — value present on file points; TEXT payload index added (no content re-index), filtered with `MatchText` |
|
||||
| Directory / path | `file_path` (files only) | `list[str]` (multi-folder) | ✅ **Implemented (Phase 2)** — TEXT payload index (no content re-index); one or more folders filtered with `MatchText`, multiple OR-ed via nested `Filter(should=...)`; picked from the native folder browser |
|
||||
| Tags | — | — | ❌ **Not indexed** — no `tags` field is written during scanning |
|
||||
| Category (notes) | — | — | ❌ Not in payload — fetched from the Notes API at verify time only |
|
||||
|
||||
@@ -186,19 +186,33 @@ express:
|
||||
it on the answer path. When demand appears it can be threaded through using exactly the same
|
||||
parameter + `parse_modified_timestamp` pattern; doing it now would add an unused parameter and
|
||||
widen the change for no user-visible gain.
|
||||
- **Phase 2 — directory / path (implemented).** Add a `path_prefix` parameter threaded through the
|
||||
same shared contract (`build_base_filter_conditions` → `FieldCondition(key="file_path",
|
||||
match=MatchText(text=path_prefix))`), and a `file_path` **TEXT** payload index in
|
||||
- **Phase 2 — directory / path (implemented).** Threaded through the same shared contract
|
||||
(`build_base_filter_conditions`) and backed by a `file_path` **TEXT** payload index in
|
||||
`_PAYLOAD_INDEX_FIELDS` (no content re-index — `file_path` is already on every file point).
|
||||
`MatchText` semantics differ by backend: **server Qdrant** tokenizes (AND-of-tokens, so
|
||||
`/Projects/Reports` matches files whose path contains both the `Projects` and `Reports` tokens),
|
||||
while **local/embedded qdrant-client** matches by substring containment — both serve folder
|
||||
scoping, neither is a strict left-anchored prefix (a future strict-prefix would need an indexed
|
||||
ancestor-path array, i.e. a re-index, so it stays out of Phase 2). Because `file_path` is only
|
||||
written for `doc_type == "file"`, a non-empty `path_prefix` implicitly restricts results to
|
||||
files; the frontend uses a native path text input enabled only when the **Files** doc type is in
|
||||
scope (rather than `NcFilePicker`, to avoid a hard `@nextcloud/vue` component-version dependency —
|
||||
same rationale as the date inputs). A blank value is treated as "no filter".
|
||||
written for `doc_type == "file"`, a non-empty path filter implicitly restricts results to files.
|
||||
- **Multi-folder (list-valued).** The filter accepts **one or more** folders via a
|
||||
`path_prefixes: list[str]` parameter (the original single `path_prefix` is retained for
|
||||
backward compatibility and folded into the list by `normalize_path_prefixes`, which trims,
|
||||
drops blanks, and de-dupes). `build_base_filter_conditions` adds a single `MatchText` to the
|
||||
`must` clause for one folder, and OR-s multiple folders via a nested `Filter(should=[...])` so a
|
||||
file under **any** selected folder matches while still AND-ing against the ACL/doc_type/date
|
||||
conditions. Every search surface parses the list: the MCP tool (`nc_semantic_search`) takes a
|
||||
real `list[str]`, the visualization API takes a JSON array body, and the viz route takes a
|
||||
**newline-separated** query param. Newline (not comma) is the on-the-wire delimiter because it
|
||||
can't appear in a POSIX path, so folder names are never split mid-value.
|
||||
- **Frontend uses the native folder picker.** Instead of a free-text path input, the Astrolabe
|
||||
app opens Nextcloud's server-side folder browser via `getFilePickerBuilder()` from
|
||||
`@nextcloud/dialogs` (already a dependency — no `@nextcloud/vue` component-version coupling),
|
||||
configured directory-only + multi-select. Picked folders are real, validated server paths
|
||||
(no typos), rendered as removable chips, and sent as a newline-joined `path_prefixes` value. The
|
||||
Astrolabe PHP `ApiController` splits on newline (capping the list to bound the OR-filter width)
|
||||
and `McpServerClient` forwards a JSON array to the MCP server. The control
|
||||
is enabled only when the **Files** doc type is in scope; an empty selection means "no filter".
|
||||
- **Phase 3 — tags (and optionally category).** Add a `tags: list[str]` payload field in
|
||||
`processor.py`, propagate Nextcloud system tags during scanning, trigger a re-index, then wire
|
||||
`NcSelectTags` (`MatchAny` over tags). Re-index cost lives here, isolated from the cheap wins.
|
||||
@@ -214,7 +228,9 @@ The Astrolabe app adds filter controls to the existing collapsible advanced pane
|
||||
specific `@nextcloud/vue` component version (`NcDateTimePicker` / `NcChip` remain a later option);
|
||||
this matches the component's existing native `<input type="range">` controls.
|
||||
- Doc types → existing checkbox grid, now also echoed as chips.
|
||||
- (Phase 2/3) path → `NcFilePicker`; tags → `NcSelectTags :fetch-tags`.
|
||||
- (Phase 2) path → native folder picker (`getFilePickerBuilder` from `@nextcloud/dialogs`),
|
||||
multi-select, rendered as one removable chip per folder. (Phase 3) tags → `NcSelectTags
|
||||
:fetch-tags`.
|
||||
|
||||
Dates cross the wire as **RFC 3339 / ISO 8601 strings** (e.g. `"2026-01-01T00:00:00Z"`) — the format
|
||||
Nextcloud's date pickers and Unified Search use. The MCP/HTTP boundary parses RFC 3339 (and bare
|
||||
|
||||
@@ -31,7 +31,10 @@ from nextcloud_mcp_server.search import (
|
||||
BM25HybridSearchAlgorithm,
|
||||
SemanticSearchAlgorithm,
|
||||
)
|
||||
from nextcloud_mcp_server.search.access_filter import list_accessible_owners
|
||||
from nextcloud_mcp_server.search.access_filter import (
|
||||
list_accessible_owners,
|
||||
normalize_path_prefixes,
|
||||
)
|
||||
from nextcloud_mcp_server.search.context import (
|
||||
get_chunk_bbox_and_page_from_qdrant,
|
||||
get_chunk_with_context,
|
||||
@@ -225,8 +228,17 @@ async def unified_search(request: Request) -> JSONResponse:
|
||||
include_pca = body.get("include_pca", False)
|
||||
include_chunks = body.get("include_chunks", True)
|
||||
doc_types = body.get("doc_types") # Optional filter
|
||||
# ADR-027 Phase 2 path filter (files only); blank ⇒ no filter.
|
||||
path_prefix = (body.get("path_prefix") or "").strip() or None
|
||||
# ADR-027 Phase 2 path filter (files only); blank ⇒ no filter. Accept a
|
||||
# path_prefixes list (multi-folder) alongside the legacy single
|
||||
# path_prefix; normalize drops blanks and de-dupes.
|
||||
# path_prefixes arrives as a JSON array (the Astrolabe PHP client sends
|
||||
# a list); any other shape is ignored rather than guessed at. The legacy
|
||||
# single path_prefix is folded in by normalize_path_prefixes.
|
||||
_path_prefixes_raw = body.get("path_prefixes")
|
||||
path_prefixes = normalize_path_prefixes(
|
||||
body.get("path_prefix"),
|
||||
_path_prefixes_raw if isinstance(_path_prefixes_raw, list) else None,
|
||||
)
|
||||
|
||||
if not query:
|
||||
return JSONResponse({"results": [], "total_found": 0})
|
||||
@@ -268,7 +280,7 @@ async def unified_search(request: Request) -> JSONResponse:
|
||||
accessible_owners=owners,
|
||||
modified_after=modified_after,
|
||||
modified_before=modified_before,
|
||||
path_prefix=path_prefix,
|
||||
path_prefixes=path_prefixes,
|
||||
)
|
||||
)
|
||||
# Sort, then cap to a fixed over-fetch budget before the result
|
||||
@@ -288,7 +300,7 @@ async def unified_search(request: Request) -> JSONResponse:
|
||||
accessible_owners=owners,
|
||||
modified_after=modified_after,
|
||||
modified_before=modified_before,
|
||||
path_prefix=path_prefix,
|
||||
path_prefixes=path_prefixes,
|
||||
)
|
||||
return results
|
||||
|
||||
@@ -440,8 +452,17 @@ async def vector_search(request: Request) -> JSONResponse:
|
||||
limit = min(body.get("limit", 10), 50) # Enforce max limit
|
||||
include_pca = body.get("include_pca", True)
|
||||
doc_types = body.get("doc_types") # Optional list of document types
|
||||
# ADR-027 Phase 2 path filter (files only); blank ⇒ no filter.
|
||||
path_prefix = (body.get("path_prefix") or "").strip() or None
|
||||
# ADR-027 Phase 2 path filter (files only); blank ⇒ no filter. Accept a
|
||||
# path_prefixes list (multi-folder) alongside the legacy single
|
||||
# path_prefix; normalize drops blanks and de-dupes.
|
||||
# path_prefixes arrives as a JSON array (the Astrolabe PHP client sends
|
||||
# a list); any other shape is ignored rather than guessed at. The legacy
|
||||
# single path_prefix is folded in by normalize_path_prefixes.
|
||||
_path_prefixes_raw = body.get("path_prefixes")
|
||||
path_prefixes = normalize_path_prefixes(
|
||||
body.get("path_prefix"),
|
||||
_path_prefixes_raw if isinstance(_path_prefixes_raw, list) else None,
|
||||
)
|
||||
# ADR-027 modified-date range filter. Accepts RFC 3339 / ISO 8601
|
||||
# datetimes or Unix seconds; normalized to int Unix seconds. None ⇒ open.
|
||||
try:
|
||||
@@ -507,7 +528,7 @@ async def vector_search(request: Request) -> JSONResponse:
|
||||
accessible_owners=owners,
|
||||
modified_after=modified_after,
|
||||
modified_before=modified_before,
|
||||
path_prefix=path_prefix,
|
||||
path_prefixes=path_prefixes,
|
||||
)
|
||||
)
|
||||
# Sort merged results by score and limit
|
||||
@@ -522,7 +543,7 @@ async def vector_search(request: Request) -> JSONResponse:
|
||||
accessible_owners=owners,
|
||||
modified_after=modified_after,
|
||||
modified_before=modified_before,
|
||||
path_prefix=path_prefix,
|
||||
path_prefixes=path_prefixes,
|
||||
)
|
||||
return results
|
||||
|
||||
|
||||
@@ -33,7 +33,10 @@ from nextcloud_mcp_server.search import (
|
||||
BM25HybridSearchAlgorithm,
|
||||
SemanticSearchAlgorithm,
|
||||
)
|
||||
from nextcloud_mcp_server.search.access_filter import list_accessible_owners
|
||||
from nextcloud_mcp_server.search.access_filter import (
|
||||
list_accessible_owners,
|
||||
normalize_path_prefixes,
|
||||
)
|
||||
from nextcloud_mcp_server.search.context import (
|
||||
get_chunk_bbox_and_page_from_qdrant,
|
||||
get_chunk_with_context,
|
||||
@@ -144,8 +147,17 @@ async def vector_visualization_search(request: Request) -> JSONResponse:
|
||||
doc_types_param = request.query_params.get("doc_types", "")
|
||||
doc_types = doc_types_param.split(",") if doc_types_param else None
|
||||
|
||||
# ADR-027 Phase 2 path filter (files only); blank ⇒ no filter.
|
||||
path_prefix = (request.query_params.get("path_prefix") or "").strip() or None
|
||||
# ADR-027 Phase 2 path filter (files only); blank ⇒ no filter. Accept a
|
||||
# newline-separated path_prefixes list (multi-folder) plus the legacy single
|
||||
# path_prefix; normalize_path_prefixes drops blanks and de-dupes. Newline is
|
||||
# the delimiter because it can't appear in a POSIX path (unlike a comma), so
|
||||
# folder names are never split mid-value.
|
||||
path_prefix = request.query_params.get("path_prefix")
|
||||
_raw_prefixes = request.query_params.get("path_prefixes")
|
||||
path_prefixes = normalize_path_prefixes(
|
||||
path_prefix,
|
||||
_raw_prefixes.split("\n") if _raw_prefixes else None,
|
||||
)
|
||||
|
||||
# Parse ADR-027 modified-date range filter. Accepts RFC 3339 / ISO 8601
|
||||
# datetimes or Unix seconds; normalized to int Unix seconds. Absent ⇒
|
||||
@@ -235,7 +247,7 @@ async def vector_visualization_search(request: Request) -> JSONResponse:
|
||||
accessible_owners=accessible_owners,
|
||||
modified_after=modified_after,
|
||||
modified_before=modified_before,
|
||||
path_prefix=path_prefix,
|
||||
path_prefixes=path_prefixes,
|
||||
)
|
||||
all_results.extend(unverified_results)
|
||||
else:
|
||||
@@ -258,7 +270,7 @@ async def vector_visualization_search(request: Request) -> JSONResponse:
|
||||
accessible_owners=accessible_owners,
|
||||
modified_after=modified_after,
|
||||
modified_before=modified_before,
|
||||
path_prefix=path_prefix,
|
||||
path_prefixes=path_prefixes,
|
||||
)
|
||||
all_results.extend(unverified_results)
|
||||
# Sort by score, then cap to the same limit*2 over-fetch budget
|
||||
|
||||
@@ -27,6 +27,7 @@ from __future__ import annotations
|
||||
import logging
|
||||
import time
|
||||
from collections import OrderedDict
|
||||
from collections.abc import Iterable
|
||||
from typing import Any, Protocol
|
||||
|
||||
from qdrant_client.models import (
|
||||
@@ -43,6 +44,12 @@ from nextcloud_mcp_server.vector.placeholder import get_placeholder_filter
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Upper bound on the number of folder filters a single search may apply. Caps
|
||||
# the width of the ``Filter(should=[...])`` OR-clause built from path_prefixes
|
||||
# so no caller can degrade query latency with a huge folder list. Mirrored by
|
||||
# the ``Field(max_length=...)`` on the MCP tool and the Astrolabe PHP cap.
|
||||
MAX_PATH_PREFIXES = 20
|
||||
|
||||
# Short-lived per-user cache for the OCS shares lookup, which otherwise runs on
|
||||
# every search/viz request. Trades up to this many seconds of share-visibility
|
||||
# staleness (a freshly-granted share is searchable a little late) for avoiding
|
||||
@@ -189,6 +196,54 @@ def build_ownership_filter(
|
||||
return Filter(should=conditions)
|
||||
|
||||
|
||||
def normalize_path_prefixes(
|
||||
path_prefix: str | None = None,
|
||||
path_prefixes: Iterable[str] | None = None,
|
||||
) -> list[str]:
|
||||
"""Merge the legacy single ``path_prefix`` and list ``path_prefixes`` into
|
||||
one clean, de-duplicated, bounded list of folder filters.
|
||||
|
||||
Blank/whitespace entries are dropped (an empty UI field must mean "no
|
||||
filter", not "match everything"), surrounding whitespace is stripped, and
|
||||
order is preserved while removing duplicates. Accepting both inputs keeps
|
||||
the pre-ADR-027-Phase-2 single-value contract working while callers migrate
|
||||
to the multi-folder list.
|
||||
|
||||
The result is capped at ``MAX_PATH_PREFIXES`` so no caller — the MCP tool,
|
||||
the REST/viz endpoints, or a misbehaving client — can build an unbounded
|
||||
``Filter(should=[...])`` OR-clause that would widen every Qdrant query.
|
||||
This is the single server-side enforcement point (the MCP tool also
|
||||
declares ``Field(max_length=...)`` for an earlier, clearer validation
|
||||
error, and the Astrolabe PHP controller caps before forwarding).
|
||||
|
||||
Args:
|
||||
path_prefix: Legacy single folder filter (deprecated; folded into the
|
||||
returned list).
|
||||
path_prefixes: Zero or more folder filters.
|
||||
|
||||
Returns:
|
||||
Ordered, de-duplicated list of non-empty folder filters, capped at
|
||||
``MAX_PATH_PREFIXES`` (possibly empty).
|
||||
"""
|
||||
raw: list[str] = []
|
||||
if path_prefix:
|
||||
raw.append(path_prefix)
|
||||
if path_prefixes:
|
||||
raw.extend(path_prefixes)
|
||||
|
||||
# Two-pass on purpose: the ``if path_prefix:`` guard above is truthy for a
|
||||
# whitespace-only string like ``" "``, so the strip-and-drop pass below is
|
||||
# what actually removes it — collecting first keeps the dedup order stable.
|
||||
seen: set[str] = set()
|
||||
cleaned: list[str] = []
|
||||
for value in raw:
|
||||
stripped = value.strip()
|
||||
if stripped and stripped not in seen:
|
||||
seen.add(stripped)
|
||||
cleaned.append(stripped)
|
||||
return cleaned[:MAX_PATH_PREFIXES]
|
||||
|
||||
|
||||
def build_base_filter_conditions(
|
||||
user_id: str,
|
||||
accessible_owners: list[str] | None = None,
|
||||
@@ -196,6 +251,7 @@ def build_base_filter_conditions(
|
||||
modified_after: int | None = None,
|
||||
modified_before: int | None = None,
|
||||
path_prefix: str | None = None,
|
||||
path_prefixes: Iterable[str] | None = None,
|
||||
) -> list[Condition]:
|
||||
"""Build the common ``must`` conditions shared by every search algorithm.
|
||||
|
||||
@@ -211,7 +267,10 @@ def build_base_filter_conditions(
|
||||
2. ``build_ownership_filter(...)`` — ACL-aware ``owner_id``/``user_id`` scope.
|
||||
3. ``doc_type`` exact match — only when ``doc_type`` is truthy.
|
||||
4. ``modified_at`` range — only when at least one bound is given.
|
||||
5. ``file_path`` text match — only when ``path_prefix`` is given.
|
||||
5. ``file_path`` text match — only when a path filter is given. One folder
|
||||
adds a single ``MatchText`` to ``must``; multiple folders are OR-ed via a
|
||||
nested ``Filter(should=[...])`` so a file under *any* selected folder
|
||||
matches.
|
||||
|
||||
Args:
|
||||
user_id: Querying user.
|
||||
@@ -220,10 +279,13 @@ def build_base_filter_conditions(
|
||||
doc_type: Optional single document-type filter.
|
||||
modified_after: Inclusive lower bound on ``modified_at`` (Unix seconds).
|
||||
modified_before: Inclusive upper bound on ``modified_at`` (Unix seconds).
|
||||
path_prefix: Optional folder/path filter on the ``file_path`` payload
|
||||
field (ADR-027 Phase 2). Implemented with ``MatchText`` against the
|
||||
text-indexed ``file_path``. ``file_path`` is only written for
|
||||
``doc_type == "file"`` points, so a non-empty ``path_prefix``
|
||||
path_prefix: Deprecated single folder/path filter; folded into
|
||||
``path_prefixes``. Kept for backward compatibility.
|
||||
path_prefixes: Optional folder/path filters on the ``file_path`` payload
|
||||
field (ADR-027 Phase 2). Each is implemented with ``MatchText``
|
||||
against the text-indexed ``file_path`` and multiple folders are
|
||||
OR-ed together. ``file_path`` is only written for
|
||||
``doc_type == "file"`` points, so any non-empty path filter
|
||||
implicitly restricts results to files. NOTE the match semantics
|
||||
differ by backend: server Qdrant tokenizes (AND-of-tokens, so
|
||||
``"/Projects/Reports"`` matches files whose path contains both the
|
||||
@@ -255,9 +317,23 @@ def build_base_filter_conditions(
|
||||
)
|
||||
)
|
||||
|
||||
if path_prefix:
|
||||
# One folder ⇒ a single ``must`` condition (the original Phase 2 shape).
|
||||
# Multiple folders ⇒ OR them in a nested ``Filter(should=...)`` so a file
|
||||
# under any selected folder matches, while still AND-ing against the other
|
||||
# ``must`` conditions (ACL, doc_type, date).
|
||||
folders = normalize_path_prefixes(path_prefix, path_prefixes)
|
||||
if len(folders) == 1:
|
||||
conditions.append(
|
||||
FieldCondition(key="file_path", match=MatchText(text=path_prefix))
|
||||
FieldCondition(key="file_path", match=MatchText(text=folders[0]))
|
||||
)
|
||||
elif len(folders) > 1:
|
||||
conditions.append(
|
||||
Filter(
|
||||
should=[
|
||||
FieldCondition(key="file_path", match=MatchText(text=folder))
|
||||
for folder in folders
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
return conditions
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
import logging
|
||||
from abc import ABC, abstractmethod
|
||||
from collections.abc import Iterable
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Protocol, runtime_checkable
|
||||
|
||||
@@ -300,6 +301,7 @@ class SearchAlgorithm(ABC):
|
||||
modified_after: int | None = None,
|
||||
modified_before: int | None = None,
|
||||
path_prefix: str | None = None,
|
||||
path_prefixes: Iterable[str] | None = None,
|
||||
**kwargs: Any,
|
||||
) -> list[SearchResult]:
|
||||
"""Execute search with the given parameters.
|
||||
@@ -321,10 +323,13 @@ class SearchAlgorithm(ABC):
|
||||
``accessible_owners`` (ADR-027). ``None`` ⇒ open-ended.
|
||||
modified_before: Optional inclusive upper bound on ``modified_at``
|
||||
(Unix seconds, UTC). ``None`` ⇒ open-ended.
|
||||
path_prefix: Optional folder/path filter on the ``file_path`` payload
|
||||
field (ADR-027 Phase 2). Only ``doc_type == "file"`` points carry
|
||||
``file_path``, so a non-empty value implicitly restricts results
|
||||
to files. ``None`` ⇒ no path filter.
|
||||
path_prefix: Deprecated single folder/path filter; folded into
|
||||
``path_prefixes``. Kept for backward compatibility.
|
||||
path_prefixes: Optional folder/path filters on the ``file_path``
|
||||
payload field (ADR-027 Phase 2), OR-ed together. Only
|
||||
``doc_type == "file"`` points carry ``file_path``, so any
|
||||
non-empty value implicitly restricts results to files. ``None``
|
||||
or empty ⇒ no path filter.
|
||||
**kwargs: Algorithm-specific parameters
|
||||
|
||||
Returns:
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
"""BM25 hybrid search algorithm using Qdrant native RRF fusion."""
|
||||
|
||||
import logging
|
||||
from collections.abc import Iterable
|
||||
from typing import Any
|
||||
|
||||
from qdrant_client import models
|
||||
@@ -75,6 +76,7 @@ class BM25HybridSearchAlgorithm(SearchAlgorithm):
|
||||
modified_after: int | None = None,
|
||||
modified_before: int | None = None,
|
||||
path_prefix: str | None = None,
|
||||
path_prefixes: Iterable[str] | None = None,
|
||||
**kwargs: Any,
|
||||
) -> list[SearchResult]:
|
||||
"""
|
||||
@@ -100,8 +102,11 @@ class BM25HybridSearchAlgorithm(SearchAlgorithm):
|
||||
seconds, UTC); ``None`` ⇒ open-ended (ADR-027).
|
||||
modified_before: Inclusive upper bound on ``modified_at`` (Unix
|
||||
seconds, UTC); ``None`` ⇒ open-ended (ADR-027).
|
||||
path_prefix: Folder/path filter on ``file_path`` (files only);
|
||||
``None`` ⇒ no path filter (ADR-027 Phase 2).
|
||||
path_prefix: Deprecated single folder filter; folded into
|
||||
``path_prefixes`` (ADR-027 Phase 2).
|
||||
path_prefixes: Folder/path filters on ``file_path`` (files only),
|
||||
OR-ed together; ``None``/empty ⇒ no path filter (ADR-027
|
||||
Phase 2).
|
||||
**kwargs: Additional parameters (score_threshold override)
|
||||
|
||||
Returns:
|
||||
@@ -152,6 +157,7 @@ class BM25HybridSearchAlgorithm(SearchAlgorithm):
|
||||
modified_after=modified_after,
|
||||
modified_before=modified_before,
|
||||
path_prefix=path_prefix,
|
||||
path_prefixes=path_prefixes,
|
||||
)
|
||||
|
||||
query_filter = Filter(must=filter_conditions)
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
"""Semantic search algorithm using vector similarity (Qdrant)."""
|
||||
|
||||
import logging
|
||||
from collections.abc import Iterable
|
||||
from typing import Any
|
||||
|
||||
from qdrant_client.models import FieldCondition, Filter, MatchAny
|
||||
@@ -55,6 +56,7 @@ class SemanticSearchAlgorithm(SearchAlgorithm):
|
||||
modified_after: int | None = None,
|
||||
modified_before: int | None = None,
|
||||
path_prefix: str | None = None,
|
||||
path_prefixes: Iterable[str] | None = None,
|
||||
**kwargs: Any,
|
||||
) -> list[SearchResult]:
|
||||
"""Execute semantic search using vector similarity.
|
||||
@@ -79,8 +81,11 @@ class SemanticSearchAlgorithm(SearchAlgorithm):
|
||||
seconds, UTC); ``None`` ⇒ open-ended (ADR-027).
|
||||
modified_before: Inclusive upper bound on ``modified_at`` (Unix
|
||||
seconds, UTC); ``None`` ⇒ open-ended (ADR-027).
|
||||
path_prefix: Folder/path filter on ``file_path`` (files only);
|
||||
``None`` ⇒ no path filter (ADR-027 Phase 2).
|
||||
path_prefix: Deprecated single folder filter; folded into
|
||||
``path_prefixes`` (ADR-027 Phase 2).
|
||||
path_prefixes: Folder/path filters on ``file_path`` (files only),
|
||||
OR-ed together; ``None``/empty ⇒ no path filter (ADR-027
|
||||
Phase 2).
|
||||
**kwargs:
|
||||
- score_threshold (float): override the instance default
|
||||
|
||||
@@ -122,6 +127,7 @@ class SemanticSearchAlgorithm(SearchAlgorithm):
|
||||
modified_after=modified_after,
|
||||
modified_before=modified_before,
|
||||
path_prefix=path_prefix,
|
||||
path_prefixes=path_prefixes,
|
||||
)
|
||||
|
||||
# ACL pre-filter (design §11), opt-in via ACL_PREFILTER_ENABLED and OFF
|
||||
|
||||
@@ -32,7 +32,11 @@ from nextcloud_mcp_server.models.semantic import (
|
||||
from nextcloud_mcp_server.observability.metrics import (
|
||||
instrument_tool,
|
||||
)
|
||||
from nextcloud_mcp_server.search.access_filter import list_accessible_owners
|
||||
from nextcloud_mcp_server.search.access_filter import (
|
||||
MAX_PATH_PREFIXES,
|
||||
list_accessible_owners,
|
||||
normalize_path_prefixes,
|
||||
)
|
||||
from nextcloud_mcp_server.search.bm25_hybrid import BM25HybridSearchAlgorithm
|
||||
from nextcloud_mcp_server.search.context import get_chunk_with_context
|
||||
from nextcloud_mcp_server.search.verification import verify_search_results
|
||||
@@ -88,6 +92,7 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
str | None,
|
||||
Field(
|
||||
description=(
|
||||
"Deprecated single-folder filter; prefer path_prefixes. "
|
||||
"Restrict to files under this folder/path "
|
||||
"(e.g. '/Projects/Reports'). Matches the file_path of "
|
||||
"indexed files only, so setting it implicitly limits "
|
||||
@@ -95,6 +100,20 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
),
|
||||
),
|
||||
] = None,
|
||||
path_prefixes: Annotated[
|
||||
list[str] | None,
|
||||
Field(
|
||||
max_length=MAX_PATH_PREFIXES,
|
||||
description=(
|
||||
"Restrict to files under any of these folders/paths "
|
||||
"(e.g. ['/Projects/Reports', '/Shared/Specs']). Folders are "
|
||||
"OR-ed together. Matches the file_path of indexed files "
|
||||
"only, so setting it implicitly limits results to files. "
|
||||
f"Capped at {MAX_PATH_PREFIXES} folders to bound the "
|
||||
"OR-filter width. None or empty = no path filter."
|
||||
),
|
||||
),
|
||||
] = None,
|
||||
) -> SemanticSearchResponse:
|
||||
"""
|
||||
Search Nextcloud content using BM25 hybrid search with cross-app support.
|
||||
@@ -127,9 +146,12 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
modified_before: Only return documents whose last-modified time is at or before this
|
||||
instant. Same formats as modified_after. None = no upper bound (default). Must be
|
||||
>= modified_after when both are supplied.
|
||||
path_prefix: Restrict to files under this folder/path (e.g. "/Projects/Reports").
|
||||
Matches the file_path of indexed files only — setting it implicitly limits results
|
||||
to files. None = no path filter (default).
|
||||
path_prefix: Deprecated single-folder filter; prefer path_prefixes. Restrict to files
|
||||
under this folder/path (e.g. "/Projects/Reports"). Folded into path_prefixes.
|
||||
path_prefixes: Restrict to files under any of these folders/paths (OR-ed), e.g.
|
||||
["/Projects/Reports", "/Shared/Specs"]. Matches the file_path of indexed files
|
||||
only — setting it implicitly limits results to files. None/empty = no path filter
|
||||
(default).
|
||||
|
||||
Returns:
|
||||
SemanticSearchResponse with matching documents ranked by fusion scores.
|
||||
@@ -200,11 +222,10 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
)
|
||||
)
|
||||
|
||||
# Treat a blank/whitespace path_prefix as "no filter" so an empty UI
|
||||
# field doesn't filter out every result (ADR-027 Phase 2).
|
||||
path_prefix = path_prefix.strip() if path_prefix else None
|
||||
if not path_prefix:
|
||||
path_prefix = None
|
||||
# Merge the legacy single path_prefix and the path_prefixes list into one
|
||||
# cleaned list, dropping blank/whitespace entries so an empty UI field
|
||||
# doesn't filter out every result (ADR-027 Phase 2).
|
||||
folder_prefixes = normalize_path_prefixes(path_prefix, path_prefixes)
|
||||
|
||||
# Expand the caller's identity to every owner whose content they
|
||||
# have read access to via Nextcloud shares. Lets a user find files
|
||||
@@ -252,7 +273,7 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
accessible_owners=accessible_owners,
|
||||
modified_after=modified_after_ts,
|
||||
modified_before=modified_before_ts,
|
||||
path_prefix=path_prefix,
|
||||
path_prefixes=folder_prefixes,
|
||||
)
|
||||
all_results.extend(unverified_results)
|
||||
else:
|
||||
@@ -280,7 +301,7 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
accessible_owners=accessible_owners,
|
||||
modified_after=modified_after_ts,
|
||||
modified_before=modified_before_ts,
|
||||
path_prefix=path_prefix,
|
||||
path_prefixes=folder_prefixes,
|
||||
)
|
||||
all_results.extend(unverified_results)
|
||||
|
||||
|
||||
@@ -5,14 +5,16 @@ from __future__ import annotations
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
from qdrant_client.models import FieldCondition, MatchText, Range
|
||||
from qdrant_client.models import FieldCondition, Filter, MatchText, Range
|
||||
|
||||
from nextcloud_mcp_server.search import access_filter
|
||||
from nextcloud_mcp_server.search.access_filter import (
|
||||
MAX_PATH_PREFIXES,
|
||||
build_base_filter_conditions,
|
||||
build_ownership_filter,
|
||||
clear_accessible_owners_cache,
|
||||
list_accessible_owners,
|
||||
normalize_path_prefixes,
|
||||
)
|
||||
|
||||
|
||||
@@ -264,11 +266,90 @@ class TestBuildBaseFilterConditions:
|
||||
assert match.text == prefix
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_no_path_condition_when_prefix_absent(self) -> None:
|
||||
conditions = build_base_filter_conditions("alice", None, path_prefix=None)
|
||||
@pytest.mark.parametrize(
|
||||
"kwargs",
|
||||
[
|
||||
{"path_prefix": None},
|
||||
{"path_prefixes": None},
|
||||
{"path_prefixes": []},
|
||||
{"path_prefix": " ", "path_prefixes": ["", " "]},
|
||||
],
|
||||
)
|
||||
def test_no_path_condition_when_prefix_absent(self, kwargs) -> None:
|
||||
# No folder filter (None, empty list, or blank-only) must add neither a
|
||||
# flat file_path condition nor a nested path OR.
|
||||
conditions = build_base_filter_conditions("alice", None, **kwargs)
|
||||
assert not any(
|
||||
isinstance(c, FieldCondition) and c.key == "file_path" for c in conditions
|
||||
)
|
||||
assert self._path_should_texts(conditions) is None
|
||||
|
||||
@staticmethod
|
||||
def _path_should_texts(conditions) -> set[str] | None:
|
||||
"""Return the file_path texts from the nested OR Filter, or None if no
|
||||
such filter is present. Ignores the ownership Filter (which ORs
|
||||
user_id/owner_id, not file_path)."""
|
||||
for cond in conditions:
|
||||
if not isinstance(cond, Filter) or not cond.should:
|
||||
continue
|
||||
if all(
|
||||
isinstance(c, FieldCondition) and c.key == "file_path"
|
||||
for c in cond.should
|
||||
):
|
||||
return {
|
||||
c.match.text
|
||||
for c in cond.should
|
||||
if isinstance(c, FieldCondition) and isinstance(c.match, MatchText)
|
||||
}
|
||||
return None
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_multiple_path_prefixes_or_in_nested_should(self) -> None:
|
||||
# 3+ folders must OR together: a single nested Filter(should=[...]) is
|
||||
# appended (not separate must conditions, which would AND and match
|
||||
# nothing). 3 folders also guards the list comprehension against an
|
||||
# off-by-one.
|
||||
conditions = build_base_filter_conditions(
|
||||
"alice", None, path_prefixes=["/Projects", "/Archive", "/Shared"]
|
||||
)
|
||||
assert self._path_should_texts(conditions) == {
|
||||
"/Projects",
|
||||
"/Archive",
|
||||
"/Shared",
|
||||
}
|
||||
# No bare file_path FieldCondition in must for the multi-folder case.
|
||||
assert not any(
|
||||
isinstance(c, FieldCondition) and c.key == "file_path" for c in conditions
|
||||
)
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_path_prefix_and_path_prefixes_merge_and_dedupe(self) -> None:
|
||||
# Legacy single + list inputs merge; duplicates collapse so a folder
|
||||
# passed both ways yields two distinct conditions, not three.
|
||||
conditions = build_base_filter_conditions(
|
||||
"alice",
|
||||
None,
|
||||
path_prefix="/Projects",
|
||||
path_prefixes=["/Projects", "/Archive"],
|
||||
)
|
||||
assert self._path_should_texts(conditions) == {"/Projects", "/Archive"}
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_single_effective_prefix_uses_flat_must_condition(self) -> None:
|
||||
# When dedupe/blank-stripping leaves exactly one folder, keep the
|
||||
# original flat MatchText in must rather than a one-element should.
|
||||
conditions = build_base_filter_conditions(
|
||||
"alice", None, path_prefixes=["/Projects", " ", "/Projects"]
|
||||
)
|
||||
# The only nested Filter should be ownership, never a path OR.
|
||||
assert self._path_should_texts(conditions) is None
|
||||
path_conds = [
|
||||
c
|
||||
for c in conditions
|
||||
if isinstance(c, FieldCondition) and c.key == "file_path"
|
||||
]
|
||||
assert len(path_conds) == 1
|
||||
assert path_conds[0].match.text == "/Projects"
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_all_filters_compose(self) -> None:
|
||||
@@ -282,3 +363,27 @@ class TestBuildBaseFilterConditions:
|
||||
path_prefix="/Projects",
|
||||
)
|
||||
assert len(conditions) == 5
|
||||
|
||||
|
||||
class TestNormalizePathPrefixes:
|
||||
@pytest.mark.unit
|
||||
def test_empty_inputs_return_empty_list(self) -> None:
|
||||
assert normalize_path_prefixes(None, None) == []
|
||||
assert normalize_path_prefixes("", []) == []
|
||||
assert normalize_path_prefixes(" ", ["", " "]) == []
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_strips_dedupes_and_preserves_order(self) -> None:
|
||||
result = normalize_path_prefixes(
|
||||
" /Projects ", ["/Archive", "/Projects", " ", "/Specs"]
|
||||
)
|
||||
assert result == ["/Projects", "/Archive", "/Specs"]
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_caps_at_max_path_prefixes(self) -> None:
|
||||
# A huge list is truncated to MAX_PATH_PREFIXES so no caller can build
|
||||
# an unbounded OR-clause; the first N (order-preserving) survive.
|
||||
folders = [f"/dir{i}" for i in range(MAX_PATH_PREFIXES + 30)]
|
||||
result = normalize_path_prefixes(None, folders)
|
||||
assert len(result) == MAX_PATH_PREFIXES
|
||||
assert result == folders[:MAX_PATH_PREFIXES]
|
||||
|
||||
Reference in New Issue
Block a user