feat(search): ADR-027 Phase 1 — modified-date range filter
Add a modified_after/modified_before date-range filter to semantic search, honoured on both the MCP tool path (BM25HybridSearchAlgorithm) and the dense-only visualization/API path (SemanticSearchAlgorithm) through one shared contract. - Promote modified_after/modified_before to explicit keyword params on the SearchAlgorithm ABC and both concrete algorithms; factor the shared placeholder+ownership+doc_type+date filter into access_filter.build_base_filter_conditions so new filters land in one place. - nc_semantic_search: accept RFC 3339 / ISO 8601 (or Unix seconds) bounds via utils.validation.parse_modified_timestamp; Annotated/Field constraints on the numeric args; explicit McpError guard for after > before. Thread the parsed bounds through the cross-app and per-doc_type dispatch. - /api/v1 search endpoints + viz route parse the same formats and 400 on bad or inverted ranges. - Add a modified_at INTEGER payload index to _PAYLOAD_INDEX_FIELDS; the idempotent _ensure_payload_indexes() startup path migrates existing collections with no content re-index. - Update ADR-027 to resolve the review feedback (validation placement, shared algorithm contract, deferral of nc_semantic_search_answer, payload index, RFC-3339-at-the-boundary rationale). Add unit tests. Refs ADR-027. Deck #177. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
f6ab04b2d9
commit
c2c8dc1a08
@@ -1,6 +1,6 @@
|
||||
# ADR-027: Rich Search Filters for Semantic Search
|
||||
|
||||
**Status**: Proposed
|
||||
**Status**: Accepted (Phase 1 implemented; Phases 2–3 deferred)
|
||||
**Date**: 2026-06-02
|
||||
**Depends On**: ADR-012 (Unified Multi-Algorithm Search), ADR-014 (BM25 Search), ADR-019 (Verify-on-Read for Semantic Search)
|
||||
**Tracking**: Astrolabe Cloud POC Deck card #177
|
||||
@@ -66,16 +66,33 @@ Filters can only be applied to fields that exist in the Qdrant payload (built in
|
||||
|
||||
| Desired filter | Payload field | Type | Status |
|
||||
|---|---|---|---|
|
||||
| Modified-date range | `modified_at` | `int` (Unix ts) | ✅ **Ready** — numeric, range-filterable today |
|
||||
| Modified-date range | `modified_at` | `int` (Unix ts) | ✅ **Ready** — value present on every point; needs a payload index (added in Phase 1, no content re-index) |
|
||||
| Document type | `doc_type` | keyword-indexed `str` | ✅ Implemented |
|
||||
| Directory / path | `file_path` (files only) | `str` | ⚠️ Stored but **not keyword-indexed** — prefix/match needs a payload index |
|
||||
| Tags | — | — | ❌ **Not indexed** — no `tags` field is written during scanning |
|
||||
| Category (notes) | — | — | ❌ Not in payload — fetched from the Notes API at verify time only |
|
||||
|
||||
`modified_at` is a deliberate cross-app normalization the scanner already performs: the Notes API
|
||||
and Deck return int Unix seconds natively, News timestamps are unit-normalized to seconds, and
|
||||
WebDAV's RFC-1123 `getlastmodified` *string* is parsed to `int(dt.timestamp())`
|
||||
(`client/webdav.py`). One `int` field therefore covers every doc type.
|
||||
|
||||
Two consequences:
|
||||
|
||||
- **`modified_at` is the cheap win.** It is already a numeric Unix timestamp on every point, so a
|
||||
`Range` condition works against the existing index with no re-index.
|
||||
- **`modified_at` is the cheap win — but it still needs a payload *index*.** The value is on every
|
||||
point, so **no content re-index** is required. However, Qdrant only evaluates a `Range`
|
||||
efficiently against an *indexed* field; without an index every dated query full-scans the
|
||||
collection (and 400s on Qdrant Cloud strict payload-validation mode). Phase 1 therefore adds a
|
||||
single `modified_at: INTEGER` entry to `_PAYLOAD_INDEX_FIELDS`
|
||||
(`vector/qdrant_client.py`); the existing idempotent `_ensure_payload_indexes()` startup path
|
||||
creates it on new **and** existing collections with no operator action. `INTEGER` (not `FLOAT`)
|
||||
because the stored value is an int Unix-second timestamp.
|
||||
- **Why not a Qdrant `datetime` index?** Qdrant's `datetime` index / `DatetimeRange` would let us
|
||||
filter with RFC 3339 strings directly, but its indexer only ingests *string* payload values
|
||||
(`value.as_str()` in Qdrant's `numeric_index/value_indexer.rs`) — it silently skips integer
|
||||
payloads. Adopting it would mean re-storing `modified_at` as RFC 3339 strings on every point,
|
||||
i.e. the full re-index Phase 1 is designed to avoid. We instead keep int storage + a numeric
|
||||
`Range` and accept RFC 3339 only at the request boundary (see §3).
|
||||
- **Tags / path / category are not free.** `file_path` filtering needs a Qdrant payload index
|
||||
before `MatchText`/prefix matching is performant; `tags` and `category` are not in the payload at
|
||||
all and require extending `processor.py` plus a full re-index. Conflating these with the date
|
||||
@@ -83,40 +100,97 @@ Two consequences:
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. Generalise the filter contract
|
||||
### 1. Generalise the filter contract through one shared helper
|
||||
|
||||
Every structured filter follows the `doc_type` path: **tool parameter → `search()` keyword arg →
|
||||
`FieldCondition` appended to `filter_conditions` → `Filter(must=[...])` on both prefetch branches.**
|
||||
Filters are always applied at the Qdrant layer, **before** verify-on-read (ADR-019), so that
|
||||
`verified_chunk_count` / `dropped_document_count` describe the already-filtered set and the verifier
|
||||
never wastes Nextcloud round-trips on documents the filter excluded.
|
||||
Every structured filter follows the `doc_type` path: **tool parameter → explicit `search()`
|
||||
keyword arg → `FieldCondition` in the shared `filter_conditions` builder → `Filter(must=[...])`
|
||||
on both prefetch branches.** Filters are always applied at the Qdrant layer, **before**
|
||||
verify-on-read (ADR-019), so that `verified_chunk_count` / `dropped_document_count` describe the
|
||||
already-filtered set and the verifier never wastes Nextcloud round-trips on documents the filter
|
||||
excluded.
|
||||
|
||||
Date/range bounds use `qdrant_client.models.Range`:
|
||||
**The filter is added to the `SearchAlgorithm` ABC contract, not just one algorithm.** The two
|
||||
algorithms that back the search surfaces — `BM25HybridSearchAlgorithm` (the MCP tool path,
|
||||
`nc_semantic_search`) and `SemanticSearchAlgorithm` (the dense-only visualization / `/api/v1`
|
||||
path, `api/visualization.py` + `auth/viz_routes.py`) — today build a *byte-identical*
|
||||
placeholder + ownership + `doc_type` filter block. Rather than copy the new condition into both,
|
||||
the common block moves into one helper, `search/access_filter.py::build_base_filter_conditions`,
|
||||
and both algorithms call it. New filters are therefore honoured on **every** path (hybrid and
|
||||
dense-only) by editing one function:
|
||||
|
||||
```python
|
||||
from qdrant_client.models import FieldCondition, Range
|
||||
|
||||
if modified_after is not None or modified_before is not None:
|
||||
filter_conditions.append(
|
||||
# search/access_filter.py — the single source of the filter contract
|
||||
def build_base_filter_conditions(
|
||||
user_id, accessible_owners=None, doc_type=None,
|
||||
modified_after=None, modified_before=None,
|
||||
) -> list[Condition]:
|
||||
conditions = [get_placeholder_filter(), build_ownership_filter(user_id, accessible_owners)]
|
||||
if doc_type:
|
||||
conditions.append(FieldCondition(key="doc_type", match=MatchValue(value=doc_type)))
|
||||
if modified_after is not None or modified_before is not None:
|
||||
conditions.append(
|
||||
FieldCondition(
|
||||
key="modified_at",
|
||||
range=Range(gte=modified_after, lte=modified_before), # None bounds are open-ended
|
||||
)
|
||||
)
|
||||
return conditions
|
||||
```
|
||||
|
||||
`Range` treats `None` bounds as open, so the same condition serves after-only, before-only, and
|
||||
both-bounds queries. Validation that `modified_after <= modified_before` lives in the Pydantic
|
||||
request model, not the algorithm.
|
||||
both-bounds queries. New range/match filters are added here once; each algorithm wraps the
|
||||
returned list in `Filter(must=...)` and may append its own additive conditions afterward (e.g.
|
||||
`SemanticSearchAlgorithm`'s opt-in ACL pre-filter, which `BM25HybridSearchAlgorithm` does **not**
|
||||
apply).
|
||||
|
||||
`modified_after` / `modified_before` are promoted to **explicit named keyword params** on
|
||||
`SearchAlgorithm.search()` (and both concrete impls), exactly as `accessible_owners` was — not
|
||||
left in `**kwargs`. Explicit params keep them discoverable and make a misspelled keyword a type
|
||||
error instead of a silently-ignored filter. When `doc_types` is a list, the tool's per-type
|
||||
dispatch loop forwards the same `modified_after`/`modified_before` to each per-type
|
||||
`search()` call.
|
||||
|
||||
**Input validation.** FastMCP exposes no request-model object to hang a Pydantic
|
||||
`@model_validator` on, so validation is split across three layers, each handling what it can
|
||||
express:
|
||||
|
||||
- **Per-argument scalar bounds** use `Annotated[..., Field(...)]` on the tool signature — FastMCP
|
||||
builds the input schema from these and rejects bad values before the body runs. This is the
|
||||
repo's first use of `Annotated`/`Field` on a tool; the existing numeric knobs (`limit` `ge=1`,
|
||||
`score_threshold` `0.0–1.0`, `context_chars` `ge=0`) are tightened the same way and the pattern
|
||||
is what future filters reuse.
|
||||
- **Format normalization for the date bounds.** `modified_after` / `modified_before` are typed
|
||||
`str | int | None` and accept an **RFC 3339 / ISO 8601 datetime** (e.g. `"2026-01-01T00:00:00Z"`,
|
||||
naive ⇒ UTC) *or* Unix seconds. A shared `utils/validation.py::parse_modified_timestamp` helper
|
||||
normalizes either to int Unix seconds (the payload representation) and raises `ValueError` on an
|
||||
unparseable value; the tool converts that to `McpError`, the HTTP endpoints to a 400. The same
|
||||
helper is reused by `nc_semantic_search`, the `/api/v1` search endpoints, and the viz route so
|
||||
every surface accepts identical formats.
|
||||
- **The cross-field invariant `modified_after <= modified_before`** can't be a per-field
|
||||
constraint, so it is an explicit guard (on the *parsed* int values) that raises
|
||||
`McpError(ErrorData(code=-1, ...))` in the tool — the established input-error idiom (mirrors the
|
||||
`VECTOR_SYNC_ENABLED` guard) — and a 400 on the HTTP endpoints.
|
||||
|
||||
### 2. Phase the rollout by payload readiness
|
||||
|
||||
- **Phase 1 — modified-date range (this ADR's committed scope).** Add `modified_after` /
|
||||
`modified_before` (Unix seconds, UTC) to `nc_semantic_search` and `bm25_hybrid.search()`. No
|
||||
re-index. Ship the frontend chip UX against this plus the existing doc-type filter to prove the
|
||||
end-to-end plumbing on fields that already exist.
|
||||
`modified_before` (RFC 3339 / ISO 8601 at the boundary, Unix seconds accepted too) to the
|
||||
`SearchAlgorithm` contract and both algorithm impls (so the MCP tool *and* the dense-only
|
||||
`/api/v1` path honour it), plus a `modified_at` INTEGER payload index. No content re-index. Ship
|
||||
the frontend chip UX against this plus the existing doc-type filter to prove the end-to-end
|
||||
plumbing on fields that already exist.
|
||||
- **`nc_semantic_search_answer` is explicitly deferred, not part of Phase 1.** It is a thin RAG
|
||||
wrapper that today threads only `query`/`limit`/`score_threshold`/`fusion`/context options to
|
||||
`nc_semantic_search` and does not even expose `doc_types` — it always searches cross-app. Date
|
||||
scoping is a search-box affordance, not an answer-synthesis one, and there is no UI surface for
|
||||
it on the answer path. When demand appears it can be threaded through using exactly the same
|
||||
parameter + `parse_modified_timestamp` pattern; doing it now would add an unused parameter and
|
||||
widen the change for no user-visible gain.
|
||||
- **Phase 2 — directory / path.** Create a Qdrant payload index on `file_path`, add a `path_prefix`
|
||||
parameter, and add an `NcFilePicker` folder chooser. Scoped to `doc_type == "file"`.
|
||||
parameter, and add an `NcFilePicker` folder chooser. Scoped to `doc_type == "file"`: the frontend
|
||||
must **hide/disable the path filter unless the file doc type is selected**, because `file_path`
|
||||
is only written to the payload for `doc_type == "file"` (`processor.py`) — a path condition on
|
||||
any other type matches nothing and silently returns zero results.
|
||||
- **Phase 3 — tags (and optionally category).** Add a `tags: list[str]` payload field in
|
||||
`processor.py`, propagate Nextcloud system tags during scanning, trigger a re-index, then wire
|
||||
`NcSelectTags` (`MatchAny` over tags). Re-index cost lives here, isolated from the cheap wins.
|
||||
@@ -124,17 +198,32 @@ request model, not the algorithm.
|
||||
### 3. Frontend: filter chips, structured payload
|
||||
|
||||
The Astrolabe app adds filter controls to the existing collapsible advanced panel and renders each
|
||||
**active** filter as a closable `NcChip` (the same component Nextcloud Unified Search uses):
|
||||
**active** filter as a closable chip:
|
||||
|
||||
- Modified-date range → `NcDateTimePicker type="datetime-range"` (model is `[Date, Date]`).
|
||||
- Modified-date range → two native `<input type="datetime-local">` fields (a local wall-clock
|
||||
picker the browser validates), serialized to RFC 3339 UTC via `Date.toISOString()`. Phase 1
|
||||
deliberately uses native inputs + lightweight CSS chips rather than taking a hard dependency on a
|
||||
specific `@nextcloud/vue` component version (`NcDateTimePicker` / `NcChip` remain a later option);
|
||||
this matches the component's existing native `<input type="range">` controls.
|
||||
- Doc types → existing checkbox grid, now also echoed as chips.
|
||||
- (Phase 2/3) path → `NcFilePicker`; tags → `NcSelectTags :fetch-tags`.
|
||||
|
||||
The `/apps/astrolabe/api/search` endpoint moves from `GET` with query params to **`POST` with a JSON
|
||||
body**, because the filter set is structured and multi-valued and will keep growing. Dates are sent
|
||||
as **Unix seconds (UTC)** to match the `modified_at` payload representation exactly — no timezone or
|
||||
string-parsing ambiguity crosses the wire. Empty or partially-filled filters are omitted from the
|
||||
body rather than sent as nulls.
|
||||
Dates cross the wire as **RFC 3339 / ISO 8601 strings** (e.g. `"2026-01-01T00:00:00Z"`) — the format
|
||||
Nextcloud's date pickers and Unified Search use. The MCP/HTTP boundary parses RFC 3339 (and bare
|
||||
Unix seconds, for resilience) to int Unix seconds via `parse_modified_timestamp` before filtering,
|
||||
so the friendly wire format never forces a re-index of the integer `modified_at` payload.
|
||||
|
||||
**Transport.** Phase 1 keeps the existing `GET /apps/astrolabe/api/search`: the two scalar date
|
||||
strings ride as query params alongside the already comma-joined `doc_types`, so no transport change
|
||||
is needed yet. The move to **`POST` with a JSON body** is deferred to Phase 2, when the filter set
|
||||
becomes genuinely structured/multi-valued (path + tags) and outgrows query params.
|
||||
|
||||
The **Astrolabe UI validates the date range client-side** — the native picker constrains each field
|
||||
to a real datetime, and `performSearch` rejects an *after > before* range before issuing the request
|
||||
(`McpServerClient` forwards the RFC 3339 strings; `ApiController` re-validates and returns a 400 on a
|
||||
bad/inverted range). The server-side `parse_modified_timestamp` + cross-field guard is the
|
||||
authoritative backstop. (Astrolabe-side work is tracked on Deck card **#177**, label
|
||||
`repo:astrolabe`.)
|
||||
|
||||
## Consequences
|
||||
|
||||
@@ -150,8 +239,9 @@ body rather than sent as nulls.
|
||||
|
||||
- Path and tag filters require index work (a payload index; a new payload field + full re-index)
|
||||
that this ADR explicitly defers — the readiness table makes that cost visible rather than implicit.
|
||||
- Moving `/api/search` to POST is a breaking change to that endpoint's contract; the PHP app and the
|
||||
MCP backend must ship together.
|
||||
- The eventual `/api/search` GET→POST migration (Phase 2) will be a breaking change to that
|
||||
endpoint's contract; the PHP app and the MCP backend must ship together when it lands. Phase 1
|
||||
avoids it by keeping GET.
|
||||
- Pre-fusion filtering on a very selective `Range` can still under-fill `limit` if the candidate
|
||||
pool (`limit * 2`) is exhausted; if this proves a problem we revisit the prefetch multiplier
|
||||
rather than filtering post-fusion.
|
||||
|
||||
@@ -37,7 +37,10 @@ from nextcloud_mcp_server.search.context import (
|
||||
get_chunk_with_context,
|
||||
)
|
||||
from nextcloud_mcp_server.search.verification import verify_search_results
|
||||
from nextcloud_mcp_server.utils.validation import is_valid_nextcloud_doc_id
|
||||
from nextcloud_mcp_server.utils.validation import (
|
||||
is_valid_nextcloud_doc_id,
|
||||
parse_modified_timestamp,
|
||||
)
|
||||
from nextcloud_mcp_server.vector.oauth_sync import (
|
||||
NotProvisionedError,
|
||||
get_user_client_basic_auth,
|
||||
@@ -198,6 +201,22 @@ async def unified_search(request: Request) -> JSONResponse:
|
||||
1.0,
|
||||
"score_threshold",
|
||||
)
|
||||
|
||||
# ADR-027 modified-date range filter. Accepts RFC 3339 / ISO 8601
|
||||
# datetimes or Unix seconds; normalized to int Unix seconds for the
|
||||
# numeric Range filter. Absent bound ⇒ open-ended.
|
||||
modified_after = parse_modified_timestamp(
|
||||
body.get("modified_after"), param_name="modified_after"
|
||||
)
|
||||
modified_before = parse_modified_timestamp(
|
||||
body.get("modified_before"), param_name="modified_before"
|
||||
)
|
||||
if (
|
||||
modified_after is not None
|
||||
and modified_before is not None
|
||||
and modified_after > modified_before
|
||||
):
|
||||
raise ValueError("modified_after must be <= modified_before")
|
||||
except ValueError as e:
|
||||
return JSONResponse({"error": str(e)}, status_code=400)
|
||||
|
||||
@@ -245,6 +264,8 @@ async def unified_search(request: Request) -> JSONResponse:
|
||||
limit=search_limit,
|
||||
doc_type=doc_type,
|
||||
accessible_owners=owners,
|
||||
modified_after=modified_after,
|
||||
modified_before=modified_before,
|
||||
)
|
||||
)
|
||||
# Sort, then cap to a fixed over-fetch budget before the result
|
||||
@@ -262,6 +283,8 @@ async def unified_search(request: Request) -> JSONResponse:
|
||||
user_id=user_id,
|
||||
limit=search_limit,
|
||||
accessible_owners=owners,
|
||||
modified_after=modified_after,
|
||||
modified_before=modified_before,
|
||||
)
|
||||
return results
|
||||
|
||||
@@ -413,6 +436,17 @@ async def vector_search(request: Request) -> JSONResponse:
|
||||
limit = min(body.get("limit", 10), 50) # Enforce max limit
|
||||
include_pca = body.get("include_pca", True)
|
||||
doc_types = body.get("doc_types") # Optional list of document types
|
||||
# ADR-027 modified-date range filter. Accepts RFC 3339 / ISO 8601
|
||||
# datetimes or Unix seconds; normalized to int Unix seconds. None ⇒ open.
|
||||
try:
|
||||
modified_after = parse_modified_timestamp(
|
||||
body.get("modified_after"), param_name="modified_after"
|
||||
)
|
||||
modified_before = parse_modified_timestamp(
|
||||
body.get("modified_before"), param_name="modified_before"
|
||||
)
|
||||
except ValueError as e:
|
||||
return JSONResponse({"error": str(e)}, status_code=400)
|
||||
|
||||
if not query:
|
||||
return JSONResponse(
|
||||
@@ -420,6 +454,16 @@ async def vector_search(request: Request) -> JSONResponse:
|
||||
status_code=400,
|
||||
)
|
||||
|
||||
if (
|
||||
modified_after is not None
|
||||
and modified_before is not None
|
||||
and modified_after > modified_before
|
||||
):
|
||||
return JSONResponse(
|
||||
{"error": "modified_after must be <= modified_before"},
|
||||
status_code=400,
|
||||
)
|
||||
|
||||
# Validate algorithm
|
||||
valid_algorithms = {"semantic", "bm25", "hybrid"}
|
||||
if algorithm not in valid_algorithms:
|
||||
@@ -455,6 +499,8 @@ async def vector_search(request: Request) -> JSONResponse:
|
||||
limit=limit,
|
||||
doc_type=doc_type,
|
||||
accessible_owners=owners,
|
||||
modified_after=modified_after,
|
||||
modified_before=modified_before,
|
||||
)
|
||||
)
|
||||
# Sort merged results by score and limit
|
||||
@@ -467,6 +513,8 @@ async def vector_search(request: Request) -> JSONResponse:
|
||||
user_id=user_id,
|
||||
limit=limit,
|
||||
accessible_owners=owners,
|
||||
modified_after=modified_after,
|
||||
modified_before=modified_before,
|
||||
)
|
||||
return results
|
||||
|
||||
|
||||
@@ -39,7 +39,10 @@ from nextcloud_mcp_server.search.context import (
|
||||
get_chunk_with_context,
|
||||
)
|
||||
from nextcloud_mcp_server.search.verification import verify_search_results
|
||||
from nextcloud_mcp_server.utils.validation import is_valid_nextcloud_doc_id
|
||||
from nextcloud_mcp_server.utils.validation import (
|
||||
is_valid_nextcloud_doc_id,
|
||||
parse_modified_timestamp,
|
||||
)
|
||||
from nextcloud_mcp_server.vector.oauth_sync import (
|
||||
NotProvisionedError,
|
||||
get_user_client_basic_auth,
|
||||
@@ -141,6 +144,31 @@ async def vector_visualization_search(request: Request) -> JSONResponse:
|
||||
doc_types_param = request.query_params.get("doc_types", "")
|
||||
doc_types = doc_types_param.split(",") if doc_types_param else None
|
||||
|
||||
# Parse ADR-027 modified-date range filter. Accepts RFC 3339 / ISO 8601
|
||||
# datetimes or Unix seconds; normalized to int Unix seconds. Absent ⇒
|
||||
# open-ended. Unparseable input or an inverted range returns 400.
|
||||
try:
|
||||
modified_after = parse_modified_timestamp(
|
||||
request.query_params.get("modified_after"), param_name="modified_after"
|
||||
)
|
||||
modified_before = parse_modified_timestamp(
|
||||
request.query_params.get("modified_before"), param_name="modified_before"
|
||||
)
|
||||
except ValueError as exc:
|
||||
return JSONResponse(
|
||||
{"success": False, "error": str(exc)},
|
||||
status_code=400,
|
||||
)
|
||||
if (
|
||||
modified_after is not None
|
||||
and modified_before is not None
|
||||
and modified_after > modified_before
|
||||
):
|
||||
return JSONResponse(
|
||||
{"success": False, "error": "modified_after must be <= modified_before"},
|
||||
status_code=400,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Viz search: user=%s, query='%s', algorithm=%s, fusion=%s, limit=%s, doc_types=%s",
|
||||
username,
|
||||
@@ -202,6 +230,8 @@ async def vector_visualization_search(request: Request) -> JSONResponse:
|
||||
doc_type=None, # Search all types
|
||||
score_threshold=score_threshold,
|
||||
accessible_owners=accessible_owners,
|
||||
modified_after=modified_after,
|
||||
modified_before=modified_before,
|
||||
)
|
||||
all_results.extend(unverified_results)
|
||||
else:
|
||||
@@ -222,6 +252,8 @@ async def vector_visualization_search(request: Request) -> JSONResponse:
|
||||
doc_type=doc_type,
|
||||
score_threshold=score_threshold,
|
||||
accessible_owners=accessible_owners,
|
||||
modified_after=modified_after,
|
||||
modified_before=modified_before,
|
||||
)
|
||||
all_results.extend(unverified_results)
|
||||
# Sort by score, then cap to the same limit*2 over-fetch budget
|
||||
|
||||
@@ -29,7 +29,16 @@ import time
|
||||
from collections import OrderedDict
|
||||
from typing import Any, Protocol
|
||||
|
||||
from qdrant_client.models import Condition, FieldCondition, Filter, MatchAny, MatchValue
|
||||
from qdrant_client.models import (
|
||||
Condition,
|
||||
FieldCondition,
|
||||
Filter,
|
||||
MatchAny,
|
||||
MatchValue,
|
||||
Range,
|
||||
)
|
||||
|
||||
from nextcloud_mcp_server.vector.placeholder import get_placeholder_filter
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -177,3 +186,60 @@ def build_ownership_filter(
|
||||
0, FieldCondition(key="owner_id", match=MatchAny(any=other_owners))
|
||||
)
|
||||
return Filter(should=conditions)
|
||||
|
||||
|
||||
def build_base_filter_conditions(
|
||||
user_id: str,
|
||||
accessible_owners: list[str] | None = None,
|
||||
doc_type: str | None = None,
|
||||
modified_after: int | None = None,
|
||||
modified_before: int | None = None,
|
||||
) -> list[Condition]:
|
||||
"""Build the common ``must`` conditions shared by every search algorithm.
|
||||
|
||||
This is the single place the structured-filter contract (ADR-027) lives, so
|
||||
both the BM25-hybrid (MCP tool) and dense-only (visualization/API) algorithms
|
||||
apply identical placeholder/ACL/doc_type/date filtering. Each algorithm wraps
|
||||
the returned list in ``Filter(must=...)`` and may append its own additive
|
||||
conditions afterward (e.g. the dense algorithm's opt-in ACL pre-filter).
|
||||
|
||||
The conditions, in order:
|
||||
|
||||
1. ``get_placeholder_filter()`` — exclude in-flight placeholder points.
|
||||
2. ``build_ownership_filter(...)`` — ACL-aware ``owner_id``/``user_id`` scope.
|
||||
3. ``doc_type`` exact match — only when ``doc_type`` is truthy.
|
||||
4. ``modified_at`` range — only when at least one bound is given.
|
||||
|
||||
Args:
|
||||
user_id: Querying user.
|
||||
accessible_owners: Owner UIDs the user can read (see
|
||||
``build_ownership_filter``). ``None`` ⇒ self-only.
|
||||
doc_type: Optional single document-type filter.
|
||||
modified_after: Inclusive lower bound on ``modified_at`` (Unix seconds).
|
||||
modified_before: Inclusive upper bound on ``modified_at`` (Unix seconds).
|
||||
|
||||
Returns:
|
||||
A list of Qdrant ``Condition`` objects for a parent ``must`` clause.
|
||||
"""
|
||||
conditions: list[Condition] = [
|
||||
get_placeholder_filter(),
|
||||
build_ownership_filter(user_id, accessible_owners),
|
||||
]
|
||||
|
||||
if doc_type:
|
||||
conditions.append(
|
||||
FieldCondition(key="doc_type", match=MatchValue(value=doc_type))
|
||||
)
|
||||
|
||||
# ``Range`` treats ``None`` bounds as open-ended, so the same condition serves
|
||||
# after-only, before-only, and both-bounds queries. Appended only when at
|
||||
# least one bound is set so unfiltered searches add no condition.
|
||||
if modified_after is not None or modified_before is not None:
|
||||
conditions.append(
|
||||
FieldCondition(
|
||||
key="modified_at",
|
||||
range=Range(gte=modified_after, lte=modified_before),
|
||||
)
|
||||
)
|
||||
|
||||
return conditions
|
||||
|
||||
@@ -289,6 +289,8 @@ class SearchAlgorithm(ABC):
|
||||
doc_type: str | None = None,
|
||||
*,
|
||||
accessible_owners: list[str] | None = None,
|
||||
modified_after: int | None = None,
|
||||
modified_before: int | None = None,
|
||||
**kwargs: Any,
|
||||
) -> list[SearchResult]:
|
||||
"""Execute search with the given parameters.
|
||||
@@ -304,6 +306,12 @@ class SearchAlgorithm(ABC):
|
||||
buried in ``**kwargs`` — so a misspelled keyword is a type error
|
||||
instead of a silent fall back to self-only scope. ``None`` means
|
||||
self-only (``[user_id]``).
|
||||
modified_after: Optional inclusive lower bound on the document's
|
||||
``modified_at`` payload field (Unix seconds, UTC). Declared
|
||||
explicitly for the same discoverability/type-safety reason as
|
||||
``accessible_owners`` (ADR-027). ``None`` ⇒ open-ended.
|
||||
modified_before: Optional inclusive upper bound on ``modified_at``
|
||||
(Unix seconds, UTC). ``None`` ⇒ open-ended.
|
||||
**kwargs: Algorithm-specific parameters
|
||||
|
||||
Returns:
|
||||
|
||||
@@ -4,19 +4,18 @@ import logging
|
||||
from typing import Any
|
||||
|
||||
from qdrant_client import models
|
||||
from qdrant_client.models import FieldCondition, Filter, MatchValue
|
||||
from qdrant_client.models import Filter
|
||||
|
||||
from nextcloud_mcp_server.config import get_settings
|
||||
from nextcloud_mcp_server.embedding import get_bm25_service, get_embedding_service
|
||||
from nextcloud_mcp_server.observability.metrics import record_qdrant_operation
|
||||
from nextcloud_mcp_server.observability.tracing import trace_operation
|
||||
from nextcloud_mcp_server.search.access_filter import build_ownership_filter
|
||||
from nextcloud_mcp_server.search.access_filter import build_base_filter_conditions
|
||||
from nextcloud_mcp_server.search.algorithms import (
|
||||
SearchAlgorithm,
|
||||
SearchResult,
|
||||
build_search_result_from_point,
|
||||
)
|
||||
from nextcloud_mcp_server.vector.placeholder import get_placeholder_filter
|
||||
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -73,6 +72,8 @@ class BM25HybridSearchAlgorithm(SearchAlgorithm):
|
||||
doc_type: str | None = None,
|
||||
*,
|
||||
accessible_owners: list[str] | None = None,
|
||||
modified_after: int | None = None,
|
||||
modified_before: int | None = None,
|
||||
**kwargs: Any,
|
||||
) -> list[SearchResult]:
|
||||
"""
|
||||
@@ -94,6 +95,10 @@ class BM25HybridSearchAlgorithm(SearchAlgorithm):
|
||||
accessible_owners: Owner UIDs the user can read (self + share
|
||||
senders), pre-computed by the caller from the OCS Sharing API.
|
||||
Defaults to ``[user_id]`` (self-only) when ``None``.
|
||||
modified_after: Inclusive lower bound on ``modified_at`` (Unix
|
||||
seconds, UTC); ``None`` ⇒ open-ended (ADR-027).
|
||||
modified_before: Inclusive upper bound on ``modified_at`` (Unix
|
||||
seconds, UTC); ``None`` ⇒ open-ended (ADR-027).
|
||||
**kwargs: Additional parameters (score_threshold override)
|
||||
|
||||
Returns:
|
||||
@@ -134,19 +139,15 @@ class BM25HybridSearchAlgorithm(SearchAlgorithm):
|
||||
len(sparse_embedding["indices"]),
|
||||
)
|
||||
|
||||
# Build Qdrant filter
|
||||
filter_conditions = [
|
||||
get_placeholder_filter(), # Always exclude placeholders from user-facing queries
|
||||
build_ownership_filter(user_id, accessible_owners),
|
||||
]
|
||||
|
||||
# Add doc_type filter if specified
|
||||
if doc_type:
|
||||
filter_conditions.append(
|
||||
FieldCondition(
|
||||
key="doc_type",
|
||||
match=MatchValue(value=doc_type),
|
||||
)
|
||||
# Build Qdrant filter (placeholder + ACL + doc_type + modified_at range).
|
||||
# Shared with the dense-only SemanticSearchAlgorithm via the common
|
||||
# ADR-027 helper so every search surface applies one filter contract.
|
||||
filter_conditions = build_base_filter_conditions(
|
||||
user_id=user_id,
|
||||
accessible_owners=accessible_owners,
|
||||
doc_type=doc_type,
|
||||
modified_after=modified_after,
|
||||
modified_before=modified_before,
|
||||
)
|
||||
|
||||
query_filter = Filter(must=filter_conditions)
|
||||
|
||||
@@ -3,20 +3,19 @@
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from qdrant_client.models import FieldCondition, Filter, MatchAny, MatchValue
|
||||
from qdrant_client.models import FieldCondition, Filter, MatchAny
|
||||
|
||||
from nextcloud_mcp_server.acl_hash import accessible_hash_set
|
||||
from nextcloud_mcp_server.config import get_settings
|
||||
from nextcloud_mcp_server.embedding import get_embedding_service
|
||||
from nextcloud_mcp_server.observability.metrics import record_qdrant_operation
|
||||
from nextcloud_mcp_server.search.access_filter import build_ownership_filter
|
||||
from nextcloud_mcp_server.search.access_filter import build_base_filter_conditions
|
||||
from nextcloud_mcp_server.search.algorithms import (
|
||||
SearchAlgorithm,
|
||||
SearchResult,
|
||||
build_search_result_from_point,
|
||||
)
|
||||
from nextcloud_mcp_server.vector.payload_keys import ACL_HASH
|
||||
from nextcloud_mcp_server.vector.placeholder import get_placeholder_filter
|
||||
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -53,6 +52,8 @@ class SemanticSearchAlgorithm(SearchAlgorithm):
|
||||
doc_type: str | None = None,
|
||||
*,
|
||||
accessible_owners: list[str] | None = None,
|
||||
modified_after: int | None = None,
|
||||
modified_before: int | None = None,
|
||||
**kwargs: Any,
|
||||
) -> list[SearchResult]:
|
||||
"""Execute semantic search using vector similarity.
|
||||
@@ -73,6 +74,10 @@ class SemanticSearchAlgorithm(SearchAlgorithm):
|
||||
accessible_owners: Owner UIDs the user can read (self + share
|
||||
senders), pre-computed by the caller from the OCS Sharing API.
|
||||
Defaults to ``[user_id]`` (self-only) when ``None``.
|
||||
modified_after: Inclusive lower bound on ``modified_at`` (Unix
|
||||
seconds, UTC); ``None`` ⇒ open-ended (ADR-027).
|
||||
modified_before: Inclusive upper bound on ``modified_at`` (Unix
|
||||
seconds, UTC); ``None`` ⇒ open-ended (ADR-027).
|
||||
**kwargs:
|
||||
- score_threshold (float): override the instance default
|
||||
|
||||
@@ -103,19 +108,16 @@ class SemanticSearchAlgorithm(SearchAlgorithm):
|
||||
"Generated embedding for query (dimension=%s)", len(query_embedding)
|
||||
)
|
||||
|
||||
# Build Qdrant filter
|
||||
filter_conditions = [
|
||||
get_placeholder_filter(), # Always exclude placeholders from user-facing queries
|
||||
build_ownership_filter(user_id, accessible_owners),
|
||||
]
|
||||
|
||||
# Add doc_type filter if specified
|
||||
if doc_type:
|
||||
filter_conditions.append(
|
||||
FieldCondition(
|
||||
key="doc_type",
|
||||
match=MatchValue(value=doc_type),
|
||||
)
|
||||
# Build Qdrant filter (placeholder + ACL + doc_type + modified_at range).
|
||||
# Shared with BM25HybridSearchAlgorithm via the common ADR-027 helper so
|
||||
# the dense-only (API/visualization) and hybrid (MCP tool) paths apply
|
||||
# one filter contract.
|
||||
filter_conditions = build_base_filter_conditions(
|
||||
user_id=user_id,
|
||||
accessible_owners=accessible_owners,
|
||||
doc_type=doc_type,
|
||||
modified_after=modified_after,
|
||||
modified_before=modified_before,
|
||||
)
|
||||
|
||||
# ACL pre-filter (design §11), opt-in via ACL_PREFILTER_ENABLED and OFF
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
"""Semantic search MCP tools using vector database."""
|
||||
|
||||
import logging
|
||||
from typing import Annotated
|
||||
|
||||
import anyio
|
||||
from httpx import RequestError
|
||||
@@ -16,6 +17,7 @@ from mcp.types import (
|
||||
TextContent,
|
||||
ToolAnnotations,
|
||||
)
|
||||
from pydantic import Field
|
||||
from qdrant_client.models import Filter
|
||||
|
||||
from nextcloud_mcp_server.auth import require_scopes
|
||||
@@ -34,6 +36,7 @@ from nextcloud_mcp_server.search.access_filter import list_accessible_owners
|
||||
from nextcloud_mcp_server.search.bm25_hybrid import BM25HybridSearchAlgorithm
|
||||
from nextcloud_mcp_server.search.context import get_chunk_with_context
|
||||
from nextcloud_mcp_server.search.verification import verify_search_results
|
||||
from nextcloud_mcp_server.utils.validation import parse_modified_timestamp
|
||||
from nextcloud_mcp_server.vector.placeholder import get_placeholder_filter
|
||||
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
|
||||
|
||||
@@ -55,12 +58,32 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
async def nc_semantic_search(
|
||||
query: str,
|
||||
ctx: Context,
|
||||
limit: int = 10,
|
||||
limit: Annotated[int, Field(ge=1, le=100)] = 10,
|
||||
doc_types: list[str] | None = None,
|
||||
score_threshold: float = 0.0,
|
||||
score_threshold: Annotated[float, Field(ge=0.0, le=1.0)] = 0.0,
|
||||
fusion: str = "rrf",
|
||||
include_context: bool = False,
|
||||
context_chars: int = 300,
|
||||
context_chars: Annotated[int, Field(ge=0)] = 300,
|
||||
modified_after: Annotated[
|
||||
str | int | None,
|
||||
Field(
|
||||
description=(
|
||||
"Only return documents modified at or after this time. "
|
||||
"RFC 3339 / ISO 8601 datetime (e.g. '2026-01-01T00:00:00Z') "
|
||||
"or Unix seconds. None = no lower bound."
|
||||
),
|
||||
),
|
||||
] = None,
|
||||
modified_before: Annotated[
|
||||
str | int | None,
|
||||
Field(
|
||||
description=(
|
||||
"Only return documents modified at or before this time. "
|
||||
"RFC 3339 / ISO 8601 datetime or Unix seconds. "
|
||||
"None = no upper bound."
|
||||
),
|
||||
),
|
||||
] = None,
|
||||
) -> SemanticSearchResponse:
|
||||
"""
|
||||
Search Nextcloud content using BM25 hybrid search with cross-app support.
|
||||
@@ -86,6 +109,13 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
DBSF: Uses distribution-based normalization, may better balance different score ranges
|
||||
include_context: Whether to expand results with surrounding context (default: False)
|
||||
context_chars: Number of characters to include before/after matched chunk (default: 300)
|
||||
modified_after: Only return documents whose last-modified time is at or after this
|
||||
instant. Accepts an RFC 3339 / ISO 8601 datetime (e.g. "2026-01-01T00:00:00Z";
|
||||
a naive datetime is treated as UTC) or Unix seconds. None = no lower bound
|
||||
(default).
|
||||
modified_before: Only return documents whose last-modified time is at or before this
|
||||
instant. Same formats as modified_after. None = no upper bound (default). Must be
|
||||
>= modified_after when both are supplied.
|
||||
|
||||
Returns:
|
||||
SemanticSearchResponse with matching documents ranked by fusion scores.
|
||||
@@ -122,6 +152,40 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
)
|
||||
)
|
||||
|
||||
# Normalize the RFC 3339 / Unix-seconds date bounds to int Unix seconds
|
||||
# for the numeric ``modified_at`` Range filter (ADR-027). A bad format
|
||||
# surfaces as a clean McpError rather than a 500.
|
||||
try:
|
||||
modified_after_ts = parse_modified_timestamp(
|
||||
modified_after, param_name="modified_after"
|
||||
)
|
||||
modified_before_ts = parse_modified_timestamp(
|
||||
modified_before, param_name="modified_before"
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise McpError(ErrorData(code=-1, message=str(exc))) from exc
|
||||
|
||||
# Cross-field invariant: a per-parameter pydantic ``Field`` constraint
|
||||
# (validated by FastMCP from the signature) bounds each date on its own
|
||||
# but cannot express the relationship between them. Guard it here so an
|
||||
# inverted range surfaces a clean McpError rather than silently
|
||||
# returning zero results (ADR-027).
|
||||
if (
|
||||
modified_after_ts is not None
|
||||
and modified_before_ts is not None
|
||||
and modified_after_ts > modified_before_ts
|
||||
):
|
||||
raise McpError(
|
||||
ErrorData(
|
||||
code=-1,
|
||||
message=(
|
||||
"modified_after must be <= modified_before "
|
||||
f"(got modified_after={modified_after!r}, "
|
||||
f"modified_before={modified_before!r})"
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
# Expand the caller's identity to every owner whose content they
|
||||
# have read access to via Nextcloud shares. Lets a user find files
|
||||
# owners have shared with them without having to re-index those
|
||||
@@ -166,6 +230,8 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
doc_type=None, # Signal to search all types
|
||||
score_threshold=score_threshold,
|
||||
accessible_owners=accessible_owners,
|
||||
modified_after=modified_after_ts,
|
||||
modified_before=modified_before_ts,
|
||||
)
|
||||
all_results.extend(unverified_results)
|
||||
else:
|
||||
@@ -191,6 +257,8 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
doc_type=dtype,
|
||||
score_threshold=score_threshold,
|
||||
accessible_owners=accessible_owners,
|
||||
modified_after=modified_after_ts,
|
||||
modified_before=modified_before_ts,
|
||||
)
|
||||
all_results.extend(unverified_results)
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
"""Shared validators for primitive types crossing system boundaries."""
|
||||
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
|
||||
# Nextcloud object IDs are unsigned ints from MySQL AUTO_INCREMENT, which
|
||||
# starts at 1. Restrict to ASCII positive integers to exclude Unicode digit
|
||||
@@ -9,7 +10,77 @@ import re
|
||||
# and leading zeros.
|
||||
_NEXTCLOUD_DOC_ID_RE = re.compile(r"^[1-9][0-9]*$")
|
||||
|
||||
# A bare non-negative integer string is accepted as Unix seconds (the
|
||||
# pre-RFC-3339 wire format) so older callers keep working.
|
||||
_UNIX_SECONDS_RE = re.compile(r"^[0-9]+$")
|
||||
|
||||
|
||||
def is_valid_nextcloud_doc_id(value: str) -> bool:
|
||||
"""True iff `value` is the str form of a positive ASCII integer (>= 1)."""
|
||||
return bool(_NEXTCLOUD_DOC_ID_RE.fullmatch(value))
|
||||
|
||||
|
||||
def parse_modified_timestamp(
|
||||
value: str | int | float | None,
|
||||
*,
|
||||
param_name: str = "modified_at",
|
||||
) -> int | None:
|
||||
"""Normalize a search date-filter bound to an int Unix-second timestamp.
|
||||
|
||||
ADR-027: callers (the MCP tool, the ``/api/v1`` search endpoints, and the
|
||||
visualization route) accept **RFC 3339 / ISO 8601** datetimes at the
|
||||
boundary — the ergonomic, Nextcloud-Unified-Search-style format — while the
|
||||
``modified_at`` Qdrant payload stays an int Unix-second timestamp (a
|
||||
cross-app normalization done by the scanner). This converts the former to
|
||||
the latter so the numeric ``Range`` filter and INTEGER payload index work
|
||||
without re-indexing.
|
||||
|
||||
Accepts:
|
||||
|
||||
- ``None`` / empty string ⇒ ``None`` (open-ended bound).
|
||||
- ``int`` / ``float`` ⇒ truncated to int seconds.
|
||||
- A bare non-negative integer string ⇒ Unix seconds (legacy wire format).
|
||||
- An RFC 3339 / ISO 8601 string, e.g. ``"2026-01-01T00:00:00Z"`` or
|
||||
``"2026-01-01T00:00:00+02:00"``. A naive datetime (no offset) is assumed
|
||||
to be UTC, matching the payload representation.
|
||||
|
||||
Args:
|
||||
value: The raw bound from the request.
|
||||
param_name: Field name used in error messages.
|
||||
|
||||
Returns:
|
||||
Int Unix-second timestamp (UTC), or ``None`` for an open bound.
|
||||
|
||||
Raises:
|
||||
ValueError: If the value is negative or cannot be parsed.
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
# bool is an int subclass — reject it explicitly so True/False aren't
|
||||
# silently read as 1/0 seconds.
|
||||
if isinstance(value, bool):
|
||||
raise ValueError(f"{param_name} must be a datetime string or Unix seconds")
|
||||
if isinstance(value, (int, float)):
|
||||
seconds = int(value)
|
||||
if seconds < 0:
|
||||
raise ValueError(f"{param_name} must be >= 0, got {seconds}")
|
||||
return seconds
|
||||
if isinstance(value, str):
|
||||
text = value.strip()
|
||||
if not text:
|
||||
return None
|
||||
if _UNIX_SECONDS_RE.fullmatch(text):
|
||||
return int(text)
|
||||
# RFC 3339 / ISO 8601. ``fromisoformat`` accepts a trailing "Z" only on
|
||||
# Python 3.11+; normalize it for safety across versions.
|
||||
try:
|
||||
parsed = datetime.fromisoformat(text.replace("Z", "+00:00"))
|
||||
except ValueError as exc:
|
||||
raise ValueError(
|
||||
f"{param_name} must be an RFC 3339 datetime or Unix seconds, "
|
||||
f"got {value!r}"
|
||||
) from exc
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=timezone.utc)
|
||||
return int(parsed.timestamp())
|
||||
raise ValueError(f"{param_name} must be a datetime string or Unix seconds")
|
||||
|
||||
@@ -52,6 +52,16 @@ _PAYLOAD_INDEX_FIELDS: dict[str, PayloadSchemaType] = {
|
||||
"chunk_index": PayloadSchemaType.INTEGER,
|
||||
"chunk_start_offset": PayloadSchemaType.INTEGER,
|
||||
"chunk_end_offset": PayloadSchemaType.INTEGER,
|
||||
# modified_at is the ADR-027 date-range filter field: searches apply
|
||||
# Range(key="modified_at", gte=..., lte=...) (see
|
||||
# search/access_filter.build_base_filter_conditions). Qdrant requires a
|
||||
# payload index to evaluate a Range efficiently — without one every dated
|
||||
# query full-scans the collection (and 400s on Qdrant Cloud strict mode).
|
||||
# INTEGER (not FLOAT): modified_at is an int Unix-second timestamp on every
|
||||
# point (vector/processor.py, vector/placeholder.py). _ensure_payload_indexes
|
||||
# is idempotent, so existing collections gain this index at startup with no
|
||||
# content re-index and no operator action.
|
||||
"modified_at": PayloadSchemaType.INTEGER,
|
||||
}
|
||||
|
||||
# Sentinel point that records "this collection has been backfilled to str
|
||||
|
||||
@@ -5,9 +5,11 @@ from __future__ import annotations
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
from qdrant_client.models import FieldCondition, Range
|
||||
|
||||
from nextcloud_mcp_server.search import access_filter
|
||||
from nextcloud_mcp_server.search.access_filter import (
|
||||
build_base_filter_conditions,
|
||||
build_ownership_filter,
|
||||
clear_accessible_owners_cache,
|
||||
list_accessible_owners,
|
||||
@@ -186,3 +188,75 @@ class TestBuildOwnershipFilter:
|
||||
(user_branch,) = flt.should
|
||||
assert user_branch.key == "user_id"
|
||||
assert user_branch.match.value == "alice"
|
||||
|
||||
|
||||
class TestBuildBaseFilterConditions:
|
||||
"""The shared ADR-027 filter contract used by both search algorithms."""
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_minimal_is_placeholder_plus_ownership(self) -> None:
|
||||
# No doc_type, no date bounds -> exactly placeholder + ownership.
|
||||
conditions = build_base_filter_conditions("alice", None)
|
||||
assert len(conditions) == 2
|
||||
# No modified_at Range condition present.
|
||||
assert not any(
|
||||
isinstance(c, FieldCondition) and c.key == "modified_at" for c in conditions
|
||||
)
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_doc_type_appends_match_condition(self) -> None:
|
||||
conditions = build_base_filter_conditions("alice", None, doc_type="note")
|
||||
doc_type_conds = [
|
||||
c
|
||||
for c in conditions
|
||||
if isinstance(c, FieldCondition) and c.key == "doc_type"
|
||||
]
|
||||
assert len(doc_type_conds) == 1
|
||||
assert doc_type_conds[0].match.value == "note"
|
||||
|
||||
@pytest.mark.unit
|
||||
@pytest.mark.parametrize(
|
||||
"after,before,expected_gte,expected_lte",
|
||||
[
|
||||
(100, 200, 100, 200),
|
||||
(100, None, 100, None), # after-only
|
||||
(None, 200, None, 200), # before-only
|
||||
],
|
||||
)
|
||||
def test_modified_at_range_appended(
|
||||
self, after, before, expected_gte, expected_lte
|
||||
) -> None:
|
||||
conditions = build_base_filter_conditions(
|
||||
"alice", None, modified_after=after, modified_before=before
|
||||
)
|
||||
range_conds = [
|
||||
c
|
||||
for c in conditions
|
||||
if isinstance(c, FieldCondition) and c.key == "modified_at"
|
||||
]
|
||||
assert len(range_conds) == 1
|
||||
rng = range_conds[0].range
|
||||
assert isinstance(rng, Range)
|
||||
assert rng.gte == expected_gte
|
||||
assert rng.lte == expected_lte
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_no_range_when_both_bounds_none(self) -> None:
|
||||
conditions = build_base_filter_conditions(
|
||||
"alice", None, modified_after=None, modified_before=None
|
||||
)
|
||||
assert not any(
|
||||
isinstance(c, FieldCondition) and c.key == "modified_at" for c in conditions
|
||||
)
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_all_filters_compose(self) -> None:
|
||||
# placeholder + ownership + doc_type + modified_at range = 4 conditions.
|
||||
conditions = build_base_filter_conditions(
|
||||
"alice",
|
||||
["alice", "bob"],
|
||||
doc_type="file",
|
||||
modified_after=100,
|
||||
modified_before=200,
|
||||
)
|
||||
assert len(conditions) == 4
|
||||
|
||||
@@ -1,8 +1,13 @@
|
||||
"""Unit tests for shared boundary validators."""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
|
||||
from nextcloud_mcp_server.utils.validation import is_valid_nextcloud_doc_id
|
||||
from nextcloud_mcp_server.utils.validation import (
|
||||
is_valid_nextcloud_doc_id,
|
||||
parse_modified_timestamp,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
@@ -52,3 +57,62 @@ def test_accepts_positive_ascii_integers(value):
|
||||
def test_rejects_invalid_doc_ids(value, reason):
|
||||
"""Reject empty/zero/leading-zero/non-ASCII/non-digit inputs."""
|
||||
assert is_valid_nextcloud_doc_id(value) is False, f"should reject: {reason}"
|
||||
|
||||
|
||||
# parse_modified_timestamp (ADR-027) ---------------------------------------
|
||||
|
||||
_JAN_2026_UTC = int(datetime(2026, 1, 1, tzinfo=timezone.utc).timestamp())
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
@pytest.mark.parametrize(
|
||||
"value,expected",
|
||||
[
|
||||
(None, None),
|
||||
("", None),
|
||||
(" ", None),
|
||||
# RFC 3339 / ISO 8601
|
||||
("2026-01-01T00:00:00Z", _JAN_2026_UTC),
|
||||
("2026-01-01T00:00:00+00:00", _JAN_2026_UTC),
|
||||
# +02:00 offset is two hours earlier in UTC
|
||||
("2026-01-01T02:00:00+02:00", _JAN_2026_UTC),
|
||||
# Naive datetime is assumed UTC
|
||||
("2026-01-01T00:00:00", _JAN_2026_UTC),
|
||||
# Date-only ISO form
|
||||
("2026-01-01", _JAN_2026_UTC),
|
||||
# Bare Unix seconds (string and int) pass through
|
||||
(str(_JAN_2026_UTC), _JAN_2026_UTC),
|
||||
(_JAN_2026_UTC, _JAN_2026_UTC),
|
||||
(0, 0),
|
||||
(1767225600.9, 1767225600), # float truncates
|
||||
],
|
||||
)
|
||||
def test_parse_modified_timestamp_accepts(value, expected):
|
||||
"""RFC 3339 strings, Unix seconds, and None normalize to int seconds (UTC)."""
|
||||
assert parse_modified_timestamp(value) == expected
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
@pytest.mark.parametrize(
|
||||
"value,reason",
|
||||
[
|
||||
("not-a-date", "unparseable string"),
|
||||
("2026-13-01T00:00:00Z", "invalid month"),
|
||||
(-1, "negative int"),
|
||||
(-5.0, "negative float"),
|
||||
(True, "bool is not a timestamp"),
|
||||
(False, "bool is not a timestamp"),
|
||||
([], "wrong type"),
|
||||
],
|
||||
)
|
||||
def test_parse_modified_timestamp_rejects(value, reason):
|
||||
"""Bad formats / negatives / bools raise ValueError (→ McpError or HTTP 400)."""
|
||||
with pytest.raises(ValueError):
|
||||
parse_modified_timestamp(value)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_parse_modified_timestamp_error_names_param():
|
||||
"""The param_name is surfaced in the error for caller-friendly messages."""
|
||||
with pytest.raises(ValueError, match="modified_after"):
|
||||
parse_modified_timestamp("nope", param_name="modified_after")
|
||||
|
||||
@@ -73,6 +73,21 @@ def _record(point_id: int | str, doc_id: int | str | None) -> SimpleNamespace:
|
||||
return SimpleNamespace(id=point_id, payload=payload)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _PAYLOAD_INDEX_FIELDS contract
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_modified_at_indexed_as_integer():
|
||||
"""ADR-027: the date-range filter needs a numeric index on modified_at.
|
||||
|
||||
INTEGER (not FLOAT/DATETIME) because the payload stores an int Unix-second
|
||||
timestamp; a numeric Range filters it without a content re-index.
|
||||
"""
|
||||
assert _PAYLOAD_INDEX_FIELDS.get("modified_at") == PayloadSchemaType.INTEGER
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _ensure_payload_indexes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user