docs: add ADR-027 for rich search filters
Add ADR-027 describing rich, chip-style filters for Astrolabe semantic search (modified-date range, doc type, path, tags). Generalises the existing doc_type filter contract (tool param -> search() kwarg -> Qdrant FieldCondition applied pre-fusion and pre-verify-on-read) and phases the rollout by payload readiness: date range ships now, path and tags defer behind a payload index / re-index. Tracking: Astrolabe Cloud POC Deck card #177 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
7e4b83dc94
commit
f6ab04b2d9
@@ -0,0 +1,174 @@
|
||||
# ADR-027: Rich Search Filters for Semantic Search
|
||||
|
||||
**Status**: Proposed
|
||||
**Date**: 2026-06-02
|
||||
**Depends On**: ADR-012 (Unified Multi-Algorithm Search), ADR-014 (BM25 Search), ADR-019 (Verify-on-Read for Semantic Search)
|
||||
**Tracking**: Astrolabe Cloud POC Deck card #177
|
||||
|
||||
## Context
|
||||
|
||||
`nc_semantic_search` today exposes one structured filter — `doc_types` — on top of the query
|
||||
string. Everything else a user might want to narrow by (when a document was last modified, which
|
||||
folder it lives in, which tags it carries) is invisible to the search layer. As Astrolabe's corpus
|
||||
grows across Notes, Files (PDFs), Deck cards, and News items, a single relevance ranking over the
|
||||
whole index is increasingly blunt: the user knows "the spec I edited last week, somewhere under
|
||||
/Projects" but can only type words and hope.
|
||||
|
||||
The Astrolabe PHP app surfaces semantic search through a plain `NcTextField` plus a doc-type
|
||||
checkbox grid (`astrolabe/src/App.vue`). There is no visual affordance for any other dimension. We
|
||||
want to add **rich, visually-indicated filters** — modelled on Nextcloud Unified Search's
|
||||
filter-*chip* interaction — and weave them through the search backend without disturbing the
|
||||
existing fusion + verify-on-read pipeline.
|
||||
|
||||
This ADR defines:
|
||||
|
||||
1. The **contract** for how a structured filter travels from the MCP tool signature down to a
|
||||
Qdrant `FieldCondition` (so every future filter follows one pattern).
|
||||
2. The **payload-readiness** of each desired filter, which drives a phased rollout.
|
||||
3. What the **frontend** sends and how it presents active filters.
|
||||
|
||||
### How filtering works today (the pattern to generalise)
|
||||
|
||||
A single filter — `doc_type` — already threads through three layers. New filters mirror it exactly.
|
||||
|
||||
1. **MCP tool signature** — `nextcloud_mcp_server/server/semantic.py` (`nc_semantic_search`) accepts
|
||||
`doc_types: list[str] | None` and dispatches one `search_algo.search(...)` call per type (or one
|
||||
call with `doc_type=None` for cross-app search).
|
||||
2. **Algorithm** — `nextcloud_mcp_server/search/bm25_hybrid.py` `search()` receives `doc_type` and
|
||||
builds the Qdrant filter:
|
||||
|
||||
```python
|
||||
filter_conditions = [
|
||||
get_placeholder_filter(), # exclude pending placeholders
|
||||
build_ownership_filter(user_id, accessible_owners), # ACL
|
||||
]
|
||||
if doc_type:
|
||||
filter_conditions.append(
|
||||
FieldCondition(key="doc_type", match=MatchValue(value=doc_type))
|
||||
)
|
||||
query_filter = Filter(must=filter_conditions)
|
||||
```
|
||||
|
||||
3. **Qdrant query** — `query_filter` is passed to **both** the dense and sparse `Prefetch` branches
|
||||
of the `query_points` call, so the filter applies *before* fusion. Filtering before fusion (not
|
||||
after) keeps the `limit * 2` candidate pools meaningful and avoids returning fewer than `limit`
|
||||
results when a filter is selective.
|
||||
|
||||
`build_ownership_filter` (`search/access_filter.py`) and `get_placeholder_filter`
|
||||
(`vector/placeholder.py`) demonstrate the full matcher vocabulary we will reuse: `MatchValue`
|
||||
(exact), `MatchAny` (OR-list), `Range` (numeric bounds), and `Filter(must=...)` / `Filter(should=...)`
|
||||
for AND / OR composition.
|
||||
|
||||
### Payload readiness governs what we can ship
|
||||
|
||||
Filters can only be applied to fields that exist in the Qdrant payload (built in
|
||||
`nextcloud_mcp_server/vector/processor.py`). Auditing the payload schema:
|
||||
|
||||
| Desired filter | Payload field | Type | Status |
|
||||
|---|---|---|---|
|
||||
| Modified-date range | `modified_at` | `int` (Unix ts) | ✅ **Ready** — numeric, range-filterable today |
|
||||
| Document type | `doc_type` | keyword-indexed `str` | ✅ Implemented |
|
||||
| Directory / path | `file_path` (files only) | `str` | ⚠️ Stored but **not keyword-indexed** — prefix/match needs a payload index |
|
||||
| Tags | — | — | ❌ **Not indexed** — no `tags` field is written during scanning |
|
||||
| Category (notes) | — | — | ❌ Not in payload — fetched from the Notes API at verify time only |
|
||||
|
||||
Two consequences:
|
||||
|
||||
- **`modified_at` is the cheap win.** It is already a numeric Unix timestamp on every point, so a
|
||||
`Range` condition works against the existing index with no re-index.
|
||||
- **Tags / path / category are not free.** `file_path` filtering needs a Qdrant payload index
|
||||
before `MatchText`/prefix matching is performant; `tags` and `category` are not in the payload at
|
||||
all and require extending `processor.py` plus a full re-index. Conflating these with the date
|
||||
filter would make a small UX improvement wait on an expensive indexing migration.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. Generalise the filter contract
|
||||
|
||||
Every structured filter follows the `doc_type` path: **tool parameter → `search()` keyword arg →
|
||||
`FieldCondition` appended to `filter_conditions` → `Filter(must=[...])` on both prefetch branches.**
|
||||
Filters are always applied at the Qdrant layer, **before** verify-on-read (ADR-019), so that
|
||||
`verified_chunk_count` / `dropped_document_count` describe the already-filtered set and the verifier
|
||||
never wastes Nextcloud round-trips on documents the filter excluded.
|
||||
|
||||
Date/range bounds use `qdrant_client.models.Range`:
|
||||
|
||||
```python
|
||||
from qdrant_client.models import FieldCondition, Range
|
||||
|
||||
if modified_after is not None or modified_before is not None:
|
||||
filter_conditions.append(
|
||||
FieldCondition(
|
||||
key="modified_at",
|
||||
range=Range(gte=modified_after, lte=modified_before), # None bounds are open-ended
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
`Range` treats `None` bounds as open, so the same condition serves after-only, before-only, and
|
||||
both-bounds queries. Validation that `modified_after <= modified_before` lives in the Pydantic
|
||||
request model, not the algorithm.
|
||||
|
||||
### 2. Phase the rollout by payload readiness
|
||||
|
||||
- **Phase 1 — modified-date range (this ADR's committed scope).** Add `modified_after` /
|
||||
`modified_before` (Unix seconds, UTC) to `nc_semantic_search` and `bm25_hybrid.search()`. No
|
||||
re-index. Ship the frontend chip UX against this plus the existing doc-type filter to prove the
|
||||
end-to-end plumbing on fields that already exist.
|
||||
- **Phase 2 — directory / path.** Create a Qdrant payload index on `file_path`, add a `path_prefix`
|
||||
parameter, and add an `NcFilePicker` folder chooser. Scoped to `doc_type == "file"`.
|
||||
- **Phase 3 — tags (and optionally category).** Add a `tags: list[str]` payload field in
|
||||
`processor.py`, propagate Nextcloud system tags during scanning, trigger a re-index, then wire
|
||||
`NcSelectTags` (`MatchAny` over tags). Re-index cost lives here, isolated from the cheap wins.
|
||||
|
||||
### 3. Frontend: filter chips, structured payload
|
||||
|
||||
The Astrolabe app adds filter controls to the existing collapsible advanced panel and renders each
|
||||
**active** filter as a closable `NcChip` (the same component Nextcloud Unified Search uses):
|
||||
|
||||
- Modified-date range → `NcDateTimePicker type="datetime-range"` (model is `[Date, Date]`).
|
||||
- Doc types → existing checkbox grid, now also echoed as chips.
|
||||
- (Phase 2/3) path → `NcFilePicker`; tags → `NcSelectTags :fetch-tags`.
|
||||
|
||||
The `/apps/astrolabe/api/search` endpoint moves from `GET` with query params to **`POST` with a JSON
|
||||
body**, because the filter set is structured and multi-valued and will keep growing. Dates are sent
|
||||
as **Unix seconds (UTC)** to match the `modified_at` payload representation exactly — no timezone or
|
||||
string-parsing ambiguity crosses the wire. Empty or partially-filled filters are omitted from the
|
||||
body rather than sent as nulls.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**
|
||||
|
||||
- One filter pattern for the whole search surface; adding a filter is a localized, testable change.
|
||||
- Phase 1 ships immediately with zero re-index risk and proves the UX contract end-to-end.
|
||||
- Filtering before verify-on-read keeps ACL/ghost semantics intact and avoids wasted verification
|
||||
round-trips.
|
||||
- The chip UX matches Nextcloud conventions, so it reads as native to users.
|
||||
|
||||
**Negative / costs**
|
||||
|
||||
- Path and tag filters require index work (a payload index; a new payload field + full re-index)
|
||||
that this ADR explicitly defers — the readiness table makes that cost visible rather than implicit.
|
||||
- Moving `/api/search` to POST is a breaking change to that endpoint's contract; the PHP app and the
|
||||
MCP backend must ship together.
|
||||
- Pre-fusion filtering on a very selective `Range` can still under-fill `limit` if the candidate
|
||||
pool (`limit * 2`) is exhausted; if this proves a problem we revisit the prefetch multiplier
|
||||
rather than filtering post-fusion.
|
||||
|
||||
**Neutral**
|
||||
|
||||
- `SemanticSearchResponse` is unchanged — filters live entirely in the request. MCP clients that
|
||||
ignore the new parameters behave exactly as before (backward compatible).
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
- **Free-text `key:value` query parsing** (`modified:>2026-01-01 path:/Projects`). Powerful but
|
||||
invites injection-shaped ambiguity and a parser to maintain, and gives no visual affordance for
|
||||
"what can I filter by?". Structured params + chips answer the user's discoverability question
|
||||
directly. Could be layered on later as sugar over the same params.
|
||||
- **Post-fusion / client-side filtering** (like the current score-threshold slider). Simple, but
|
||||
defeats the point of a recall layer: the index would return mostly-irrelevant candidates that get
|
||||
thrown away, and `limit` becomes unpredictable. Rejected in favour of pushing filters into Qdrant.
|
||||
- **Indexing everything up front** so all filters ship at once. Forces a large re-index and couples
|
||||
a cheap UX win to an expensive migration. Rejected in favour of the readiness-driven phasing.
|
||||
Reference in New Issue
Block a user