fix: paginate tagged-folder SEARCH so the scanner discovers all files
The vector-sync scanner expanded a tagged folder into its PDF descendants via `WebdavClient.find_by_type(scope=dir)` with no result limit. A WebDAV SEARCH with no `<d:nresults>` returns only Nextcloud's default page (~100 on the affected instance), so large tagged folders were silently truncated and most documents were never queued for indexing (e.g. a 220-file folder yielded 100). Add `search_files_all`, which pages the SEARCH to completion. It uses `<d:firstresult>` offset paging where supported and, because Nextcloud 31 ignores offset (verified against a live instance), detects the repeated page and falls back to a single bounded fetch with an explicit large `<d:nresults>`. `find_all_by_type` wraps this and is now used for tagged-folder expansion; `find_by_type` is unchanged for the interactive MCP tools. Crossing `WEBDAV_SEARCH_MAX_RESULTS` logs a warning and increments the new `astrolabe_document_scan_truncated_total` metric, so a coverage cap can never again hide files silently. Scope: this fixes discovery only. Cross-user double-processing of identical shared files (point-ID collisions) is tracked separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
0919513f21
commit
01cb7cf08c
@@ -278,6 +278,18 @@ documents_indexed_total = Counter(
|
||||
["source", "status"], # source: note | file | deck_card | news_item
|
||||
)
|
||||
|
||||
# --- Document discovery / coverage ------------------------------------------
|
||||
#
|
||||
# Fires when a paged WebDAV SEARCH (folder-expansion during a scan) hits the
|
||||
# WEBDAV_SEARCH_MAX_RESULTS ceiling, meaning the discovered file set was capped
|
||||
# and some tagged documents may never be queued for indexing. This is the
|
||||
# alertable signal that prevents the old *silent* 100-result truncation from
|
||||
# recurring. Tenant is the Kubernetes ``namespace`` label, as elsewhere.
|
||||
document_scan_truncated_total = Counter(
|
||||
"astrolabe_document_scan_truncated_total",
|
||||
"Times a folder-expansion SEARCH hit the result ceiling (coverage truncated)",
|
||||
)
|
||||
|
||||
# =============================================================================
|
||||
# Database Metrics
|
||||
# =============================================================================
|
||||
|
||||
Reference in New Issue
Block a user