feat(vector): page-aware PDF chunking for predictable per-page retrieval

Add PageAwareChunker, which splits paginated documents (PDFs) on page
boundaries first and only character-splits pages larger than chunk_size.
No chunk spans a page boundary, so page_number is always exact and stored
excerpts never lead with a neighbouring page's text. When chunk_size is at
least the largest page, this yields exactly one chunk per page: a
predictable vector count (== page count), a flat per-page embedding cost,
and zero cross-page overlap duplication.

Gated by DOCUMENT_CHUNK_PAGE_AWARE (default true). When false, the legacy
char-based DocumentChunker + post-hoc assign_page_numbers path runs
unchanged. Only doc_type="file" with page_boundaries (PDFs) takes the
page-aware path; notes/deck/news are unaffected.

Measured on a 15-page record (query "leadership award louis", target =
top-half of page 15): char-based degraded the target to dense-rank 10 at
cs=2048 (OCR) and mislabeled its page; page-aware restored rank 1 across
every fusion/modality and chunk size, with correct page labels and clean
snippets.

BREAKING CHANGE: PDFs are re-chunked page-aware by default. Existing
deployments will re-index PDF content on the next vector sync (different
chunk counts and page_number labels). Set DOCUMENT_CHUNK_PAGE_AWARE=false
to retain the previous char-based behaviour.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-06 13:42:25 +02:00
co-authored by Claude Opus 4.8
parent 40e56aeaea
commit 2f2a7f9659
7 changed files with 347 additions and 10 deletions
+4
View File
@@ -206,6 +206,10 @@ NEXTCLOUD_PASSWORD=
# Configure how documents are split before embedding
#DOCUMENT_CHUNK_SIZE=512
#DOCUMENT_CHUNK_OVERLAP=50
# Page-aware chunking for PDFs: split on page boundaries first so no chunk spans
# a page (exact page numbers, clean snippets, ~1 chunk/page when chunk size >=
# the largest page). Set false to use the legacy char-based path. Default: true
#DOCUMENT_CHUNK_PAGE_AWARE=true
# ===== SEMANTIC SEARCH TUNING =====
# Advanced parameters for vector sync background operations