feat(vector): page-aware PDF chunking for predictable per-page retrieval
Add PageAwareChunker, which splits paginated documents (PDFs) on page boundaries first and only character-splits pages larger than chunk_size. No chunk spans a page boundary, so page_number is always exact and stored excerpts never lead with a neighbouring page's text. When chunk_size is at least the largest page, this yields exactly one chunk per page: a predictable vector count (== page count), a flat per-page embedding cost, and zero cross-page overlap duplication. Gated by DOCUMENT_CHUNK_PAGE_AWARE (default true). When false, the legacy char-based DocumentChunker + post-hoc assign_page_numbers path runs unchanged. Only doc_type="file" with page_boundaries (PDFs) takes the page-aware path; notes/deck/news are unaffected. Measured on a 15-page record (query "leadership award louis", target = top-half of page 15): char-based degraded the target to dense-rank 10 at cs=2048 (OCR) and mislabeled its page; page-aware restored rank 1 across every fusion/modality and chunk size, with correct page labels and clean snippets. BREAKING CHANGE: PDFs are re-chunked page-aware by default. Existing deployments will re-index PDF content on the next vector sync (different chunk counts and page_number labels). Set DOCUMENT_CHUNK_PAGE_AWARE=false to retain the previous char-based behaviour. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
40e56aeaea
commit
2f2a7f9659
@@ -206,6 +206,10 @@ NEXTCLOUD_PASSWORD=
|
||||
# Configure how documents are split before embedding
|
||||
#DOCUMENT_CHUNK_SIZE=512
|
||||
#DOCUMENT_CHUNK_OVERLAP=50
|
||||
# Page-aware chunking for PDFs: split on page boundaries first so no chunk spans
|
||||
# a page (exact page numbers, clean snippets, ~1 chunk/page when chunk size >=
|
||||
# the largest page). Set false to use the legacy char-based path. Default: true
|
||||
#DOCUMENT_CHUNK_PAGE_AWARE=true
|
||||
|
||||
# ===== SEMANTIC SEARCH TUNING =====
|
||||
# Advanced parameters for vector sync background operations
|
||||
|
||||
Reference in New Issue
Block a user