feat(vector): page-aware PDF chunking for predictable per-page retrieval

Add PageAwareChunker, which splits paginated documents (PDFs) on page
boundaries first and only character-splits pages larger than chunk_size.
No chunk spans a page boundary, so page_number is always exact and stored
excerpts never lead with a neighbouring page's text. When chunk_size is at
least the largest page, this yields exactly one chunk per page: a
predictable vector count (== page count), a flat per-page embedding cost,
and zero cross-page overlap duplication.

Gated by DOCUMENT_CHUNK_PAGE_AWARE (default true). When false, the legacy
char-based DocumentChunker + post-hoc assign_page_numbers path runs
unchanged. Only doc_type="file" with page_boundaries (PDFs) takes the
page-aware path; notes/deck/news are unaffected.

Measured on a 15-page record (query "leadership award louis", target =
top-half of page 15): char-based degraded the target to dense-rank 10 at
cs=2048 (OCR) and mislabeled its page; page-aware restored rank 1 across
every fusion/modality and chunk size, with correct page labels and clean
snippets.

BREAKING CHANGE: PDFs are re-chunked page-aware by default. Existing
deployments will re-index PDF content on the next vector sync (different
chunk counts and page_number labels). Set DOCUMENT_CHUNK_PAGE_AWARE=false
to retain the previous char-based behaviour.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

nextcloud_mcp_server/vector/processor.py

@@ -30,7 +30,10 @@ from nextcloud_mcp_server.observability.metrics import (
 from nextcloud_mcp_server.observability.tracing import trace_operation
 from nextcloud_mcp_server.search.pdf_highlighter import PDFHighlighter
 from nextcloud_mcp_server.vector import payload_keys
-from nextcloud_mcp_server.vector.document_chunker import DocumentChunker
+from nextcloud_mcp_server.vector.document_chunker import (
+    DocumentChunker,
+    PageAwareChunker,
+)
 from nextcloud_mcp_server.vector.html_processor import html_to_markdown
 from nextcloud_mcp_server.vector.placeholder import (
     delete_placeholder_point,
@@ -682,27 +685,46 @@ async def _index_document(
                 logger.error("Failed to process file %s: %s", file_path, e)
                 raise
 
-    # Tokenize and chunk (using configured chunk size and overlap)
+    # Tokenize and chunk (using configured chunk size and overlap). Paginated
+    # files (PDFs with page_boundaries) use the page-aware chunker when enabled,
+    # which assigns page numbers inline; everything else uses the char-based
+    # chunker followed by post-hoc page assignment.
+    page_boundaries = file_metadata.get("page_boundaries")
+    use_page_aware = (
+        settings.document_chunk_page_aware
+        and doc_task.doc_type == "file"
+        and page_boundaries is not None
+    )
     with trace_operation(
         "vector_sync.chunk_text",
         attributes={
             "vector_sync.input_chars": len(content),
             "vector_sync.chunk_size": settings.document_chunk_size,
             "vector_sync.overlap": settings.document_chunk_overlap,
+            "vector_sync.page_aware": use_page_aware,
         },
     ) as chunk_span:
-        chunker = DocumentChunker(
-            chunk_size=settings.document_chunk_size,
-            overlap=settings.document_chunk_overlap,
-        )
-        chunks = await chunker.chunk_text(content)
+        if use_page_aware:
+            page_boundaries_list = cast(list[dict[str, Any]], page_boundaries)
+            chunks = await PageAwareChunker(
+                chunk_size=settings.document_chunk_size,
+                overlap=settings.document_chunk_overlap,
+            ).chunk_text(content, page_boundaries_list)
+        else:
+            chunks = await DocumentChunker(
+                chunk_size=settings.document_chunk_size,
+                overlap=settings.document_chunk_overlap,
+            ).chunk_text(content)
         record_document_chunks(doc_task.doc_type, len(chunks))
         if chunk_span is not None:
             chunk_span.set_attribute(_ATTR_CHUNK_COUNT, len(chunks))
 
-    # Assign page numbers to chunks if page boundaries are available (PDFs)
-    page_boundaries = file_metadata.get("page_boundaries")
-    if doc_task.doc_type == "file" and page_boundaries is not None:
+    # Assign page numbers for the char-based path (page-aware already sets them).
+    if (
+        not use_page_aware
+        and doc_task.doc_type == "file"
+        and page_boundaries is not None
+    ):
         # Type narrowing: page_boundaries is guaranteed to be list[dict] here
         page_boundaries_list = cast(list[dict[str, Any]], page_boundaries)
         with trace_operation(