fix(vector): route empty page_boundaries to char-based path; test ws offsets

Address claude-review round 1 on PR #868:
- use_page_aware now gates on `bool(page_boundaries)` instead of
  `is not None`, so a PDF that yields an empty boundary list takes the
  char-based path explicitly (assign_page_numbers no-ops on []) rather than
  the page-aware chunker's no-boundaries fallback. Same result, clearer intent.
- add test_oversized_page_with_leading_whitespace_offsets, exercising the
  start+start_index offset path for an oversized page whose sub-chunks have
  leading whitespace.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-06 13:56:30 +02:00
co-authored by Claude Opus 4.8
parent 4977216b62
commit 8d20339b3a
2 changed files with 26 additions and 1 deletions
+4 -1
View File
@@ -693,7 +693,10 @@ async def _index_document(
use_page_aware = (
settings.document_chunk_page_aware
and doc_task.doc_type == "file"
and page_boundaries is not None
# Truthy (not just "is not None"): an empty list carries no pages, so
# route it through the char-based path rather than the page-aware
# chunker's no-boundaries fallback.
and bool(page_boundaries)
)
with trace_operation(
"vector_sync.chunk_text",