fix(vector): route empty page_boundaries to char-based path; test ws offsets
Address claude-review round 1 on PR #868: - use_page_aware now gates on `bool(page_boundaries)` instead of `is not None`, so a PDF that yields an empty boundary list takes the char-based path explicitly (assign_page_numbers no-ops on []) rather than the page-aware chunker's no-boundaries fallback. Same result, clearer intent. - add test_oversized_page_with_leading_whitespace_offsets, exercising the start+start_index offset path for an oversized page whose sub-chunks have leading whitespace. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
4977216b62
commit
8d20339b3a
@@ -693,7 +693,10 @@ async def _index_document(
|
||||
use_page_aware = (
|
||||
settings.document_chunk_page_aware
|
||||
and doc_task.doc_type == "file"
|
||||
and page_boundaries is not None
|
||||
# Truthy (not just "is not None"): an empty list carries no pages, so
|
||||
# route it through the char-based path rather than the page-aware
|
||||
# chunker's no-boundaries fallback.
|
||||
and bool(page_boundaries)
|
||||
)
|
||||
with trace_operation(
|
||||
"vector_sync.chunk_text",
|
||||
|
||||
Reference in New Issue
Block a user