fix: Add async/await, PDF metadata, and type safety fixes
This commit addresses multiple issues with async operations, PDF metadata extraction, and type safety in document processing and search. ## Async/Await Fixes - processor.py:259 - Added await for chunker.chunk_text(content) - processor.py:270 - Added await for bm25_service.encode_batch(chunk_texts) - tests/unit/test_document_chunker.py - Converted all 12 test methods to async ## PDF Metadata Enhancement - pymupdf.py:143 - Added file_size metadata extraction - pymupdf.py:145-206 - Refactored to extract text page-by-page - Manually loop through pages instead of using page_chunks=True - Generate page_boundaries metadata for precise page tracking - Works around pymupdf.layout.activate() breaking page_chunks=True - processor.py:32-66 - Added assign_page_numbers() helper function - Assigns page numbers to chunks based on overlap with page boundaries - Handles chunks spanning multiple pages - processor.py:298-300 - Call assign_page_numbers() for PDF files ## Type Safety Fixes - bm25_hybrid.py:184 - Removed int() conversion of doc_id - semantic.py:131 - Removed int() conversion of doc_id - viz_routes.py:275 - Removed int() conversion of doc_id - Added comments documenting that doc_id can be int (notes) or str (file paths) ## Testing - All 18 tests passing (12 unit + 6 integration) - No type errors in modified files - Container logs show successful processing - Vector viz searches working correctly 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -15,6 +15,8 @@ class ChunkWithPosition:
|
||||
text: str
|
||||
start_offset: int # Character position where chunk starts
|
||||
end_offset: int # Character position where chunk ends (exclusive)
|
||||
page_number: int | None = None # Page number for PDF chunks (optional)
|
||||
metadata: dict | None = None # Additional processor-specific metadata (optional)
|
||||
|
||||
|
||||
class DocumentChunker:
|
||||
@@ -50,7 +52,7 @@ class DocumentChunker:
|
||||
strip_whitespace=True,
|
||||
)
|
||||
|
||||
def chunk_text(self, content: str) -> list[ChunkWithPosition]:
|
||||
async def chunk_text(self, content: str) -> list[ChunkWithPosition]:
|
||||
"""
|
||||
Split text into overlapping chunks with position tracking.
|
||||
|
||||
@@ -66,12 +68,17 @@ class DocumentChunker:
|
||||
Returns:
|
||||
List of chunks with their character positions in the original content
|
||||
"""
|
||||
import anyio
|
||||
|
||||
# Handle empty content - return single empty chunk for backward compatibility
|
||||
if not content:
|
||||
return [ChunkWithPosition(text="", start_offset=0, end_offset=0)]
|
||||
|
||||
# Use LangChain to create documents with position tracking
|
||||
docs = self.splitter.create_documents([content])
|
||||
# Run CPU-bound text splitting in thread pool to avoid blocking event loop
|
||||
docs = await anyio.to_thread.run_sync(
|
||||
self.splitter.create_documents,
|
||||
[content],
|
||||
)
|
||||
|
||||
# Convert LangChain Documents to ChunkWithPosition objects
|
||||
chunks = [
|
||||
|
||||
Reference in New Issue
Block a user