Chris CoutinhoandClaude
b8010270c1
This commit addresses multiple issues with async operations, PDF metadata extraction, and type safety in document processing and search. ## Async/Await Fixes - processor.py:259 - Added await for chunker.chunk_text(content) - processor.py:270 - Added await for bm25_service.encode_batch(chunk_texts) - tests/unit/test_document_chunker.py - Converted all 12 test methods to async ## PDF Metadata Enhancement - pymupdf.py:143 - Added file_size metadata extraction - pymupdf.py:145-206 - Refactored to extract text page-by-page - Manually loop through pages instead of using page_chunks=True - Generate page_boundaries metadata for precise page tracking - Works around pymupdf.layout.activate() breaking page_chunks=True - processor.py:32-66 - Added assign_page_numbers() helper function - Assigns page numbers to chunks based on overlap with page boundaries - Handles chunks spanning multiple pages - processor.py:298-300 - Call assign_page_numbers() for PDF files ## Type Safety Fixes - bm25_hybrid.py:184 - Removed int() conversion of doc_id - semantic.py:131 - Removed int() conversion of doc_id - viz_routes.py:275 - Removed int() conversion of doc_id - Added comments documenting that doc_id can be int (notes) or str (file paths) ## Testing - All 18 tests passing (12 unit + 6 integration) - No type errors in modified files - Container logs show successful processing - Vector viz searches working correctly 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
19 lines
526 B
Python
19 lines
526 B
Python
"""Document processing plugins for extracting text from various file formats."""
|
|
|
|
from .base import DocumentProcessor, ProcessingResult, ProcessorError
|
|
from .pymupdf import PyMuPDFProcessor
|
|
from .registry import ProcessorRegistry, get_registry
|
|
|
|
# Register processors at module initialization
|
|
_registry = get_registry()
|
|
_registry.register(PyMuPDFProcessor(), priority=10)
|
|
|
|
__all__ = [
|
|
"DocumentProcessor",
|
|
"ProcessingResult",
|
|
"ProcessorError",
|
|
"ProcessorRegistry",
|
|
"get_registry",
|
|
"PyMuPDFProcessor",
|
|
]
|