Chris CoutinhoandClaude Opus 4.8
c48a797896
Replaces single-engine pymupdf4llm extraction with a tiered pipeline (Deck #205, follows the tier-0 classifier #855). pypdfium2 becomes the default and only hot-path PDF extractor; pymupdf4llm is deprecated to a rollback toggle. Why: pymupdf4llm's O(n^2) find_tables drove the OOM (#852) and the form-PDF parse timeouts (#856), carries AGPL/commercial licensing liability, and -- per the benchmarks -- recovers near-zero usable tables on the real corpus. pypdfium2 (Apache/BSD) extracts the same text far faster (Student 1a.pdf: 120s timeout -> 0.2s) with no table-detection bomb. - document_processors/pypdfium2_fast.py: tier-1 "fast" processor emitting text + exact page_boundaries (the pdf_highlighter contract). pymupdf processor is now tier "structured" (the rollback engine), registered but not default. - registry: tiered routing in ProcessorRegistry. tier-1 fast extracts, then classification is DERIVED from that text (classifier.classify_from_text -- no PDF re-open), records the classification metrics, and escalates scanned / no-text-layer docs to the "ocr" tier when document_ocr_enabled (default off; no provider yet, so fast is terminal). Wires record_document_escalation + the real "escalated" span attribute (was hardcoded False). - Removes the separate _shadow_classify pass from vector/processor.py -- it re-opened every PDF and re-extracted text (~0.5-1.3s/doc of pure duplicated CPU that lowered throughput); classification now rides the tier-1 extraction. - Settings: document_tier1_engine ("pypdfium2" default | "pymupdf" rollback, enum-validated), document_ocr_enabled (default false). Tests: pypdfium2 extractor, registry tiering (fast routing, rollback, classify recording, OCR escalation on/off), classify_from_text. Full unit suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
25 lines
904 B
Python
25 lines
904 B
Python
"""Document processing plugins for extracting text from various file formats."""
|
|
|
|
from .base import DocumentProcessor, ProcessingResult, ProcessorError
|
|
from .pymupdf import PyMuPDFProcessor
|
|
from .pypdfium2_fast import Pypdfium2FastProcessor
|
|
from .registry import ProcessorRegistry, get_registry
|
|
|
|
# Register processors at module initialization. The tiered PDF pipeline selects
|
|
# by tier (not priority): Pypdfium2FastProcessor is the ``fast`` tier and
|
|
# PyMuPDFProcessor the ``structured`` escalation target. Priority still orders
|
|
# the non-tiered fallback path and other MIME types.
|
|
_registry = get_registry()
|
|
_registry.register(Pypdfium2FastProcessor(), priority=20)
|
|
_registry.register(PyMuPDFProcessor(), priority=10)
|
|
|
|
__all__ = [
|
|
"DocumentProcessor",
|
|
"ProcessingResult",
|
|
"ProcessorError",
|
|
"ProcessorRegistry",
|
|
"get_registry",
|
|
"PyMuPDFProcessor",
|
|
"Pypdfium2FastProcessor",
|
|
]
|