feat: tiered PDF processor with pypdfium2 fast path (deprecate pymupdf4llm)
Replaces single-engine pymupdf4llm extraction with a tiered pipeline (Deck #205, follows the tier-0 classifier #855). pypdfium2 becomes the default and only hot-path PDF extractor; pymupdf4llm is deprecated to a rollback toggle. Why: pymupdf4llm's O(n^2) find_tables drove the OOM (#852) and the form-PDF parse timeouts (#856), carries AGPL/commercial licensing liability, and -- per the benchmarks -- recovers near-zero usable tables on the real corpus. pypdfium2 (Apache/BSD) extracts the same text far faster (Student 1a.pdf: 120s timeout -> 0.2s) with no table-detection bomb. - document_processors/pypdfium2_fast.py: tier-1 "fast" processor emitting text + exact page_boundaries (the pdf_highlighter contract). pymupdf processor is now tier "structured" (the rollback engine), registered but not default. - registry: tiered routing in ProcessorRegistry. tier-1 fast extracts, then classification is DERIVED from that text (classifier.classify_from_text -- no PDF re-open), records the classification metrics, and escalates scanned / no-text-layer docs to the "ocr" tier when document_ocr_enabled (default off; no provider yet, so fast is terminal). Wires record_document_escalation + the real "escalated" span attribute (was hardcoded False). - Removes the separate _shadow_classify pass from vector/processor.py -- it re-opened every PDF and re-extracted text (~0.5-1.3s/doc of pure duplicated CPU that lowered throughput); classification now rides the tier-1 extraction. - Settings: document_tier1_engine ("pypdfium2" default | "pymupdf" rollback, enum-validated), document_ocr_enabled (default false). Tests: pypdfium2 extractor, registry tiering (fast routing, rollback, classify recording, OCR escalation on/off), classify_from_text. Full unit suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
967298ddbe
commit
c48a797896
@@ -2,10 +2,15 @@
|
||||
|
||||
from .base import DocumentProcessor, ProcessingResult, ProcessorError
|
||||
from .pymupdf import PyMuPDFProcessor
|
||||
from .pypdfium2_fast import Pypdfium2FastProcessor
|
||||
from .registry import ProcessorRegistry, get_registry
|
||||
|
||||
# Register processors at module initialization
|
||||
# Register processors at module initialization. The tiered PDF pipeline selects
|
||||
# by tier (not priority): Pypdfium2FastProcessor is the ``fast`` tier and
|
||||
# PyMuPDFProcessor the ``structured`` escalation target. Priority still orders
|
||||
# the non-tiered fallback path and other MIME types.
|
||||
_registry = get_registry()
|
||||
_registry.register(Pypdfium2FastProcessor(), priority=20)
|
||||
_registry.register(PyMuPDFProcessor(), priority=10)
|
||||
|
||||
__all__ = [
|
||||
@@ -15,4 +20,5 @@ __all__ = [
|
||||
"ProcessorRegistry",
|
||||
"get_registry",
|
||||
"PyMuPDFProcessor",
|
||||
"Pypdfium2FastProcessor",
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user