feat: tiered PDF processor with pypdfium2 fast path (deprecate pymupdf4llm)
Replaces single-engine pymupdf4llm extraction with a tiered pipeline (Deck #205, follows the tier-0 classifier #855). pypdfium2 becomes the default and only hot-path PDF extractor; pymupdf4llm is deprecated to a rollback toggle. Why: pymupdf4llm's O(n^2) find_tables drove the OOM (#852) and the form-PDF parse timeouts (#856), carries AGPL/commercial licensing liability, and -- per the benchmarks -- recovers near-zero usable tables on the real corpus. pypdfium2 (Apache/BSD) extracts the same text far faster (Student 1a.pdf: 120s timeout -> 0.2s) with no table-detection bomb. - document_processors/pypdfium2_fast.py: tier-1 "fast" processor emitting text + exact page_boundaries (the pdf_highlighter contract). pymupdf processor is now tier "structured" (the rollback engine), registered but not default. - registry: tiered routing in ProcessorRegistry. tier-1 fast extracts, then classification is DERIVED from that text (classifier.classify_from_text -- no PDF re-open), records the classification metrics, and escalates scanned / no-text-layer docs to the "ocr" tier when document_ocr_enabled (default off; no provider yet, so fast is terminal). Wires record_document_escalation + the real "escalated" span attribute (was hardcoded False). - Removes the separate _shadow_classify pass from vector/processor.py -- it re-opened every PDF and re-extracted text (~0.5-1.3s/doc of pure duplicated CPU that lowered throughput); classification now rides the tier-1 extraction. - Settings: document_tier1_engine ("pypdfium2" default | "pymupdf" rollback, enum-validated), document_ocr_enabled (default false). Tests: pypdfium2 extractor, registry tiering (fast routing, rollback, classify recording, OCR escalation on/off), classify_from_text. Full unit suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
967298ddbe
commit
c48a797896
@@ -156,3 +156,23 @@ def test_image_heavy_flag_without_ocr_routing():
|
||||
assert "image_heavy" in c.flags
|
||||
assert c.recommended_tier == "fast"
|
||||
assert c.ocr_page_fraction < clf.OCR_PAGE_FRACTION
|
||||
|
||||
|
||||
# --- classify_from_text (hot-path, derived from tier-1 extraction) -----------
|
||||
|
||||
|
||||
def test_classify_from_text_clean_routes_fast():
|
||||
txt = "the quick brown fox jumps over the lazy dog " * 3
|
||||
c = clf.classify_from_text(
|
||||
txt, [{"page": 1, "start_offset": 0, "end_offset": len(txt)}]
|
||||
)
|
||||
assert c.recommended_tier == "fast"
|
||||
assert c.mean_text_quality > 0.8
|
||||
assert c.flags == set()
|
||||
|
||||
|
||||
def test_classify_from_text_empty_routes_ocr():
|
||||
c = clf.classify_from_text("", [{"page": 1, "start_offset": 0, "end_offset": 0}])
|
||||
assert c.recommended_tier == "ocr"
|
||||
assert "no_text_layer" in c.flags
|
||||
assert c.total_chars == 0
|
||||
|
||||
Reference in New Issue
Block a user