Chris CoutinhoandClaude Opus 4.8
c48a797896
Replaces single-engine pymupdf4llm extraction with a tiered pipeline (Deck #205, follows the tier-0 classifier #855). pypdfium2 becomes the default and only hot-path PDF extractor; pymupdf4llm is deprecated to a rollback toggle. Why: pymupdf4llm's O(n^2) find_tables drove the OOM (#852) and the form-PDF parse timeouts (#856), carries AGPL/commercial licensing liability, and -- per the benchmarks -- recovers near-zero usable tables on the real corpus. pypdfium2 (Apache/BSD) extracts the same text far faster (Student 1a.pdf: 120s timeout -> 0.2s) with no table-detection bomb. - document_processors/pypdfium2_fast.py: tier-1 "fast" processor emitting text + exact page_boundaries (the pdf_highlighter contract). pymupdf processor is now tier "structured" (the rollback engine), registered but not default. - registry: tiered routing in ProcessorRegistry. tier-1 fast extracts, then classification is DERIVED from that text (classifier.classify_from_text -- no PDF re-open), records the classification metrics, and escalates scanned / no-text-layer docs to the "ocr" tier when document_ocr_enabled (default off; no provider yet, so fast is terminal). Wires record_document_escalation + the real "escalated" span attribute (was hardcoded False). - Removes the separate _shadow_classify pass from vector/processor.py -- it re-opened every PDF and re-extracted text (~0.5-1.3s/doc of pure duplicated CPU that lowered throughput); classification now rides the tier-1 extraction. - Settings: document_tier1_engine ("pypdfium2" default | "pymupdf" rollback, enum-validated), document_ocr_enabled (default false). Tests: pypdfium2 extractor, registry tiering (fast routing, rollback, classify recording, OCR escalation on/off), classify_from_text. Full unit suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
60 lines
1.8 KiB
Python
60 lines
1.8 KiB
Python
"""Unit tests for the tier-1 pypdfium2 fast PDF extractor."""
|
|
|
|
import pymupdf
|
|
import pytest
|
|
|
|
from nextcloud_mcp_server.document_processors.pypdfium2_fast import (
|
|
Pypdfium2FastProcessor,
|
|
)
|
|
|
|
pytestmark = pytest.mark.unit
|
|
|
|
|
|
def _digital_pdf(
|
|
pages: int = 3, body: str = "Hello world this is clean text. "
|
|
) -> bytes:
|
|
doc = pymupdf.open()
|
|
for _ in range(pages):
|
|
page = doc.new_page(width=595, height=842)
|
|
page.insert_text((50, 60), body * 8)
|
|
data: bytes = doc.tobytes()
|
|
doc.close()
|
|
return data
|
|
|
|
|
|
def test_processor_identity():
|
|
p = Pypdfium2FastProcessor()
|
|
assert p.name == "pypdfium2_fast"
|
|
assert p.tier == "fast"
|
|
assert "application/pdf" in p.supported_mime_types
|
|
|
|
|
|
async def test_extract_text_and_exact_page_boundaries():
|
|
p = Pypdfium2FastProcessor()
|
|
result = await p.process(_digital_pdf(pages=3), "application/pdf", filename="t.pdf")
|
|
|
|
assert result.success is True
|
|
assert "Hello world" in result.text
|
|
assert result.metadata["page_count"] == 3
|
|
|
|
boundaries = result.metadata["page_boundaries"]
|
|
assert len(boundaries) == 3
|
|
assert boundaries[0]["start_offset"] == 0
|
|
# Offsets must index exactly into the returned text (pdf_highlighter contract).
|
|
assert boundaries[-1]["end_offset"] == len(result.text)
|
|
for prev, nxt in zip(boundaries, boundaries[1:]):
|
|
assert prev["end_offset"] == nxt["start_offset"]
|
|
|
|
|
|
async def test_malformed_pdf_returns_success_false():
|
|
p = Pypdfium2FastProcessor()
|
|
result = await p.process(b"not a pdf at all", "application/pdf", filename="bad.pdf")
|
|
|
|
assert result.success is False
|
|
assert result.text == ""
|
|
assert result.metadata["parse_failed_reason"] == "error"
|
|
|
|
|
|
async def test_health_check():
|
|
assert await Pypdfium2FastProcessor().health_check() is True
|