feat: tiered PDF processor with pypdfium2 fast path (deprecate pymupdf4llm)
Replaces single-engine pymupdf4llm extraction with a tiered pipeline (Deck #205, follows the tier-0 classifier #855). pypdfium2 becomes the default and only hot-path PDF extractor; pymupdf4llm is deprecated to a rollback toggle. Why: pymupdf4llm's O(n^2) find_tables drove the OOM (#852) and the form-PDF parse timeouts (#856), carries AGPL/commercial licensing liability, and -- per the benchmarks -- recovers near-zero usable tables on the real corpus. pypdfium2 (Apache/BSD) extracts the same text far faster (Student 1a.pdf: 120s timeout -> 0.2s) with no table-detection bomb. - document_processors/pypdfium2_fast.py: tier-1 "fast" processor emitting text + exact page_boundaries (the pdf_highlighter contract). pymupdf processor is now tier "structured" (the rollback engine), registered but not default. - registry: tiered routing in ProcessorRegistry. tier-1 fast extracts, then classification is DERIVED from that text (classifier.classify_from_text -- no PDF re-open), records the classification metrics, and escalates scanned / no-text-layer docs to the "ocr" tier when document_ocr_enabled (default off; no provider yet, so fast is terminal). Wires record_document_escalation + the real "escalated" span attribute (was hardcoded False). - Removes the separate _shadow_classify pass from vector/processor.py -- it re-opened every PDF and re-extracted text (~0.5-1.3s/doc of pure duplicated CPU that lowered throughput); classification now rides the tier-1 extraction. - Settings: document_tier1_engine ("pypdfium2" default | "pymupdf" rollback, enum-validated), document_ocr_enabled (default false). Tests: pypdfium2 extractor, registry tiering (fast routing, rollback, classify recording, OCR escalation on/off), classify_from_text. Full unit suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
967298ddbe
commit
c48a797896
@@ -1,32 +1,31 @@
|
||||
"""Tier-0 document classifier.
|
||||
|
||||
A cheap (<~1s), local pre-pass over a PDF that decides which extraction tier a
|
||||
document should start in, BEFORE the expensive parse. It runs in *shadow mode*
|
||||
first: emit the signals as metrics, change no routing, and gather per-tenant
|
||||
data to tune the thresholds.
|
||||
Decides which extraction tier a PDF should escalate to, from cheap signals:
|
||||
* text_quality -- is the text layer usable, or mashed/space-less junk? (the
|
||||
"Student 147" lesson: a text layer can exist yet be unusable, e.g.
|
||||
"01322234567mobile")
|
||||
* image_coverage -- a page that is mostly a raster image is a scan/photo whose
|
||||
content isn't fully in any text layer.
|
||||
* no text layer -- the strongest OCR signal available from text alone.
|
||||
|
||||
Signals (all cheap; no get_drawings, which is itself slow on the graphics-heavy
|
||||
pages we'd want to flag -- the parse-time ``graphics_limit`` already makes those
|
||||
safe, and the tier-1 quality gate catches unrecovered tables post-extraction):
|
||||
Two entry points:
|
||||
* ``classify_from_text(text, page_boundaries)`` -- the HOT PATH. Derives the
|
||||
text-quality/no-text-layer signal from the text the registry's tier-1 step
|
||||
already extracted, so it adds ~no cost. No image analysis.
|
||||
* ``classify_pdf(content)`` -- a standalone/diagnostic pass that re-opens the
|
||||
PDF and adds image-coverage analysis. More expensive; used off the hot path.
|
||||
|
||||
* text_layer_chars -- extractable text per page
|
||||
* text_quality -- is the text layer usable, or mashed/space-less junk?
|
||||
(the "Student 147" lesson: a text layer can exist yet
|
||||
be unusable, e.g. "01322234567mobile")
|
||||
* image_coverage -- fraction of the page covered by raster images
|
||||
(full-page image + poor text => scanned)
|
||||
Recommended tier:
|
||||
* ``ocr`` -- scanned / no-usable-text-layer (route to tier 3, when enabled)
|
||||
* ``fast`` -- a usable digital text layer (stay on tier 1)
|
||||
|
||||
From these it picks a recommended starting tier:
|
||||
* ``ocr`` -- scanned / image-only / bad-text-layer (route to tier 3)
|
||||
* ``fast`` -- a usable digital text layer (route to tier 1)
|
||||
|
||||
``structured`` (tier 2 / docling) is intentionally not produced here -- that tier
|
||||
is a separate service and is reached via the tier-1 quality gate, not tier-0.
|
||||
``structured`` (tier 2 / docling) is a separate service, not produced here.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -40,6 +39,8 @@ IMAGE_COVERAGE_SCANNED = 0.80
|
||||
MIN_TEXT_QUALITY = 0.45
|
||||
# Fraction of sampled pages that must look scanned/bad for a doc->ocr verdict.
|
||||
OCR_PAGE_FRACTION = 0.5
|
||||
# A page with fewer extracted chars than this has effectively no text layer.
|
||||
MIN_PAGE_CHARS = 16
|
||||
|
||||
_WORD_RE = re.compile(r"\S+")
|
||||
|
||||
@@ -177,3 +178,53 @@ def classify_pdf(content: bytes) -> DocClassification:
|
||||
flags=flags,
|
||||
pages=pages,
|
||||
)
|
||||
|
||||
|
||||
def classify_from_text(
|
||||
full_text: str, page_boundaries: list[dict[str, Any]]
|
||||
) -> DocClassification:
|
||||
"""Classify from text already extracted by tier-1 -- no PDF re-open.
|
||||
|
||||
The hot-path classifier: it derives the text-quality signal from the
|
||||
extraction the registry already ran, so it adds ~no cost (vs ``classify_pdf``,
|
||||
which re-opens the PDF and re-extracts). It does NOT do image analysis, so it
|
||||
cannot distinguish a scanned-with-text-layer page (that needs the image pass,
|
||||
which only matters once OCR routing is enabled). A page with effectively no
|
||||
text layer is the one OCR-worthy signal available from text alone.
|
||||
|
||||
``page_boundaries`` are ``{page, start_offset, end_offset}`` indexing into
|
||||
``full_text`` (the tier-1/pdf_highlighter contract).
|
||||
"""
|
||||
pages: list[PageSignals] = []
|
||||
for b in page_boundaries:
|
||||
seg = full_text[b["start_offset"] : b["end_offset"]]
|
||||
needs_ocr = len(seg.strip()) < MIN_PAGE_CHARS
|
||||
pages.append(
|
||||
PageSignals(b["page"], len(seg), 0.0, _text_quality(seg), needs_ocr)
|
||||
)
|
||||
|
||||
sampled = len(pages)
|
||||
total_chars = sum(p.char_count for p in pages)
|
||||
mean_quality = (
|
||||
round(sum(p.text_quality for p in pages) / sampled, 3) if sampled else 0.0
|
||||
)
|
||||
ocr_frac = (sum(p.needs_ocr for p in pages) / sampled) if sampled else 1.0
|
||||
|
||||
flags: set[str] = set()
|
||||
if total_chars == 0:
|
||||
flags.add("no_text_layer")
|
||||
elif mean_quality < MIN_TEXT_QUALITY:
|
||||
flags.add("bad_text_layer")
|
||||
|
||||
recommended = "ocr" if ocr_frac >= OCR_PAGE_FRACTION else "fast"
|
||||
|
||||
return DocClassification(
|
||||
page_count=len(page_boundaries),
|
||||
sampled_pages=sampled,
|
||||
total_chars=total_chars,
|
||||
mean_text_quality=mean_quality,
|
||||
ocr_page_fraction=round(ocr_frac, 3),
|
||||
recommended_tier=recommended,
|
||||
flags=flags,
|
||||
pages=pages,
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user