fix(document-processors): escalate glyph-corrupt PDFs to the structured tier
The fast (pypdfium2) extractor can leak raw glyph codes on subset fonts with a broken /ToUnicode CMap. The result scores high on the existing text-quality heuristic -- a uniform glyph/Caesar offset preserves whitespace and token lengths -- yet is unsearchable. The structured (pymupdf) tier extracts the same pages correctly. Add a language-agnostic C0-control-character-ratio signal to the tier-0 classifier that detects this corruption and routes the document to a new `structured` recommended_tier. Wire the fast->structured hop on the inline path and generalise it so a low-quality-but-non-empty layer also tries structured before OCR -- the inline and external ingest modes now follow the full fast->structured->ocr ladder identically. A scanned / no-text-layer document (total_chars == 0) still shortcuts straight to OCR, since a text extractor cannot recover a pure raster. New per-tenant tunable DOCUMENT_GLYPH_CORRUPTION_RATIO (default 0.02); escalation metrics gain a `corrupt_glyphs` reason label. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
60df141442
commit
cf7209cd85
@@ -49,7 +49,7 @@ class EscalationDecision:
|
||||
|
||||
kind: Literal["hop", "suppressed"]
|
||||
to_tier: str
|
||||
reason: Literal["empty_text", "low_confidence"]
|
||||
reason: Literal["empty_text", "low_confidence", "corrupt_glyphs"]
|
||||
|
||||
|
||||
def next_tier(current: str) -> str | None:
|
||||
@@ -80,10 +80,11 @@ class EscalateError(Exception):
|
||||
the junk text is never indexed, and it must never be swallowed by a broad
|
||||
``except Exception`` on the indexing path.
|
||||
|
||||
``reason`` uses the existing escalation label vocabulary. This PR raises
|
||||
``empty_text`` (scanned / no text layer) and ``low_confidence`` (junk text
|
||||
layer); ``unsupported`` and ``forced`` are reserved for future callers and
|
||||
not raised yet.
|
||||
``reason`` uses the existing escalation label vocabulary: ``empty_text``
|
||||
(scanned / no text layer), ``low_confidence`` (junk text layer), and
|
||||
``corrupt_glyphs`` (a usable-looking layer whose extractor leaked raw glyph
|
||||
codes -- the broken-/ToUnicode case -- recovered by a different in-cluster
|
||||
extractor); ``unsupported`` and ``forced`` are reserved for future callers.
|
||||
"""
|
||||
|
||||
def __init__(self, *, from_tier: str, to_tier: str, reason: str) -> None:
|
||||
|
||||
Reference in New Issue
Block a user