fix(document-processors): escalate glyph-corrupt PDFs to the structured tier

The fast (pypdfium2) extractor can leak raw glyph codes on subset fonts with a
broken /ToUnicode CMap. The result scores high on the existing text-quality
heuristic -- a uniform glyph/Caesar offset preserves whitespace and token
lengths -- yet is unsearchable. The structured (pymupdf) tier extracts the same
pages correctly.

Add a language-agnostic C0-control-character-ratio signal to the tier-0
classifier that detects this corruption and routes the document to a new
`structured` recommended_tier. Wire the fast->structured hop on the inline path
and generalise it so a low-quality-but-non-empty layer also tries structured
before OCR -- the inline and external ingest modes now follow the full
fast->structured->ocr ladder identically. A scanned / no-text-layer document
(total_chars == 0) still shortcuts straight to OCR, since a text extractor
cannot recover a pure raster.

New per-tenant tunable DOCUMENT_GLYPH_CORRUPTION_RATIO (default 0.02); escalation
metrics gain a `corrupt_glyphs` reason label.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-16 20:06:35 +02:00
co-authored by Claude Opus 4.8
parent 60df141442
commit cf7209cd85
8 changed files with 412 additions and 54 deletions
@@ -49,7 +49,7 @@ class EscalationDecision:
kind: Literal["hop", "suppressed"]
to_tier: str
reason: Literal["empty_text", "low_confidence"]
reason: Literal["empty_text", "low_confidence", "corrupt_glyphs"]
def next_tier(current: str) -> str | None:
@@ -80,10 +80,11 @@ class EscalateError(Exception):
the junk text is never indexed, and it must never be swallowed by a broad
``except Exception`` on the indexing path.
``reason`` uses the existing escalation label vocabulary. This PR raises
``empty_text`` (scanned / no text layer) and ``low_confidence`` (junk text
layer); ``unsupported`` and ``forced`` are reserved for future callers and
not raised yet.
``reason`` uses the existing escalation label vocabulary: ``empty_text``
(scanned / no text layer), ``low_confidence`` (junk text layer), and
``corrupt_glyphs`` (a usable-looking layer whose extractor leaked raw glyph
codes -- the broken-/ToUnicode case -- recovered by a different in-cluster
extractor); ``unsupported`` and ``forced`` are reserved for future callers.
"""
def __init__(self, *, from_tier: str, to_tier: str, reason: str) -> None: